Today we have a super-duper-uber-fantastic guest post from the Cranky Engineer, aka DGentry of codingrelic.geekhold.com. Check out his blog. It’s WICKED AWESOME.
Also, you might recall that DGenery was the BIG WINNER of the Cranky Product Manager’s caption contest for the “7 Types of Engineers.” He won a fancy pants coffee mug. AND HE LOVES IT. Maybe YOU should buy a coffee mug too….
—————————-
From: Cranky Engineer #4
To: Product Manager #4
Date: Apr 1, 2009
Subject: Twitter reliability enhancement functional spec
Got your PRD about the 3G connectivity issues at the SXSW conference and the resulting drop in twitter volume, thanks. We couldn’t help but notice the lack of any actual requirements in the requirements document, but the main point seemed to be to “make it mo’ betta.” As mobile tower placement is subject to considerable regulation, we presume that the PRD is not calling for a build-out of a new national 3G network, but rather to eliminate dependencies on external factors beyond our control. Thus:
- No dependency on mobile network capacity
- No impact from density of surrounding buildings or obstacles
- No necessity for the presence or absence of electricity.
- No requirement for the user to own or know how to operate a computer
I think we’ve come up with a proposal which meets these requirements. The only remaining dependency is literacy. As we do not control the educational system in this country, this dependency may also need to be eliminated in a future version of this document.
Abstract of Proposal
Twitter usage is growing robustly, but is hampered by insufficient network capacity at heavily attended tech events such as SXSW and anywhere Steve Jobs gets up on a stage. Tweet volume from such events is lower than would be expected due to the connectivity issues.
It is proposed that designated drop zones be established at major tech events, where conference attendees can send and receive tweets. To avoid any dependency on telco infrastructure, tweets will be delivered to these TweetDrops by a robust and innovative mechanism.
1. Encoding
Each 140 character tweet is printed on a 5 mm wide paper strip, which is laminated and wrapped about the leg of a single carrier pigeon. The tweet is printed using a standard UPC barcode for ease of decoding at the remote end. The lamination is not necessary in theory, but in practice the messages often need to be wiped clean before processing (these are pigeons, after all).
2. Ordered Delivery
Conversations become difficult to follow if tweets are delivered out of order, but ordering is not guaranteed by the underlying network topology in this case. Therefore an 8 bit sequence number will be prepended to the barcode, which will be used to re-order pigeons arriving at the destination.
With 8 bits for sequencing information, only 256 pigeons can be launched at a time. The next wave of pigeons cannot be launched until it is certain that all members of the previous wave have left the system. To achieve acceptable throughput the natural lifespan of the pigeons cannot be used for this purpose.
Therefore, a small explosive device is fitted before launch to place a strong upper bound on the Time To Live (TTL) for that tweet. The next wave of tweets can be launched when the TTL for the previous wave has expired.
3. Loss detection
Due to the TTL mechanism employed for robust ordering it is possible that tweets will be dropped (or, more properly, exploded) in transit. To allow for retransmission an additional 8 bit acknowledgment field will be prepended to the barcode, and used to implement a sliding window protocol. (XXX Hang on, are we reinventing TCP here? Let’s have a sit-down about this before you forward it to PM, or we’ll end up rat-holing on this topic.)
Though lost tweets are expected to be a serious burden during initial deployment of this technology, it is believed that natural selection will result in a more reliable infrastructure as the less dependable population of pigeons is removed from the breeding stock by the TTL mechanism.
4. Operational Issues
The operations cost of this infrastructure is expected to be high, due to the need for replacement of pigeons whose TTL expired in transit. It is proposed that a breeding program be established in order to replenish capacity at minimal cost. This also nicely solves the problem of providing something for the pigeons to do between major tech events.
5. Direction of Future Work
The necessity of printing each tweet on a strip of paper and scanning the paper at the destination is a serious bottleneck in the communications path. A more efficient mechanism would be to download the tweets directly into the carrier pigeon via an existing Twitter API. Unfortunately the pigeons have strongly resisted attempts to implement any such mechanism.
The TTL mechanism can reasonably be expected to elicit a negative response from both animal rights activists and civil defense authorities, and should be considered only a temporary measure to reduce the time to market. A subsequent update to the infrastructure should investigate use of electromagnets to confuse the carrier’s natural homing instinct and send them somewhere where the tweet payload they carry will be harmless.
6. Security Considerations
No thorough analysis of the security of the carrier pigeon transport has been conducted. Because each pigeon will follow a different route to the destination owing to differences in wind currents, temperature gradients, and mood, it is unlikely that an attacker would be able to capture an entire conversation.
Additionally the TTL mechanism discourages interception of the pigeons in flight.
Encryption of the tweet payload is acceptable only if the pigeons can be guaranteed not to cross from within the United States to another sovereign nation, as such would violate US export and munitions laws.
7. References
[1] D. Waitzman, “A Standard for the Transmission of IP Datagrams on Avian Carriers,” RFC 1149, 1 April 1990.
