Data Transport Network
How It Works
The system is based on the premise of sending data files as attachments within Usenet messages posted to a local server. The local news server collects the messages and periodically transmits them off-site when the link conditions permit. If the network goes down or the destination server becomes unavailable (perhaps the disk is full), the data files will be continue to be stored locally. The server will periodically attempt to deliver the files. The instruments need not be aware of this, and can continue acquiring data as if nothing were wrong.
The same architecture gracefully handles situations where the network throughput does not permit files to be transmitted as quickly as they are acquired. If a low-bandwidth link is up all the time, the files can be trickled out. This assumes, of course, that the instrument eventually pauses its data collection long enough to allow the backlog to clear. Instruments such as imagers, which only operate at night, naturally fit this pattern. Some instruments, however, produce data files that are simply too large to transmit over the link. In these cases, post-processing programs need to be run to produce smaller data sets. The real-time processing of data streams can be accomplished by writing a program which subscribes to a newsgroup and is called each time a new message arrives. The reduced data sets can then be posted to another newsgroup as a new data stream. By properly configuring the news servers, only these smaller data files are sent off-site.
A large number of companion programs are usually run in the system, performing functions such as the gathering and posting data files, post-processing of records and monitoring the health of the instruments and the system itself. The majority of these programs are written in Python and communicate between themselves using CORBA. Python has proved to be an ideal language for this system because it runs on a variety of platforms, allows for the development of reliable networked applications and supports a wide array of network protocols.