Data Transport Network
Example Applications
The best way to gain an appreciation for how the system works in practice is to examine a set of real-world usage examples. These include transferring files, real-time data processing, visualization and trouble notification.
Automated Data Transfers
The first example illustrates a common application: the need to transfer data files via an FTP-like process. It also highlights the manner in which we include legacy hardware systems in the transport system. An instrument writes data files to networked data drive on the server. A program running on the server watches for these files, and upon finding one, packages the data into a MIME-encoded email message and posts it to the appropriate newsgroup. The messages are forwarded when the local news server contacts the off-site server, which might be located at the investigator's home institution. On the remote server, a companion program would be waiting for the new messages to arrive, unpack the attached data files and save them to local storage. A summary of the transfer statistics could then be produced and posted to a web site or e-mailed to those who are interested.
Integrated Computation
The second example demonstrates how programs can access the data files to generate summary information or perform post-processing. Once a message has been posted to the news server, any client program (assuming that it has sufficient privileges to do so) can access a copy of the message. Using NNTP commands, the program can query the server to determine which messages are available and if any new messages have arrived since the last query. Once a message has been retrieved from the server and the data file attachments unpacked, the program can proceed as if it were working on the exact same files written by the instrument. Any results produced by the program can in turn be posted back to the news server in a separate newsgroup as a new data stream. From the system's point of view, there is no difference between the messages generated from a hardware instrument and those produced as a result of a software program.
This last example highlights one of the most important and fundamental aspects of the transport system and deserves further elaboration. What we have achieved is a clean separation between the producers of data files and the consumers who use them. An instrument collecting data and posting files into the system need not be aware of all the different programs that will later be accessing them. Why would multiple programs need to access the data files? Besides the copy of the data being sent off-site, one program might produce a quick-look summary plot, another performs post-processing and a third generates a dynamic web page showing the instrument's health and status. These programs can all be run on a computer separate from that doing data collection. The transport network provides a uniform means of accessing data files from the instruments.
Real-Time Instrument Displays
Our third usage example shows how the system can be used to generate local real-time instrument displays. At a site with multiple instruments, it is often the case that the activities of one instrument could be dependent on the results of another. There, real-time processing and display can greatly enhance the usefulness of both instruments and their science. The data transport network makes the data available in real-time and trivially allows for multiple viewing programs to be run on the same data set. In addition, because the news server holds all of the recently posted data files, it is simple to add a history capability for retrospective analysis.
E-mail Event Notification
The last example of the system capabilities is e-mail notification of interesting system events. Most mailing list software, such as the GNU Mailman program, can monitor a newsgroup and gateway messages to people who have subscribed to the list. Monitoring programs can watch the newsgroups associated with instruments that are supposed to be continuously sending data. If the data feed stops, the program can post a message to a trouble newsgroup indicating that a problem was detected. The message will then be gatewayed to the mailing list, where both the instrument's PI will be notified as well as a local site operator who can investigate the failure. Another mailing list application that we use is to notify researchers when a particular experiment was conducted with the radar.