next up previous
Next: RFI/radars Up: E-ALFA Hardware Previous: SDFITS notes



Options for transport/storage of datasets by E-ALFA users

The size of E-ALFA data sets raises issues concerning the mode of data delivery to systems beyond the raw data archive storage location. While NAIC may assume responsibility for delivery of the level I data products, it is also likely that the observing team will need and desire access to the raw data. Data rates may be of order 0.5 MB/sec and data sets therefore, large. These volumes may be large to many of us, but they are not extraordinary by today's general standards. (The LSST will record 18 TB/night.)

Three principle options exist for the transport of raw data: (a) network transfer, (b) magnetic tape and (c) hard disk. We note several points:
  1. The current internet connection to Arecibo can provide sustained data transfer of order 340 KB/sec (according to Arun).

  2. If the raw data are stored initially in an archive maintained at NAIC but physically located at a mainland site such as CTC or NCSA, then the access rate for Internet2 sites should be higher than 0.5 MB/sec, thus keeping up with the anticipated E-ALFA data rate.

  3. The current Mammoth drives at Arecibo have a write rate of 1-5 MB/sec, max. capacity of 60 GB/tape. The technology appears not to be moving forward (according to Arun).

  4. Numerous astronomical facilities use DLT tapes. SuperDLT drives write at 16 MB/sec up to 160 GB/tape. Tape drives cost about $3.3K and a tape costs about $80 (varies some).

  5. The price of bare hard disk is currently about $0.8/GB (for USB ~$2-3/GB) and continues to fall.

  6. While tape shipment has been the designated mode for large data volume initiatives such as VLBI and SDSS, those programs are moving toward disk-based recording systems.

  7. Current setups which allow rapid access to hard disks (>~ 20 MB/sec, i.e. Firewire) are worth about $300-$500, an order of magnitude cheaper than tape drives.

  8. IDE drives appear to be surpassing SCSI ones. 2 TB of hard disk storage can now be constructed from 8 X 250 GB IDE drives in a single enclosure. Some initial investment ($5K) is needed to provide a compatible controller (e.g. with 8 IDE channels) for such a tower of spinning disk. Compatibility at both ends is required.


Recommendations:
  1. NAIC should archive the raw data, possibly in collaboration with partners, and provide access to these data by members of the E-ALFA science team via network transfer from a mainland location.

  2. A basic, inexpensive and easy-to-maintain Linux-based hard disk storage/retrieval system which serves typical researcher needs (e.g., budgets, resources) should be specified and developed as a collaborative effort led by NAIC.




next up previous
Next: RFI/radars Up: E-ALFA Hardware Previous: SDFITS notes
This page created and maintained by Martha Haynes.

Last modified: Wed Jun 18 08:51:10 EDT 2003