NGAS is short for Next Generation Archive System. It is both a concept and a 'reference' implementation. The concept is around since the year 2000 and originates at the European Southern Observatory (ESO). The implementation of the concept followed less than a year later and was deployed operationally to capture data from the Wide Field Imager (WFI) at the 2.2m telescope at the La Silla Observatory. The concept was triggered by a white paper showing that using standard off-the-shelve hard drives together with some data management software would be more economic than the usage of DVDs in juke boxes, which was the solution used at the ESO Archive at that time. A short study revealed that there was no software around which would satisfy the fairly unique requirements of ESO. In particular the requirement to be able to transport data on disks using courier between the observatory in Chile and ESO headquarters in Germany required very careful thinking. The white paper already outlined some of the basic software, hardware and operational concepts, which have been fully implemented during the following years, but it also outlined some more advanced concepts, which have not been fully implemented even now. Although the hardware changed quite dramatically, the software did not, but has proven its quite remarkable flexibility in many deployments around the world. NGAS is now being used in the following observatories/projects:
- ESO for the main archive in Garching/Germany holding all the data from the La-Silla Paranal Observatory.
- ESO for the archive holding the ALMA data for the European ALMA Regional Centre
- ALMA observatory Chile for the data capturing system at the Operations Support Facility (OSF) and the main ALMA Archive in Santiago
- NRAO for the main archive of the North American ALMA Science Centre
- NRAO for the eVLA and VLBI data archive.
- NAOJ for the main archive of the East Asian ALMA Regional Centre
- ICRAR for the data capturing and main archiving system for the Murchison Wide Field Array (MWA).
- ICRAR for the archiving system of the observational data and the final data products of the MWA GLEAM survey.
- MIT for the data archive of the MWA Epoch of Reionisation (EoR) observing project.
- CSIRO for the ASKAP post-pipeline data products and potentially for commissioning data stream to Pawsey and Sydney.
- NAOC for the FAST telescope
- Future: NRAO for the Greenbank telescope.
2 Comments
Anonymous
Nowadays, compared with the current Storage service, eg Amazon S3 http://aws.amazon.com/s3/details/ , AWS Import/Export http://aws.amazon.com/importexport/, what makes NGAS unique or superior in terms of functionality requirements and lifecycle costs ?
Andreas Wicenec
Amazon S3 would be $102,000/month for us right now and growing, and that does not include transfers out of S3 or in between S3 regions. All together we would have spent already several million on Amazon services just for MWA. But it is a valid question and indeed we do have an experimental S3 interface in NGAS as well and we are running NGAS on EC2. What's still pretty unique is the flexibility we have in setting up pretty complex data flows between several sites and even into and out-of HPC environments and also the quite powerful plugin mechanisms. Being totally independent of any vendor whatsoever is actually also quite nice and for that matter, we just know that it is working...