Theme: Delivering accessible and science-ready radio data
LOFAR data: From archive to arXiv
The LOFAR telescope has produced tens of petabytes stored on tape at three Long Term Archive locations. Typically, this data is staged and downloaded by individual researchers and processed on their local compute clusters. While this is a reasonable course of action for a few observations, processing tens, hundreds or even thousands of LOFAR observations is currently not possible at scale. We take advantage of the data locality at the Archive locations and provided compute infrastructure to create a comprehensive framework for batch and interactive processing of LOFAR data. Using this framework, it is easy to produce science ready images from thousands of observation. Moreover, we make it easy for researchers to build up and test their pipelines before deploying them in bulk. Our work makes large scale studies with LOFAR data easier to accomplish, as well as makes it easy to perform Integration tests of pipelines on real LOFAR data. Ultimately, we make it easier for LOFAR data to escape the archive and make its way to arXiv.
