Lightning talk (L14) Karan Vahi (USC Information Sciences Institute)Theme: Multi-wavelength astronomy
(P10.39) Gearing the DECam Analysis Pipeline for Multi-Messenger Astronomy using Pegasus WorkflowsIn the era of multi-messenger astronomy it has become paramount to verify gravitational waves (GWs) discoveries with their electromagnetic (EM) optical counterparts in near realtime. With these goals in mind, we have developed the Dark Energy Camera (DECam) analysis pipeline that shifts through large amounts of data and tries to identify interesting optical candidates. The pipeline is setup to run nightly at NERSC supercomputers and compares current sky data against the historical data collected from the Victor Blanco telescope + DECam archived at NOAO or at NERSC, the latter as part of the DECam Legacy Survey.
Depending on the region of sky being analyzed, images may be needed to be downloaded onto the NERSC shared filesystem on demand from NOAO. This access pattern can result in requesting the same input datasets repeatedly, increasing the traffic on NOAO’s servers. In order to streamline access, we have archived much of the dataset (~500TB) on NERSC’s tape storage system, accessible via HPSS. Additionally the pipeline has been ported to Pegasus WMS to take advantage of advanced data management techniques and the ability to use additional computational resources on demand, such as the Open Science Grid or local clusters, when an alert is issued. Pegasus allows us to describe our pipeline in a resource agnostic manner, generates executable workflow with additional data management tasks that retrieve background FITS files from tape, automatically remove data no longer required and stage-out generated outputs. The scientific codes are packaged in a container executed using Shifter and deployed automatically.