Poster Abstract

P.5 Vishambhar Nath Pandey (ASTRON / Kapteyn Astronomical Institute)

HPC and Big Data Management System for the LOFAR EoR Project

Successful detection of the EoR signal with LOFAR requires overcoming challenges on several fronts from carrying out more than thousand hours of observations (data volumes exceeding several PBs), faint RFI detection and excision, accurate source model construction (including foregrounds), multi-directional full polarization calibration (including ionospheric corrections), deep imaging (high dynamic range), source subtraction, power spectrum to astrophysical interpretation of the results.
In this paper we describe our High Performance Computing (HPC) and Data management system which is essential to be able to efficiently carry on such big projects. In addition to the regular optimised bulk processing (to detect the EoR signal) via pipelines, the HPC system also handles the intake of new observations, their efficient storage on its own online backup storage (~60km away from compute systems) and transfer of required data (via high speed public internet) from the LOFAR Long Term Archive (LTA) centres spread across Europe.
We also discuss the salient characteristics of the HPC system (with high speed interconnect, GPU and Xeon-Phi accelerators) including its efficient, modular and future proof design. The future proof design and maintenance is also needed as such projects often overrun the typical hardware life time of an HPC cluster.