P3.11 Oisin Creaner (Dublin Institute for Advanced Studies)
Theme: Data science challenges: tools from statistics to machine learning
I-LOFAR Data Analytics with Kx Systems
This poster presents initial results of a novel system which harnesses the power of Kx Systems to allow for rapid data ingestion and processing of astronomical data. Here, data is ingested into a kdb+ database with data structure optimised for rapid data ingestion and staging. This database is structured in a layered data format with regular checkpointing and archiving which is discussed in the poster. LOFAR is a radio telescope which receives signals from multiple directions simultaneously and which uses software systems to process these signals into usable data products. Applying the new system to I-LOFAR data analysis challenges show an improvement in data ingestion speeds by a factor of between 2 and 11. Detailed performance metrics are presented for both archival and streaming applications of this system Interactive dashboards have also been developed to carry out standard I-LOFAR analyses. The results of a series of user evaluations of alpha-test versions of this system are presented here and form the basis of the future evolution of this project.
