The Low Frequency Array (LOFAR) observes the low-frequency radio sky with unprecedented resolution and sensitivity. With this phenomenal increase in imaging the fine details of our Universe we have entered a new era of big data in Astronomy. At the same time, however, this comes with the need to handle the tens Petabyte-sized LOFAR data collections. Such collections require new e-Science solutions, on a Dutch and on a European scale.

To respond to this data challenge we founded the LOFAR e-infra group in 2015. Originally a collaboration between Leiden Observatory and the Dutch national e-infrastructure SURFsara we now also work with the Netherlands Foundation for Radio Astronomy (ASTRON), the GLOW consortium and the Poznan Supercomputing and Network Center.

In the LOFAR e-infra group, we are responsible for developing, testing, and deploying LOFAR workflows for Petabyte-scale data. Leveraging the Grid processing facility at SURFsara, a batch cluster capable of the required high-throughput data processing rate, we parallelise the calibration & imaging workflow of LOFAR. We further automate our processing using leading workflow, whilst continuously tracking and optimising its performance.  

Taking advantage of close coupling between the Grid processing cluster and SURFsara's Grid storage facility, that is one of the storage locations in the LOFAR long term archive (LTA), we are able to process an archived LOFAR dataset within hours of its arrival in the LTA. This has allowed us to serve ready made pipelines and science quality products to LOFAR users in a timely manner.

In addition to the SURFsara Grid facility, we have begun porting our processing solution to the other two LOFAR LTA sites, located at the Forschungszentrum Juelich (Germany) and the Poznan Supercomputing and Networking Center (Poland). In Juelich, together with the Glow team, we now routinely process LOFAR data and in Poznan the implementation is well underway.

Our e-Science solutions are developed in close collaboration with LOFAR scientists and user communities, such as the LOFAR Surveys key science project and the radio recombination line group that together are comprised of more than 200 active scientists. Our work has contributed to over 20 scientific publications since 2017, and the number of our users is continuing to grow.

Publications and software repositories

Publications

https://arxiv.org/abs/1808.10735 : Workflow manager used to automate arbitrary pipelines on a distributed architecture
https://arxiv.org/abs/1712.00312 : Portable method for scaling astronomical processing across a distributed architecture

Software

https://github.com/apmechev/GRID_LRT : Software tools to build, deploy and manage LOFAR jobs to a distributed architecture
https://github.com/apmechev/GRID_PiCaS_Launcher : Launcher used to execute self-contained jobs on a managed infrastructure
https://github.com/apmechev/AGLOW : Upgrade of Apache Airflow to handle radio astronomy data and automate LOFAR workflows