First machine learning gravitational-wave search mock data challenge

Open Access

First machine learning gravitational-wave search mock data challenge

Marlin B. Schäfer, Ondřej Zelenka, Alexander H. Nitz, He Wang, Shichao Wu, Zong-Kuan Guo, Zhoujian Cao, Zhixiang Ren, Paraskevi Nousi, Nikolaos Stergioulas, Panagiotis Iosif, Alexandra E. Koloniari, Anastasios Tefas, Nikolaos Passalis, Francesco Salemi, Gabriele Vedovato, Sergey Klimenko, Tanmaya Mishra, Bernd Brügmann, Elena Cuoco, E. A. Huerta, Chris Messenger, and Frank Ohme

Phys. Rev. D 107, 023021 – Published 27 January 2023

Abstract

We present the results of the first Machine Learning Gravitational-Wave Search Mock Data Challenge. For this challenge, participating groups had to identify gravitational-wave signals from binary black hole mergers of increasing complexity and duration embedded in progressively more realistic noise. The final of the 4 provided datasets contained real noise from the O3a observing run and signals up to a duration of 20 s with the inclusion of precession effects and higher order modes. We present the average sensitivity distance and run-time for the 6 entered algorithms derived from 1 month of test data unknown to the participants prior to submission. Of these, 4 are machine learning algorithms. We find that the best machine learning based algorithms are able to achieve up to 95% of the sensitive distance of matched-filtering based production analyses for simulated Gaussian noise at a false-alarm rate (FAR) of one per month. In contrast, for real noise, the leading machine learning search achieved 70%. For higher FARs the differences in sensitive distance shrink to the point where select machine learning submissions outperform traditional search algorithms at FARs $\geq 200$ per month on some datasets. Our results show that current machine learning search algorithms may already be sensitive enough in limited parameter regions to be useful for some production settings. To improve the state-of-the-art, machine learning algorithms need to reduce the false-alarm rates at which they are capable of detecting signals and extend their validity to regions of parameter space where modeled searches are computationally expensive to run. Based on our findings we compile a list of research areas that we believe are the most important to elevate machine learning searches to an invaluable tool in gravitational-wave signal detection.

Received 23 September 2022
Accepted 28 November 2022

DOI:https://doi.org/10.1103/PhysRevD.107.023021

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Open access publication funded by the Max Planck Society.

Published by the American Physical Society

Physics Subject Headings (PhySH)

Gravitational wave detection Gravitational waves

Machine learning

Gravitation, Cosmology & Astrophysics

Authors & Affiliations

Marlin B. Schäfer ^1,2, Ondřej Zelenka ^3,4, Alexander H. Nitz ^1,2, He Wang ⁵, Shichao Wu^1,2, Zong-Kuan Guo ⁵, Zhoujian Cao ⁶, Zhixiang Ren ⁷, Paraskevi Nousi ⁸, Nikolaos Stergioulas ⁹, Panagiotis Iosif ^10,9, Alexandra E. Koloniari ⁹, Anastasios Tefas⁸, Nikolaos Passalis⁸, Francesco Salemi ^11,12, Gabriele Vedovato ¹³, Sergey Klimenko¹⁴, Tanmaya Mishra ¹⁴, Bernd Brügmann ^3,4, Elena Cuoco ^15,16,17, E. A. Huerta ^18,19, Chris Messenger²⁰, and Frank Ohme ^1,2

¹Max-Planck-Institut für Gravitationsphysik, Albert-Einstein-Institut, D-30167 Hannover, Germany
²Leibniz Universität Hannover, D-30167 Hannover, Germany
³Friedrich-Schiller-Universität Jena, D-07743 Jena, Germany
⁴Michael Stifel Center Jena, D-07743 Jena, Germany
⁵CAS Key Laboratory of Theoretical Physics, Institute of Theoretical Physics, Chinese Academy of Sciences, Beijing 100190, China
⁶Department of Astronomy, Beijing Normal University, Beijing 100875, China
⁷Peng Cheng Laboratory, Shenzhen, 518055, China
⁸Department of Informatics, Aristotle University of Thessaloniki, GR-54124 Thessaloniki, Greece
⁹Department of Physics, Aristotle University of Thessaloniki, GR-54124 Thessaloniki, Greece
¹⁰GSI Helmholtz Center for Heavy Ion Research, Planckstraße 1, 64291 Darmstadt, Germany
¹¹Università di Trento, Dipartimento di Fisica, I-38123 Povo, Trento, Italy
¹²INFN, Trento Institute for Fundamental Physics and Applications, I-38123 Povo, Trento, Italy
¹³INFN, Sezione di Padova, I-35131 Padova, Italy
¹⁴Department of Physics, University of Florida, PO Box 118440, Gainesville, Florida 32611-8440, USA
¹⁵European Gravitational Observatory (EGO), I-56021 Cascina, Pisa, Italy
¹⁶Scuola Normale Superiore, Piazza dei Cavalieri 7, I-56126 Pisa, Italy
¹⁷INFN, Sezione di Pisa, Largo Bruno Pontecorvo, 3, I-56127 Pisa, Italy
¹⁸Data Science and Learning Division, Argonne National Laboratory, Lemont, Illinois 60439, USA
¹⁹Department of Computer Science, University of Chicago, Chicago, Illinois 60637, USA
²⁰SUPA, School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, United Kingdom

Article Text

Click to Expand

References

Click to Expand

Issue

Vol. 107, Iss. 2 — 15 January 2023

Reuse & Permissions

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
An illustration of the range for the intrinsic parameters covered by this challenge. The left panel (a) shows a typical range for the component masses used by state-of-the-art searches [14]. The color indicates the duration of the waveform from 20 Hz. The triangles show the parameter regions covered by this challenge. The right panel (b) shows the component-spin $χ_{i}$ distribution of the different datasets in this challenge.
Reuse & Permissions
Figure 2
The sensitive distances of all submissions and all four datasets as functions of the FAR. Submissions that made use of a machine learning algorithm at their core are shown with solid lines, others with dashed lines. The FAR was calculated on a background set that does not contain any injections.
Reuse & Permissions
Figure 3
The sensitive distances of all submissions and all four datasets as functions of the FAR. The sensitive distances are calculated using only the data from the foreground file. The FAR is determined from the false positives on that data. Submissions that made use of a machine learning algorithm at their core are shown with solid lines, others with dashed lines. This figure differs from Fig. 2 as the algorithms from MFCNN and CNN-Coinc behave differently on the foreground and the background.
Reuse & Permissions
Figure 4
The injections from dataset 1 identified by the PyCBC and TPI FSU Jena submissions with a $FAR \leq 10^{3}$ per month in the chirp-mass $M_{c}$ versus decisive effective chirp-distance $D_{c, eff}$ plane. The blue bars in the histograms show the one-dimensional marginal distributions of the found injections. The gray bars show the distribution of injected signals, including those missed by the search. The color shows the “stat” value attributed to the injection by the algorithm. The red lines in the color bar highlight the thresholds on the “stat” to achieve different FARs.
Reuse & Permissions
Figure 5
The injections from dataset 2 identified by the PyCBC and TPI FSU Jena submissions with a $FAR \leq 10^{3}$ per month in the signal duration $τ_{0}$ versus mass ratio $q$ plane. The scatter plot shows injections that are found only by one of the two algorithms. Injections that are missed or found by both are only shown in the 1D marginal distributions.
Reuse & Permissions
Figure 6
The injections from dataset 3 identified by the PyCBC and TPI FSU Jena submissions with a $FAR \leq 10^{3}$ per month in the inclination to spin axis $θ_{j n}$ to $χ_{p}$ plane. The scatter plot shows injections that are found only by one of the two algorithms. Injections that are missed or found by both are only shown in the 1D marginal distributions.
Reuse & Permissions

Physical Review D

covering particles, fields, gravitation, and cosmology