Design and anatomy of GWCelery

Conceptual overview

Several online gravitational-wave transient search pipelines (currently Gstlal, PyCBC, MBTA, CWB, and Mly) upload candidates in real time to GraceDB, the central database and web portal for low-latency LIGO/Virgo analyses. Whenever an event is uploaded or altered, GraceDB pushes machine-readable notifications through IGWN Alert, a pubsub system based on Kafka.

The business logic for selecting and sending alerts to astronomers resides not in GraceDB itself but in GWCelery. The role of GWCelery in the LIGO/Virgo alert infrastructure is to drive the workflow of aggregating and annotating gravitational-wave candidates and sending public notices to astronomers via Kafka.

GWCelery interacts with GraceDB by listening for IGWN Alert messages and making REST API requests through the GraceDB client. GWCelery sends public alerts to astronomers and receives external trigger alerts (Fermi, Swift, SNEWS) via Kafka.

The major subsystems of GWCelery are:

• the IGWN Alert listener

• the GraceDB client

• the Superevent Manager, which clusters and merges related candidates into “superevents”

• the External Trigger Manager, which correlates gravitational-wave events with GRB, neutrino, and supernova events

• the Orchestrator, which executes the per-event annotation workflow

Block diagram

Below is a diagram illustrating the conceptual relationships of these subsystems. Nodes in the graph are hyperlinks to the relevant API documentation.

Processes

A complete deployment of GWCelery (whether launched from the shell or from HTCondor) consists of several processes:

1. Message Broker

   Routes and distributes Celery task messages and stores results of tasks for later retrieval. See Choosing a Broker in the Celery manual for more details. For technical reasons, we use a Redis broker.

2. Celery Beat

   Scheduler for periodic tasks (the Celery equivalent of cron jobs). For more information, see Periodic Tasks in the Celery manual.

3. Monitoring Console (optional)

   You can optionally run Flower, a web monitoring console for Celery.

4. OpenMP Worker

   A Celery worker that has been configured to accept only computationally intensive tasks that use OpenMP parallelism. To route a task to the OpenMP worker, pass the keyword argument queue='openmp' to the @app.task decorator when you declare it.

   There are two tasks that run in the OpenMP queue:

   • gwcelery.tasks.bayestar.localize()

   • gwcelery.tasks.skymaps.plot_volume()

5. Multiprocessing Worker

   A Celery worker that has been configured to accept only computationally intensive tasks that use Python python:multiprocessing parallelism. To route a task to the multiprocessing worker, pass the keyword argument queue='multiprocessing' to the @app.task decorator when you declare it.

   There is one task that run in the multiprocessing queue:

   • gwcelery.tasks.skymaps.skymap_from_samples()

6. Superevent Worker

   A Celery worker that is dedicated to serially process triggers from low latency pipelines and create/modify superevents in GraceDB. There is only one task that runs on the Superevent queue:

   • gwcelery.tasks.superevents.handle()

7. External Trigger Worker

   A Celery worker that is dedicated to serially process external triggers from GRB alerts received from Fermi, Swift, SVOM and neutrino alerts received from SNEWS and create/modify external trigger events in GraceDB.

8. EM-Bright Worker

   A Celery worker that is dedicated to computing source properties of compact binary coalescences.

9. High Memory Worker

   A Celery worker with low concurrency that is dedicated to running tasks that use a large amount of memory.

10. General-Purpose Worker

    A Celery worker that accepts all other tasks. This worker also runs an embedded IGWN Alert listener service that is started and stopped as a bootstep.

11. Flask Web Application

    A web application that provides forms to manually initiate certain tasks, including sending an update alert or creating a mock event.

Handlers

A recurring pattern in GWCelery is that an eternal task listens continuously to a remote connection, receives packets of data over that connection, and dispatches further handling to other tasks based on packet type.

A decorator is provided to register a function as a Celery task and also plug it in as a handler for one or more packet types.

IGWN Alert messages

IGWN Alert message handler tasks are declared using the gwcelery.tasks.igwn_alert.handler() decorator:

from gwcelery.tasks import igwn_alert

@igwn_alert.handler('cbc_gstlal',
                    'cbc_spiir',
                    'cbc_pycbc',
                    'cbc_mbta',
                    'cbc_sgnl')
def handle_cbc(alert):
    # do work here...