Design and anatomy of GWCelery

Conceptual overview

Several online gravitational-wave transient search pipelines (currently Gstlal, PyCBC, cWB, and oLIB) upload candidates in real time to GraceDB, the central database and web portal for low-latency LIGO/Virgo analyses. Whenever an event is uploaded or altered, GraceDB pushes machine-readable notifications through IGWN Alert, a pubsub system based on Kafka.

The business logic for selecting and sending alerts to astronomers resides not in GraceDB itself but in GWCelery. The role of GWCelery in the LIGO/Virgo alert infrastructure is to drive the workflow of aggregating and annotating gravitational-wave candidates and sending GCN Notices to astronomers.

GWCelery interacts with GraceDB by listening for IGWN Alert messages and making REST API requests through the GraceDB client. GWCelery interacts with GCN by listening for and sending GCN Notices using the Comet VOEvent broker.

The major subsystems of GWCelery are:

  • the IGWN Alert listener

  • the GraceDB client

  • the GCN listener

  • the GCN broker

  • the Superevent Manager, which clusters and merges related candidates into “superevents”

  • the External Trigger Manager, which correlates gravitational-wave events with GRB, neutrino, and supernova events

  • the Orchestrator, which executes the per-event annotation workflow

Block diagram

Below is a diagram illustrating the conceptual relationships of these subsystems. Nodes in the graph are hyperlinks to the relevant API documentation.

digraph superevents { compound = true splines = ortho node [ fillcolor = white shape = box style = filled target = "_top" ] graph [ labeljust = "left" style = filled target = "_top" ] gracedb [ label = "GraceDB" ] igwn_alert [ label = "IGWN Alert" ] { rank = source gstlal [ label = "Gstlal\nSearch" ] pycbc [ label = "PyCBC\nSearch" ] cwb [ label = "cWB\nSearch" ] olib [ label = "oLIB\nSearch" ] } subgraph cluster_gwcelery { label = "GWCelery" { rank = same igwn_alert_listener [ href = "../gwcelery.tasks.igwn_alert.html" label = "IGWN Alert\nListener" ] superevent_manager [ href = "../gwcelery.tasks.superevents.html" label = "Superevent\nManager" ] gracedb_client [ href = "../gwcelery.tasks.gracedb.html" label = "GraceDB\nClient" ] } raven [ href = "../gwcelery.tasks.external_triggers.html" label = "External\nTrigger\nManager" ] subgraph cluster_orchestrator { href = "../gwcelery.tasks.orchestrator.html" label = "Orchestrator" { rank = same detchar [ href = "../gwcelery.tasks.detchar.html" label = "Detchar" ] bayestar [ href = "../gwcelery.tasks.bayestar.html" label = "BAYESTAR" ] inference [ href = "../gwcelery.tasks.inference.html" label = "Parameter Estimation" ] } { rank = same skymaps [ href = "../gwcelery.tasks.skymaps.html" label = "Sky Map\nVisualization" ] classification [ label = "Source\nClassification" ] circulars [ href = "../gwcelery.tasks.circulars.html" label = "Circular\nTemplates" ] } } { rank = same gcn_listener [ href = "../gwcelery.tasks.gcn.html" label = "GCN\nListener" ] gcn_broker [ html = "gwcelery.tasks.gcn.html" label = "GCN\nBroker" ] } } gcn [ label = "GCN" ] { rank = sink astronomers [ label = "Astronomers" ] } gstlal -> gracedb pycbc -> gracedb cwb -> gracedb olib -> gracedb gracedb -> igwn_alert igwn_alert -> igwn_alert_listener gracedb -> gracedb_client [dir=back] igwn_alert_listener -> superevent_manager igwn_alert_listener -> detchar [lhead=cluster_orchestrator] igwn_alert_listener -> raven superevent_manager -> gracedb_client inference -> gracedb_client [ltail=cluster_orchestrator] raven -> gracedb_client detchar -> bayestar [style=invis] bayestar -> inference [style=invis] detchar -> skymaps [style=invis] bayestar -> classification [style=invis] inference -> circulars [style=invis] skymaps -> classification [style=invis] classification -> circulars [style=invis] classification -> gcn_broker [ltail=cluster_orchestrator] classification -> gcn_listener [dir=back, ltail=cluster_orchestrator] superevent_manager -> raven [style=invis] raven -> detchar [style=invis] raven -> bayestar [style=invis] raven -> inference [style=invis] gcn_listener -> gcn [dir=back] gcn_broker -> gcn gcn -> astronomers gcn -> astronomers [dir=back] }

Processes

A complete deployment of GWCelery (whether launched from the shell or from HTCondor) consists of several processes:

  1. Message Broker

    Routes and distributes Celery task messages and stores results of tasks for later retrieval. See Choosing a Broker in the Celery manual for more details. For technical reasons, we use a Redis broker.

  2. Celery Beat

    Scheduler for periodic tasks (the Celery equivalent of cron jobs). For more information, see Periodic Tasks in the Celery manual.

  3. Monitoring Console (optional)

    You can optionally run Flower, a web monitoring console for Celery.

  4. OpenMP Worker

    A Celery worker that has been configured to accept only computationally intensive tasks that use OpenMP parallelism. To route a task to the OpenMP worker, pass the keyword argument queue='openmp' to the @app.task decorator when you declare it.

    There are two tasks that run in the OpenMP queue:

  5. Multiprocessing Worker

    A Celery worker that has been configured to accept only computationally intensive tasks that use Python python:multiprocessing parallelism. To route a task to the multiprocessing worker, pass the keyword argument queue='multiprocessing' to the @app.task decorator when you declare it.

    There is one task that run in the multiprocessing queue:

  6. Superevent Worker

    A Celery worker that is dedicated to serially process triggers from low latency pipelines and create/modify superevents in GraceDB. There is only one task that runs on the Superevent queue:

  7. External Trigger Worker

    A Celery worker that is dedicated to serially process external triggers from GRB alerts received from Fermi, Swift, Integral and neutrino alerts received from SNEWS and create/modify external trigger events in GraceDB:

    • gwcelery.tasks.external_triggers.handle_gcn()

  8. VOEvent Worker

    A Celery worker that is dedicated to sending and receiving VOEvents. It runs an embedded instance of the Comet VOEvent broker, which is started and stopped using a set of custom Celery bootsteps. Note that the VOEvent worker must be started with the --pool=solo option so that tasks are executed in the same Python process that is running the VOEvent broker.

  9. EM-Bright Worker

    A Celery worker that is dedicated to computing source properties of compact binary coalescences.

  10. General-Purpose Worker

    A Celery worker that accepts all other tasks. This worker also runs an embedded IGWN Alert listener service that is started and stopped as a bootstep.

  1. Flask Web Application

    A web application that provides forms to manually initiate certain tasks, including sending an update alert or creating a mock event.

Handlers

A recurring pattern in GWCelery is that an eternal task listens continuously to a remote connection, receives packets of data over that connection, and dispatches further handling to other tasks based on packet type.

A decorator is provided to register a function as a Celery task and also plug it in as a handler for one or more packet types. This pattern is used for both GCN notices and IGWN Alert message handlers.

GCN notices

GCN notice handler tasks are declared using the gwcelery.tasks.gcn.handler() decorator:

import lxml.etree
from gwcelery.tasks import gcn

@gcn.handler(gcn.NoticeType.FERMI_GBM_GND_POS,
             gcn.NoticeType.FERMI_GBM_FIN_POS)
def handle_fermi(payload):
    root = lxml.etree.fromstring(payload)
    # do work here...

IGWN Alert messages

IGWN Alert message handler tasks are declared using the gwcelery.tasks.igwn_alert.handler() decorator:

from gwcelery.tasks import igwn_alert

@igwn_alert.handler('cbc_gstlal',
                    'cbc_spiir',
                    'cbc_pycbc',
                    'cbc_mbta')
def handle_cbc(alert):
    # do work here...