Changelog

0.1.6 (2018-11-14)

  • Update URL for static example event.

0.1.5 (2018-11-13)

  • Add tasks for submitting HTCondor DAGs.
  • Add a new module, gwcelery.tasks.lalinference, which provides tasks to start parameter estimation with LALInference and upload the results to GraceDB.
  • Depend on lalsuite nightly build from 2018-11-04 to pick up changes to LALInference for Python 3 support.
  • Send static example VOEvents from the Open Alert Users Guide. This will provide a stream of example alerts for astronomers until GraceDb is ready for public access.
  • Add trials factor correction to the event FAR when comparing against FAR threshold to send out preliminary GCN.
  • Require that LIGO/Virgo VOEvents that we receive from GCN match the original VOEvents from GraceDb byte-for-byte, since GCN will now pass through our VOEvents without modification.

0.1.4 (2018-10-29)

  • Work around a bug in astropy.visualization.wcsaxes that affected all-sky plots when Matplotlib’s text.usetex rcparam is set to True (https://github.com/astropy/astropy/issues/8004). This bug has evidently been present since at least astropy 1.3, but was not being triggered until recently: it is likely that some other package that we import (e.g. lalsuite) is now globally setting text.usetex to True.
  • A try except is added around updateSuperevent to handle a bad request error from server side when updating superevent parameters which have nearby values.
  • Send automatic preliminary alerts only for events with a false alarm rate below a maximum value specified by a new configuration variable, preliminary_alert_far_threshold.
  • State vector vetoes will not suppress processing of preliminary sky maps and source classification. They will still suppress sending preliminary alerts.
  • Set open_alert to True for all automated VOEvents.

0.1.3 (2018-10-26)

  • Preliminary GCN is not sent for superevents created from offline gw events.
  • Add dqr_json function to gwcelery.tasks.detchar, which uploads a DQR-compatible json to GraceDb with the results of the detchar checks.
  • Depend on ligo.skymap >= 0.0.17.
  • Fix a bug in sending initial, update, and retraction GCN notices: we were sending the VOEvent filenames instead of the file contents.

0.1.2 (2018-10-11)

  • Setted vetted flag to true for all initial, update, and retraction alerts that are triggered by GraceDb signoffs.
  • Write GraceDb signoffs, instead of just labels, to simulate initial and retraction alerts for mock events, because merely creating the ADVNO or ADVOK label does not cause GraceDb to erase the ADVREQ label. This change makes mock alerts more realistic.
  • Change filename of cWB sky maps from skyprobcc_cWB.fits to cWB.fits.gz for consistency with other pipelines.
  • Any time that we send a VOEvent, first change the GraceDb permissions on the corresponding superevent so that it is visible to the public. Note that this has no effect during the ongoing software engineering runs because LVEM and unauthenticated access are currently disabled in GraceDb.

0.1.1 (2018-10-04)

  • Use the public tag instead of the lvem tag to mark preliminary sky maps for public access rather than LV-EM partner access. Note that GraceDb has not yet actually implemented unauthenticated access, so this should have no effect during our ongoing software engineering runs.
  • Add check_idq function to detchar module, which reads probabilities generated by iDQ.
  • Automated DQV labels should not trigger retraction notices because they prevent preliminary notices from being sent in the first place.
  • The criterion for selecting a superevent’s preferred event now prefers multiple-detector events to single-detector events, with precedence over source type (CBC versus burst). Any remaining tie is broken by using SNR for CBC and FAR for Burst triggers.
  • By default, initial and update alerts will find and send the most recently added public sky map.
  • The initial and update sky maps no longer perform sky map annotations, because they would only be duplicating the annotations performed as part of the preliminary alert.
  • Mock events now include example initial and retraction notices. Two minutes after each mock event is uploaded, there will be either an ADVOK or an ADVNO label applied at random, triggering either an initial or a retraction notice respectively.
  • Depend on ligo-gracedb >= 2.0.1 in order to pull in a bug fix for VOEvents with ProbHasNS or ProbHasRemnant set to 0.0.
  • Use the sentry-sdk package instead of the deprecated raven package for Sentry integration.

0.1.0 (2018-09-26)

  • Separated the external GCN listening handlers into two: one that listens to GCNs about SNEWS triggers and another that listens to Fermi and Swift.
  • Fixed calls to the raven temporal coincidence search so that search results separate SNEWS triggers from Fermi and Swift.
  • Add space-time FAR calculation for GRB and GW superevent coincidences. This only runs when skymaps from both triggers are available to download.
  • Add human vetting for initial GCN notices. For each new superevent that passes state vector checks, the ADVREQ label is applied. Rapid response team users should set their GraceDb notification preferences to alert them on ADVREQ labels. If a user sets the ADVOK label, then an initial notice is issued. If a user sets the ADVNO label, then a retraction notice is issued.
  • Update the LVAlert host for gracedb-playground.ligo.org.
  • Add experimental integration with Sentry for log aggregation and error reporting.
  • Track API and LVAlert schema changes in ligo-gracedb 2.0.0.

0.0.31 (2018-09-04)

  • Refactor external trigger handling to separate it from the orchestrator.
  • Fixed a bug in the VOEvent broker to only issue “iamalive” messages after sending the first VOEvent.
  • Pass group argument to set time windows appropriately when performing raven coincidence searches. Search in the [-600, 60]s range and [-5, 1]s range around external triggers for Burst events and CBC events respectively. Similarly, search in the [-60, 600]s and [-1, 5]s range around Burst and CBC events for external triggers.
  • Compute and upload FAR for GRB external trigger/superevent coincidence upon receipt of the EM_COINC label application to a superevent.
  • Add continuous integration testing for Python 3.7, and run test suite against all supported Python versions (3.6, 3.7).
  • Update ligo.skymap to 0.0.15.

0.0.30 (2018-08-02)

  • Manage superevents for production, test, and MDC events separately.
  • Add some more validation of LIGO/Virgo VOEvents from GCN.
  • Remove now-unused task gwcelery.tasks.orchestartor.continue_if.
  • Add check_vectors run for external triggers.
  • Change the preferred event selection criteria for burst events to be FAR instead of SNR.
  • Add gwcelery nagios subcommand for Nagios monitoring.
  • Incorporate Virgo DQ veto streams into check_vectors
  • Update ligo-raven to 1.3 and ligo-followup-advocate to 0.0.11.

0.0.29 (2018-07-31)

  • Add a workflow graph to superevents module documentation.
  • Add gwcelery condor resubmit as a shortcut for gwcelery condor rm; gwcelery condor submit.
  • Fix deprecation warning due to renaming of ligo.gracedb.rest.Gracedb.createTag to ligo.gracedb.rest.Gracedb.addTag.
  • Update ligo-gracedb to 2.0.0.dev1.

0.0.28 (2018-07-25)

  • Add injection checks to check_vector.
  • Bitmasks are now defined symbolically in detchar.
  • Refactor configuration so that it is possible to customize settings through an environment variable.

0.0.27 (2018-07-22)

  • The preferred event for superevents is now decided based on higher SNR value instead of lower FAR in the case of a tie between groups.
  • A check for the existence of the gstlal trigger database is performed so that compute_p_astro does not return None.

0.0.26 (2018-07-20)

  • Fix spelling of the label that is applied to events after p_astro finishes, changed from P_ASTRO_READY to PASTRO_READY.

  • Run p_astro calculation for mock events.

  • Overhaul preliminary alert pipeline so that it is mostly feature complete for both CBC and Burst events, and uses a common code path for both types. Sky map annotations now occur for both CBC and Burst localizations.

  • Switch to using the pre-registered port 8096 for receiving proprietary LIGO/Virgo alerts on emfollow.ligo.caltech.edu. This means that the capability to receive GCNs requires setting up a site configuration in advance with Scott Barthelmey.

    Once we switch to sending public alerts exclusively, then we can switch back to using port 8099 for anonymous access, requiring no prior site configuration.

0.0.25 (2018-07-19)

  • Reintroduce pipeline-dependent pre/post peeks for check_vector after fixing issue where pipeline information was being looked for in the wrong dictionary.
  • check_vector checks all detectors regardless of instruments used, but only appends labels based on active instruments.
  • Fix a few issues in the GCN broker:
    • Decrease the frequency of keepalive (“iamalive” in VOEvent Transport Protocol parlance) packets from once a second to once a minute at the request of Scott Barthelmey.
    • Fix a possible race condition that might have caused queued VOEvents to be thrown away unsent shortly after a scheduled keepalive packet.
    • Consume and ignore all keepalive and ack packets from the client so that the receive buffer does not overrun.
  • Add p_astro computation for gstlal pipeline. The copmutation is launched for all cbc_gstlal triggers.

0.0.24 (2018-07-18)

  • Revert pipeline-dependent pre/post peeks for check_vector because they introduced a regression: it caused the orchestrator failed without running any annotations.

0.0.23 (2018-07-18)

  • Add timeout and keepalive messages to GCN broker.
  • Update ligo-gracedb to 2.0.0.dev0 and ligo.skymap to 0.0.12.
  • Add superevent duration for gstlal-spiir pipeline.
  • Fix fallback for determining superevent duration for unknown pipelines.
  • Make check_vector pre/post peeks pipeline dependent.

0.0.22 (2018-07-11)

  • Process gstlal-spiir events.

  • Create combined LVC-Fermi skymap in case of coincident triggers and upload to GraceDb superevent page. Also upload the original external trigger sky map to the external trigger GraceDb page.

  • Generalize conditional processing of complex canvases by replacing the continue_if_group_is() task with a more general task that can be used like continue_if(group='CBC').

  • Add a check_vector_prepost configuration variable to control how much padding is added around an event for querying the state vector time series.

    This should have the beneficial side effect of fixing some crashes for burst events, for which the bare duration of the superevent segment was less than one sample.

0.0.21 (2018-07-10)

  • MBTA events in GraceDb leave the search field blank. Work around this in gwcelery.tasks.detchar.check_vectors where we expected the field to be present.
  • Track change in GraceDb JSON response for VOEvent creation.

0.0.20 (2018-07-09)

  • After fixing some minor bugs in code that had not yet been tested live, sending VOEvents to GCN now works.

0.0.19 (2018-07-09)

  • Rewrite the GCN broker so that it does not require a dedicated worker.
  • Send VOEvents for preliminary alerts to GCN.
  • Only perform state vector checks for detectors that were online, according to the preferred event.
  • Exclude mock data challenge events from state vector checks.

0.0.18 (2018-07-06)

  • Add detector state vector checks to the preliminary alert workflow.

0.0.17 (2018-07-05)

  • Undo accidental configuration change in last version.

0.0.16 (2018-07-05)

  • Stop listening for three unnecessary GCN notice types: SWIFT_BAT_ALARM_LONG, SWIFT_BAT_ALARM_SHORT, and SWIFT_BAT_KNOWN_SRC.
  • Switch to SleekXMPP for the LVAlert client, instead of PyXMPP2. Because SleekXMPP has first-class support for publish-subscribe, the LVAlert listener can now automatically subscribe to all LVAlert nodes for which our code has handlers. Most of the client code now lives in a new external package, sleek-lvalert.

0.0.15 (2018-06-29)

  • Change superevent threshold and mock event rate to once per hour.
  • Add gracedb.create_label task.
  • Always upload external triggers to the ‘External’ group.
  • Add rudimentary burst event workflow to orchestrator: it just generates VOEvents and circulars.
  • Create a label in GraceDb whenever em_bright or bayestar completes.

0.0.14 (2018-06-28)

  • Fix typo that was causing a task to fail.
  • Decrease orchestrator timeout to 15 seconds.

0.0.13 (2018-06-28)

  • Change FAR threshold for creation of superevents to 1 per day.
  • Update ligo-followup-advocate to >= 0.0.10. Re-enable automatic generation of GCN circulars.
  • Add “EM bright” classification. This is rudimentary and based only on the point mass estimates from the search pipeline because some of the EM bright classifier’s dependencies are not yet ready for Python 3.
  • Added logic to select CBC events as preferred event over Burst. FAR acts as tie breaker when groups for preferred event and new event match.
  • BAYESTAR now adds GraceDb URLs of events to FITS headers.

0.0.12 (2018-06-28)

  • Prevent receiving duplicate copies of LVAlert messages by unregistering redundant LVAlert message types.
  • Update to ligo-followup-advocate >= 0.0.9 to update GCN Circular text for superevents. Unfortunately, circulars are still disabled due to a regression in ligo-gracedb (see https://git.ligo.org/lscsoft/gracedb-client/issues/7).
  • Upload BAYESTAR sky maps and annotations to superevents.
  • Create (but do not send) preliminary VOEvents for all superevents. No vetting is performed yet.

0.0.11 (2018-06-27)

  • Submit handler tasks to Celery as a single group.

  • Retry GraceDb tasks that raise a TimeoutError exception.

  • The superevent handler now skips LVAlert messages that do not affect the false alarm rate of an event (e.g. simple log messages).

    (Note that the false alarm rate in GraceDb is set by the initial event upload and can be updated by replacing the event; however replacing the event does not produce an LVAlert message at all, so there is no way to intercept it.)

  • Added a query kwarg to superevents method to reduce latency in fetching the superevents from gracedb.

  • Refactored getting event information for update type events so that gracedb is polled only once to get the information needed for superevent manager.

  • Renamed the set_preferred_event task in gracedb.py to update_superevent to be a full wrapper around the updateSuperevent client function. Now it can be used to set preferred event and also update superevent time windows.

  • Many cwb (extra) attributes, which should be floating point numbers, are present in lvalert packet as strings. Casting them to avoid embarassing TypeErrors.

  • Reverted back the typecasting of far, gpstime into float. This is fixed in https://git.ligo.org/lscsoft/gracedb/issues/10

  • CBC t_start and t_end values are changed to 1 sec interval.

  • Added ligo-raven to run on external trigger and superevent creation lvalerts to search for coincidences. In case of coincidence, EM_COINC label is applied to the superevent and external trigger page and the external trigger is added to the list of em_events in superevent object dictionary.

  • cwb and lib nodes added to superevent handler.

  • Events are treated as finite segment window, initial superevent creation with preferred event window. Addition of events to superevents may change the superevent window and also the preferred event.

  • Change default GraceDb server to https://gracedb-playground.ligo.org/ for open public alert challenge.

  • Update to ligo-gracedb >= 1.29dev1.

  • Rename the get_superevent task to get_superevents and add a new get_superevent task that is a trivial wrapper around ligo.gracedb.rest.GraceDb.superevent().

0.0.10 (2018-06-13)

  • Model the time extent of events and superevents using the glue.segments module.
  • Replace GraceDb.get with GraceDb.superevents from the recent dev release of gracedb-client.
  • Fix possible false positive matches between GCNs for unrelated GRBs by matching on both TrigID (which is generally the mission elapsed time) and mission name.
  • Add the configuration variable superevent_far_threshold to limit the maximum false alarm rate of events that are included in superevents.
  • LVAlert handlers are now passed the actual alert data structure rather than the JSON text, so handlers are no longer responsible for calling json.loads. It is a little bit more convenient and possibly also faster for Celery to deserialize the alert messages.
  • Introduce Production, Development, Test, and Playground application configuration objects in order to facilitate quickly switching between GraceDb servers.
  • Pipeline specific start and end times for superevent segments. These values are controlled via configuration variables.

0.0.9 (2018-06-06)

  • Add missing LVAlert message types to superevent handler.

0.0.8 (2018-06-06)

  • Add some logging to the GCN and LVAlert dispatch code in order to diagnose missed messages.

0.0.7 (2018-05-31)

  • Ingest Swift, Fermi, and SNEWS GCN notices and save them in GraceDb.
  • Depend on the pre-release version of the GraceDb client, ligo-gracedb 1.29.dev0, because this is the only version that supports superevents at the moment.

0.0.6 (2018-05-26)

  • Generate GCN Circular drafts using ligo-followup-advocate.

  • In the continuous integration pipeline, validate PEP8 naming conventions using pep8-naming.

  • Add instructions for measuring test coverage and running the linter locally to the contributing guide.

  • Rename gwcelery.tasks.voevent to gwcelery.tasks.gcn to make it clear that this submodule contains functionality related to GCN notices, rather than VOEvents in general.

  • Rename gwcelery.tasks.dispatch to gwcelery.tasks.orchestrator to make it clear that this module encapsulates the behavior associated with the “orchestrator” in the O3 low-latency design document.

  • Mock up calls to BAYESTAR in test suite to speed it up.

  • Unify dispatch of LVAlert and GCN messages using decorators. GCN notice handlers are declared like this:

    import lxml.etree
from gwcelery.tasks import gcn

@gcn.handler(gcn.NoticeType.FERMI_GBM_GND_POS,
             gcn.NoticeType.FERMI_GBM_FIN_POS)
def handle_fermi(payload):
    root = lxml.etree.fromstring(payload)
    # do work here...

    LVAlert message handlers are declared like this:

    import json
from gwcelery.tasks import lvalert

@lvalert.handler('cbc_gstlal',
                 'cbc_pycbc',
                 'cbc_mbta')
def handle_cbc(alert_content):
    alert = json.loads(alert_content)
    # do work here...

  • Instead of carrying around the GraceDb service URL in tasks, store the GraceDb host name in the Celery application config.

  • Create superevents by simple clustering in time. Currently this is only supported by the gracedb-dev1 host.

0.0.5 (2018-05-08)

  • Disable socket access during most unit tests. This adds some extra assurance that we don’t accidentally interact with production servers during the unit tests.
  • Ignore BAYESTAR jobs that raise a DetectorDisabled error. These exceptions are used for control flow and do not constitute a real error. Ignoring these jobs avoids polluting logs and the Flower monitor.

0.0.4 (2018-04-28)

  • FITS history and comment entries are now displayed in a monospaced font.
  • Adjust error reporting for some tasks.
  • Depend on newer version of ligo.skymap.
  • Add unit tests for the gwcelery condor submit subcommand.

0.0.3 (2018-04-27)

  • Fix some compatibility issues between the gwcelery condor submit subcommand and the format of condor_q -totals -xml with older versions of HTCondor.

0.0.2 (2018-04-27)

  • Add gwcelery condor submit and related subcommands as shortcuts for managing GWCelery running under HTCondor.

0.0.1 (2018-04-27)

  • This is the initial release. It provides rapid sky localization with BAYESTAR, sky map annotation, and sending mock alerts.
  • By default, GWCelery is configured to listen to the test LVAlert server.
  • Sending VOEvents to GCN/TAN is disabled for now.