Configuration

Like any Celery application, GWCelery’s configuration options are stored at run time in a global configuration object, gwcelery.app.conf. There are options for Celery itself such as options that affect the task and result backends; these options are documented in the Configuration and defaults section of the Celery manual.

The configuration object also holds all of the options that are specific to GWCelery and affect the behavior of individual GWCelery tasks; examples include the GraceDB service URLs, IGWN Alert groups, GCN hostnames, and frame file types and channel names. For a list of all GWCelery-specific options, see the API documentation for the gwcelery.conf module.

GWCelery provides five preset configurations, one for each GraceDB server instance (production, deployment, testing, minikube or playground). The default configuration preset is for the playground server, gracedb-playground.ligo.org. The recommended way to select a different preset is to set the CELERY_CONFIG_MODULE environment variable before starting the workers. For example, to configure GWCelery for production:

$ export CELERY_CONFIG_MODULE=gwcelery.conf.production

Authentication

There are a few files that must be present in order to provide authentication tokens for GraceDB and IGWN Alert.

IGWN Alert

You must provide a valid username and password for IGWN Alert. You can request an account using the SCiMMA Auth portal. To get started, see IGWN Alert Userguide. The IGWN Alert username and password should be stored in your auth.toml file.

Kafka

You must provide a file named kafka_credential_map.json that maps deployment specific usernames for Kafka credentials to the logical names given in the configuration files. This file should be saved in the GWCelery XDG config directory (${HOME}/.config/gwcelery/ by default on many Linux and UNIX-like operating systems). An example file can be seen below:

{
    "consumer": {
        "fermi": "user_one",
        "swift": "user_two"
    },
    "producer": {
        "gcn": "user_one",
        "scimma": "user_three"
    }
}

Note that one user can be specified multiple times. hop auth must have information about each user specified in this file. Every Kafka producer and consumer configuration key must have an entry in this file.

GraceDB and CVMFS token

You must provide a valid credential for communicating with GracedB and for reading frames from CVMFS for detchar checks. You can obtain a robot keytab for SciToken authentication with the scopes read:/virgo gracedb.read. The keytab should be stored in the ${HOME} directory, and named krb5.keytab.

For production deployments using HTCondor, you must configure the bearer token file location. Set the BEARER_TOKEN_FILE environment variable to specify the location of the bearer token file. This is typically done in your .bashrc or deployment configuration:

export BEARER_TOKEN_FILE=/run/user/$(id -u)/bt_u$(id -u)

This environment variable should be set both in your shell environment and passed through to HTCondor jobs. GWCelery’s HTCondor integration automatically includes BEARER_TOKEN_FILE in the list of environment variables passed to worker processes.

Redis

We recommend that you make the following settings in your Redis server configuration file (which is located at /etc/redis.conf on most systems):

# Some GWCelery tasks transfer large payloads through Redis.
# The default Redis client bandwidth limits are too small.
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 256mb 64mb 60

# If worker nodes are only reachable on a specific network interface,
# then make sure to bind any additional IP addresses here.
bind 127.0.0.1 10.0.0.1  # replace 10.0.0.1 with address on cluster network

# Disable RDB snapshots.
save ""

# Enable appendonly snapshots.
appendonly yes

If you have to make any changes to your Redis configuration, be sure to restart the Redis daemon.

Cron

For deployments of GWCelery at LIGO Data Grid computing sites, it is recommended that you configure cron <cron(8)> to call the script cron.sh once per hour by adding the following to your crontab <crontab(1)>:

@hourly $HOME/cron.sh

This script automatically renews credentials, rotates log files, and cleans up old HTCondor log files.