Quick start

These instructions are suitable for installing GWCelery for development and testing on any machine.

GWCelery requires Python >= 3.7 and a Linux or UNIX-like operating system. It does not support Windows.

To install

GWCelery uses Poetry for packaging, dependency tracking, and virtual environment management; and the poetry-dynamic-versioning plugin for synchronizing the package’s version number with Git tags. First, install these two tools if you do not already have them.

  1. Run the following command to install Poetry using the recommended method:

    $ curl -sSL https://install.python-poetry.org | python3 -
  2. Then, install poetry-dynamic-versioning using pip:

    $ pip install poetry-dynamic-versioning
  3. Run these commands to clone the GWCelery git repository:

    $ git clone https://git.ligo.org/emfollow/gwcelery.git
    $ cd gwcelery
  4. Inside the cloned git repository, run this command to create a Poetry-managed virtual environment containing GWCelery and all of its dependencies:

    $ poetry install
  5. Now, whenever you want to enter a shell within the virtual environment, run this command inside the git clone directory:

    $ poetry shell

To test

First, install the extra test dependencies in the Poetry-managed virtual environment by running this command:

$ poetry install --extras=test

Then, to run the unit tests, just run pytest within the Poetry virtual environment:

$ poetry shell
$ pytest

As a shortcut, you can use poetry run to execute a single command within the virtual environment, like this:

$ poetry run pytest

To start

Before starting GWCelery, you need to authenticate for access to GraceDB and IGWN Alert and make sure that you have a Redis server running. Once you have completed those steps, you can start each of the GWCelery manually.


To authenticate for GraceDB, obtain grid credentials from ligo-proxy-utils by running ligo-proxy-init:

$ ligo-proxy-init albert.einstein

To authenticate for IGWN Alert, create an account in SCiMMA Auth portal, and follow the necessary steps in the IGWN Alert Users Guide. Make a note of the passwords and store them in your ~/.netrc file with appropriate file permissions:

$ echo > ~/.netrc
$ chmod 0600 ~/.netrc
$ echo machine kafka://kafka.scimma.org/ login albert.einstein password password-for-production >> ~/.netrc
$ echo machine kafka://kafka.scimma.org/ login albert.einstein password password-for-playground >> ~/.netrc
$ echo machine kafka://kafka.scimma.org/ login albert.einstein password password-for-test >> ~/.netrc


GWCelery requires a Redis database server for task bookkeeping. Your operating system’s package manager may be able to install, configure, and automatically launch a suitable Redis server for you.

Debian, Ubuntu, apt

Debian or Ubuntu users can install and start Redis using apt-get:

$ sudo apt-get install redis

macOS, MacPorts

Mac users with MacPorts can install Redis using port install:

$ sudo port install redis

Use port load to start the server:

$ sudo port load redis

From source

If none of the above options are available, then you can follow the Redis Quick Start instructions to build redis from source and start a server:

$ wget http://download.redis.io/redis-stable.tar.gz
$ tar xvzf redis-stable.tar.gz
$ cd redis-stable
$ make -j
$ src/redis-server

Start GWCelery components manually

GWCelery itself consists of six Celery workers and one Flask web application. Start them all by running each of the following commands:

$ gwcelery worker -l info -n gwcelery-worker -Q celery -B --igwn-alert
$ gwcelery worker -l info -n gwcelery-exttrig-worker -Q exttrig -c 1
$ gwcelery worker -l info -n gwcelery-openmp-worker -Q openmp -c 1
$ OMP_NUM_THREADS=1 gwcelery worker -l info -n gwcelery-multiprocessing-worker -Q multiprocessing -c 1
$ gwcelery worker -l info -n gwcelery-superevent-worker -Q superevent -c 1
$ gwcelery worker -l info -n gwcelery-voevent-worker -Q voevent -P solo
$ gwcelery worker -l info -n gwcelery-em-bright-worker-Q em-bright -c 2 --prefetch-multiplier 1
$ gwcelery flask run


With these arguments, each of the commands above will run until you type Control-C. You may want to run each of them in a separate terminal, or in the background using screen or nohup.