Tower API & Docs release

Tower API & Docs release

- Before software can be reusable it first has to be usable.

Tower was released just over one year ago to streamline the deployment and monitoring of Nextflow pipelines - independently of the infrastructure used. In our previous post, we detailed the growth of Tower and how, during that time, Tower users have monitored and launched more than 110 million jobs! By starting with monitoring, logging & observability, Tower unified the workflow experience into a central command-post and pulled back the curtain on distributed workflows processes running in realtime.

A primary goal with Tower is to enable any user, system or software to leverage the power of Nextflow. From lab technicians launching pipelines with a launchpad, to DevOps engineers building automated CI/CD systems, Tower promises to reduce the friction, costs and time to results for any organization. As the software has matured, it became clear that users from all backgrounds required reference documentation to make good on this promise.

Today makes another exciting milestone as we release the official API and documentation pages to simplify and bring massively scalable data analysis pipelines to everybody.

Documentation

The Tower documentation, available from help.tower.nf, covers an introduction to Tower, getting started and the different usage models. Of particular note are the detailed step-by-step guides for setting up Compute Environments for AWS Batch and Google Cloud Life Sciences. These only assume access to a cloud account and guide users through the configuration process. There are also extensive details for launching pipelines, monitoring workflows, pipeline automation and integrating Git credentials.

All documentation is available on GitHub and with links at the bottom of every page, the community can contribute to build a high-quality resource.

The Tower API 1.0

Tower is built around Micronaut, a modern, API-first microservices architecture. This allows the stack to be deployed in any infrastructure with ease. It also means that all services, including the backend and frontend applications, communicate over an API and that every action submitted using GUI can efficiently be executed programmatically.

After 12 months in development, today, we are publicly releasing the first version of the Tower API documentation and the OpenAPI schema. This API includes all the features apparent to current Tower users via the GUI, plus new exciting options.

GA4GH WES API support

Along with this, we are also releasing the support for the Workflow Execution Schema (WES) API, which provides a portable interface for the execution and monitoring of computational pipelines promoted by the Global Alliance for Genomics and Health standardization body.

The GA4GH consortium develops, among other things, common standards for the execution of genomics workflows. This work highlights the commitment to the development and support of high-quality, open standards for Nextflow developers.

API Usage

To get started with the Tower API, users must first obtain an authorization token. Tokens for the API can be found in the user's settings on the top right of the page. Select Your tokens and select Show HTTP token.

As a first example, we can use the API to query workflows in Tower using the /workflow end-point.

$ curl -H "Authorization: Basic YOUR_AUTH_TOKEN" -X GET https://tower.nf/api/workflow

This end-point returns all the workflow information in JSON format.
A small subset of the object returned is shown below.

{
  "workflows": [
    {
      "workflow": {
        "id": "iNSJSpUbKI5iu",
        "submit": "2020-11-05T16:23:23Z",
        "start": "2020-11-05T16:23:23Z",
        "complete": "2020-11-05T16:23:25Z",
        "dateCreated": "2020-11-05T16:23:25Z",
        "lastUpdated": "2020-11-05T16:23:25Z",
        "runName": "infallible_stallman",
        "sessionId": "81b0432a-39e6-4e1c-909e-5440a1321f11",
        "profile": "standard",
        "workDir": "/home/some-user/Projects/nextflow/work",
        "commitId": "3407b1f4fa2ea836a8b5f1c39fd4d3563b919411",
        "userName": "some-user",
        "scriptId": "78b27358c009e993661ed5e417522104",
        "revision": "master",
        "commandLine": "nextflow run hello -with-tower",
        "projectName": "some-user/hello",
        "scriptName": "foo.nf",
        "status": "SUCCEEDED",
        ...
      }
    }
  ]
}

This information could be used to query the status of a workflow execution or build more advanced logic into a more extensive system. In the coming weeks, we will be adding a richer API documentation with more examples and guides.

Thanks for reading and stay tuned for more exciting announcements on the way!

If your organization is interested in deploying Tower in your cloud or on-prem environment, please reach out to us at info@seqera.io, and we would be happy to discuss your requirements.