Latest improvements for QIAGEN CLC Genomics Cloud Engine

CLC Genomics Cloud Engine 21.0.5 contains CLC Genomics Server versions 21.0.1, 21.0.2, 21.0.3, 21.0.4, 21.0.5 and 21.0.6.

CLC Genomics Cloud Engine 21.0.5 release page

Upgrade notes

Upgrading from a CLC Genomics Cloud Engine 21.0.0-4 can be done by following the “Stack upgrade” process described in the GCE Administration manual.

CLC Genomics Cloud Engine 21.0.4 and earlier can no longer be deployed due to configuration changes at AWS. Please upgrade to CLC Genomics Cloud Engine 21.0.5 if continuing to use CLC Workbenches or CLC Servers in the 21 release line. Upgrade to CLC Genomics Cloud Engine 22.0.1 if you can upgrade to CLC Workbenches and CLC Servers in the 22 release line.

For existing GCE 21.x setups in ap-northeast-1 (Tokyo) and us-west-1 (California): Just before running the create_stack.py script in the upgrade process, check that the availability zones in the DB_AZ1 and DB_AZ2 fields in the config/setup.sh file in the GCE administration host root account are available in your region. If they are not, then edit the values in those fields. E.g. for the Tokyo region, DB_AZ1=a and DB_AZ2=c would be acceptable.

Improvements

• Deployment has been simplified by the inclusion of a new script, create_user_pool.py that, in a single step, deploys an AWS Cognito user pool and configures GCE to use it.
• A new script “update/update_jobmanager.py” is included in the GCE administration host to support manually applying security updates. When run, Apache Tomcat and the Apache HTTP Server are updated to the latest security releases. (See the “Applying security updates to the Job Manager” section in the GCE administration manual for details.)
• During deployment via CloudFormation access to the Job Manager can be limited to a fixed IP address or address range, making it more straightforward to apply this restriction.
• The installation chapter of the GCE administration manual has been rewritten. Other chapters have been revised.

Changes

Amazon Linux 2 is now used in the AWS Beanstalk apps.

Limiting access the GCE administration host

Access to the GCE administration host can be limited to a specified IP or IP range by going to the Network & Security | Security Groups section under EC2 in the AWS Console and finding the GCE administration host. Open the Inbound Rules tab, select the first rule in the list and click on “Edit Inbound Rules”. Click on the “Source” drop down and either choose “Select Custom” and enter an IP or IP range using CIDR notation, or  choose “Select My IP” and use the value that is then filled in for you. Save the rules.

The GCE Administration Manual incorrectly states that the IP can be restricted as part of configuring the GCE administration stack using Cloud Formation. Please follow the steps above instead, after the GCE administration stack has been created.

Compatibility

Workflow compatibility

Workflows to be run on the CLC Genomics Cloud Engine 21.0.5 are designed using a CLC Workbench. We recommend using the most up to date compatible versions when working with CLC Genomics Cloud Engine 21.0.5, which are:

• CLC Genomics Workbench 21.0.6
• CLC Main Workbench 21.0.6

Service/Client compatibility

Workflows can be submitted for execution using a compatible Workbench with the Cloud Plugin installed, and also via compatible versions of the CLC Genomics Server with the Cloud Server Plugin installed. The versions of Workbenches and CLC Servers compatible with CLC Genomics Cloud Engine 21.0.5 are 21.0.1, 21.0.2, 21.0.3, 21.0.4, 21.0.5 and 21.0.6. 

Workflows can also be submitted for execution using  the CLC Genomics Cloud Engine Command Line Tools. The corresponding version is 21.0.5. Workflow installer files that are submitted must be made with a CLC Workbench compatible with CLC Genomics Cloud Engine 21.0.5.

CLC Genomics Cloud Engine Command Line Tools 21.0.5

This is a compatibility release, as the corresponding client for the CLC Genomics Cloud Engine 21.0.5.

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Bugfixes

• Fixed an issue that prevented the successful use of instance types not built on the Nitro System.

Upgrade notes

When upgrading existing installations in the 21.x release line, where the RDS database is not installed in AWS Availability Zones “a” and “b”, then DB availability zones to use must be set in the Admin Host config/setup.sh file. For example, for ap-northeast-1/Tokyo:

DB_AZ1=a
DB_AZ2=c

Compatibility

Workflow compatibility

CLC Genomics Cloud Engine 21.0.4 uses CLC Genomics Server 21.0.5.

Workflows to be run on the CLC Genomics Cloud Engine 21.0.4 are designed using a CLC Workbench. Workflow installer files should be made using a CLC Workbench compatible with the CLC Genomics Server run by GCE. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 21.0.5
• CLC Main Workbench 21.0.5

Service/Client compatibility

CLC Genomics Cloud Engine 21.0.4 contains CLC Genomics Server 21.0.1, 21.0.2, 21.0.3, 21.0.4 and 21.0.5. Workflows can be submitted for execution using a corresponding client Workbench version with the Cloud Plugin installed. Workflows can also be submitted via corresponding versions of the CLC Genomics Server with the Cloud Server Plugin installed.

CLC Genomics Cloud Engine Command Line Tools 21.0.4 is the corresponding client for CLC Genomics Cloud Engine 21.0.4. While we recommend running corresponding versions of service and client, CLC Genomics Cloud Engine Command Line Tools 21.0, 21.0.1, 21.0.2 and 21.0.3 can be  used with CLC Genomics Cloud Engine 21.0.4, and CLC Genomics Cloud Engine Command Line Tools 21.0.4 can be used with earlier CLC Genomics Cloud Engine 21.0.x versions.

CLC Genomics Cloud Engine Command Line Tools  21.0.4

This is a compatibility release, as the corresponding client for the CLC Genomics Cloud Engine 21.0.4.

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Improvements

• Tokyo (ap-northeast-1) AWS region is now supported.
• The internal file system used has been changed from XFS to EXT4.

Changes

• CLC Genomics Cloud Engine 21.0.3 supports workflows launched using CLC Genomics Workbench 21.0.4 and CLC Main Workbench 21.0.4, as well as earlier Workbench versions in the 21 release line.

Frequently Asked Questions

CLC Genomics Cloud Engine Command Line Tools 21.0.3

This is a compatibility release, as the corresponding client for the CLC Genomics Cloud Engine 21.0.3.

Compatibility

Workflow compatibility

Workflows to be run on the CLC Genomics Cloud Engine 21.0.3 are designed using a CLC Workbench. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 21.0.4
• CLC Main Workbench 21.0.4

CLC Genomics Cloud Engine 21.0.3 contains CLC Genomics Server 21.0.1, 21.0.2, 21.0.3 and 21.0.4. Workflows can be  submitted for execution using a corresponding client Workbench version with the Cloud Plugin installed. Workflows can also be submitted via corresponding versions of the CLC Genomics Server with the Cloud Server Plugin installed.

Service/Client compatibility

The CLC Genomics Cloud Engine Command Line Tools 21.0.3 is the corresponding client for the CLC Genomics Cloud Engine 21.0.3.

While we recommend running corresponding versions of service and client, CLC Genomics Cloud Engine Command Line Tools 21.0.3 can be used with earlier CLC Genomics Cloud Engine 21.0.x versions, and CLC Genomics Cloud Engine Command Line Tools 21.0.3, can be used with earlier CLC Genomics Cloud Engine 21.0.x versions.

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Improvements

• Updates were made to accommodate changes at AWS. There are no changes to product functionality.

Frequently Asked Questions

CLC Genomics Cloud Engine Command Line Tools 21.0.2

This is a compatibility release, as the corresponding client for the CLC Genomics Cloud Engine 21.0.2.

Compatibility

Workflow compatibility

Workflows to be run on the CLC Genomics Cloud Engine 21.0.2 are designed using a CLC Workbench. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 21.0.3
• CLC Main Workbench 21.0.3

CLC Genomics Cloud Engine 21.0.2 contains CLC Genomics Server 21.0.1, 21.0.2,  and 21.0.3. Workflows can be  submitted for execution using a corresponding client Workbench version with the Cloud Plugin installed. Workflows can also be submitted via corresponding versions of the CLC Genomics Server with the Cloud Server Plugin installed.

Service/Client compatibility

The CLC Genomics Cloud Engine Command Line Tools 21.0.2 is the corresponding client for the CLC Genomics Cloud Engine 21.0.2.

While we recommend running corresponding versions of service and client, CLC Genomics Cloud Engine Command Line Tools 21.0 can be used with CLC Genomics Cloud Engine 21.0.2, and CLC Genomics Cloud Engine Command Line Tools 21.0.2, can be used with CLC Genomics Cloud Engine 21.0.

This update contains only changes to the software. There are no changes to the infrastructure configuration.

Bugfixes

• Fixed an issue that could cause workflows with Iterate elements to fail when many elements were provided as input.
• Fixed an issue causing custom beta plugins to not be installed correctly using deploy_custom_plugins.sh from the admin host.
• Fixed an issue causing Adminhost to run out of disk space when creating JobExecutor stacks.
• Fixed a timing issue that affected workflows with tools provided by a plugin, where occasionally these could fail with an error saying the server command was not installed.

Improvements

• Added CLC Genomics Server support for version 21.0.2 and 21.0.3

Workflow Compatibility

Workflows to be run on the CLC Genomics Cloud Engine 21.0.1 are designed using a CLC Workbench. The recommended versions for this purpose for this version are:

• CLC Genomics Workbench 21.0.3
• CLC Main Workbench 21.0.3

CLC Genomics Cloud Engine 21.0.1 contains CLC Genomics Server 21.0.1, 21.0.2,  and 21.0.3. Workflows can be  submitted for execution using a corresponding client Workbench version with the Cloud Plugin installed. Workflows can also be submitted via corresponding versions of the CLC Genomics Server with the Cloud Server Plugin installed.

Frequently Asked Questions

CLC Genomics Cloud Engine Command Line Tools 21.0.1

The CLC Genomics Cloud Engine Command Line Tools 21.0.1 is the corresponding client for the CLC Genomics Cloud Engine 20.0.10.

While we recommend running corresponding versions of service and client, CLC Genomics Cloud Engine Command Line Tools 21.0 can be  used with CLC Genomics Cloud Engine 21.0.1, and CLC Genomics Cloud Engine Command Line Tools 21.0.1, can be used with CLC Genomics Cloud Engine 21.0.

This update contains changes to the infrastructure as well as the software. Because of the infrastructure changes, it is necessary to redeploy the CLC Genomics Cloud Engine.

Please also refer to the Cloud Plugin 21.0 latest improvements for associated features and improvements affecting the connection to and use of CLC Genomics Cloud Engine 21.0 from CLC Genomics Workbench 21.x and CLC Genomics Server 21.x software.

New Features

• Multiple AWS EC2 instance types are supported. A default set comes preconfigured, but this list can be adjusted in the admin host. Instance types are no longer managed in the Elastic Beanstalk service.
• When workflows containing tools distributed via plugins are run, the relevant plugin version is dynamically deployed to GCE.
• Data stored in Illumina BaseSpace can be imported directly using on-the-fly import in workflows.
• Third party software can be executed from within workflows run on GCE. This is done by including elements in the workflow for containerized external applications that have been made available in Amazon ECR. A local CLC Genomics Server is required for configuring such external applications.
• GCE releases from this point forward are backward compatible with earlier CLC Genomics Server versions within the same major release line.

Improvements

Performance improvements

• Substantially improved memory use (up to 4 times less memory required for each job)
• Executor instances start up faster (up to 3 times faster than previously)
• Executor instances shut down faster

Integration improvements

• Events are sent when a batch is complete, not just when a job completes.

Other improvements

• Additional log files from the CLC Genomics Server have been added to CloudWatch.
• Default log retention has been set to 180 days.
• The Spring property: “security.oauth2.resource.id” can be used for OAuth2 configuration.
• Cache buckets can be created in regions and AWS accounts other than those the GCE is deployed in.
• The form filled out when creating a cache bucket using CloudFormation has been simplified.
• The job status description has been improved for situations where a job is rescheduled due to a timeout.
• If a job is submitted for a workflow created in a CLC Workbench version that is not compatible with a version of the CLC Genomics Server available in the GCE instance, a validation error is reported to the user and the job is not submitted.

Bug fixes

• Fixed an issue that caused an error when all plugins were deleted.

Changes

• The Java version bundled with CLC Genomics Cloud Engine 21.0 is Java 11, where we use the JRE from Amazon Correto.
• Job priority options were replaced by option to specify instance types.
• Several scripts on the admin host have been renamed or moved. The CLC Genomics Cloud Engine administrator manual contains the up-to-date script names. The changes include, but are not limited to:
  • Prepare script: admin/prepare.sh → ./prepare.py
  • Install GCE: ./create-stack.py → ./create_stack.py
  • Configure GCE stack name: /root/setup.sh → /root/config/setup.sh
  • Configure authentication: ~/job-manager-config/application-oauth2.yaml → ~/config/application-oauth2.yaml
  • Restart license server: admin/tools/restart-licenseserver.sh → ./restart_license_server.py
• The CloudFormation form used when creating an admin host no longer requires the definition of a solution stack.
• The admin host no longer uses the Elastic Beanstalk service.

Frequently Asked Questions

CLC Genomics Cloud Engine Command Line Tools 21.0.0

Bug fixes

• Fixed an issue that caused job submissions to fail if the workflow .cpw file was located in a bucket with a name that contained a period.

Compatibility

The CLC Genomics Cloud Engine Command Line Tools 21.0.0 is the corresponding client for the CLC Genomics Cloud Engine 21.0.0.

It is not possible to access the CLC Genomics Cloud Engine 21.0.0 with an earlier version of the CLC Genomics Cloud Engine Command Line Tools. Nor is it possible to access an earlier version of the CLC Genomics Cloud Engine with the CLC Genomics Cloud Engine Command Line Tools 21.0.0.

The CLC Genomics Cloud Engine uses CLC Genomics Server 21.0.1.

Workflows to be run on the CLC Genomics Cloud Engine 21.0.0 are designed using a CLC Workbench. Workflow installer files should be made using a CLC Workbench compatible with the CLC Genomics Server run by GCE. For this release, these are:

• CLC Genomics Workbench 21.0.1
• CLC Main Workbench 21.0.1

Workflows can be submitted to run on CLC Genomics Cloud Engine 21.0.0 from compatible versions of CLC Workbenches with the Cloud Plugin installed and from compatible versions of CLC Genomics Server with the Cloud Server Plugin installed.