wiki:HPC_deploy

Version 26 (modified by Pieter Neerincx, 9 years ago) (diff)

--

SOP for central deployment of software and reference data sets

Table of Contents

  1. Deploying (reference) data sets
  2. Deploying software
    1. Find existing EasyConfig or create a new one
    2. Installing the software
    3. Updating the Lmod caches and syncing installed software to nodes
  3. FAQ
    1. error: The module file is already there
    2. I try to rerun an already installed tool, but my sources are not updated

Deploying (reference) data sets

Deploying software

Responsibility for deploying and maintaining software is distributed over two teams:

  • Deployment of system software (packages from the repos of the Linux distros we use) is handled by our sys admins and beyond the scope of this SOP.
  • Deployment of bioinformatics software is handled by the bioinformaticians from the depad group.

If you want to join the depad group, please contact the helpdesk. The depad group uses ​EasyBuild, which uses EasyConfigs as recipes to enforce consistent, reproducible installations. In a nutshell an EasyConfig deployment recipe can handle the following steps:

  1. (Bootstrap an installation and satisfy dependencies)
  2. Download the (source) code
  3. Verify checksums of the downloads
  4. Unpack the downloads
  5. Configure the build
  6. Compile the code
  7. Run sanity checks to verify the build was Ok
  8. Install the (compiled) code together with it's EasyBuild log
  9. Generate a module file for use with a module system to configure the environment at runtime.

The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the Storage SOP?. EasyBuild comes with EasyConfigs for many our apps of interest out of the box; These files are stored

  • On our machines in a sub sub sub directory of where EasyBuild was installed - Latest stable release we deployed.
    The easiest way to find this directory is to load EasyBuild and search for an app like this: 
    $> module load EasyBuild
$> module list

Currently Loaded Modules:
  1) EasyBuild/2.2.0

$> eb -S GATK
== temporary log file in case of crash /tmp/eb-JHpvCj/easybuild-tR8MKl.log
== Searching (case-insensitive) for 'GATK' in /apps/software/EasyBuild/2.2.0/lib/python2.6/site-packages/easybuild_easyconfigs-2.2.0-py2.6.egg/easybuild/easyconfigs 
CFGS1=/apps/software/EasyBuild/2.2.0/lib/python2.6/site-packages/easybuild_easyconfigs-2.2.0-py2.6.egg/easybuild/easyconfigs/g/GATK
 * $CFGS1/GATK-1.0.5083.eb
 * $CFGS1/GATK-2.5-2-Java-1.7.0_10.eb
 * $CFGS1/GATK-2.6-5-Java-1.7.0_10.eb
 * $CFGS1/GATK-2.7-4-Java-1.7.0_10.eb
 * $CFGS1/GATK-2.7-4.eb
 * $CFGS1/GATK-2.8-1-Java-1.7.0_10.eb
 * $CFGS1/GATK-3.0-0-Java-1.7.0_10.eb
 * $CFGS1/GATK-3.3-0-Java-1.7.0_21.eb
== temporary log file(s) /tmp/eb-JHpvCj/easybuild-tR8MKl.log* have been removed.
== temporary directory /tmp/eb-JHpvCj has been removed.
  • On ​https://github.com/hpcugent/easybuild-easyconfigs/tree/master/easybuild/easyconfigs - Latest and greatest from the source.
  • On our machines in /apps/sources/EasyBuild/custom/ - Bleeding edge home made EasyConfigs that were not yet committed to and pull-merged into the GitHub repos.

If there is no existing EasyConfig, we have to create one ourselves.

First an example of an custom EasyBuild file created for deploying our NGS_DNA pipeline. Below the code there will be the explanation of all the steps in the script. 

name = 'NGS_DNA'
version = '3.1.2'
namelower = name.lower()
homepage = 'https://github.com/molgenis/molgenis-pipelines'
description = """This distribution already contains several pipelines/protocols/parameter files which you can use 'out-of-the-box' to align and impute your NGS data using MOLGENIS Compute."""

toolchain = {'name': 'dummy', 'version': 'dummy'}
easyblock = 'Tarball'

#dependencies
molname = 'Molgenis-Compute'
molversion = 'v15.04.1-Java-1.7.0_80'
versionsuffix = '-%s-%s' % (molname,molversion)
dependencies = [(molname,molversion)]

source_urls = [('http://github.com/molgenis/molgenis-pipelines/releases/download/%s/' % (version))]
sources = [('%s-%s.tar.gz' % (name, version))]

sanity_check_paths = {
    'files': ['workflow.csv', 'parameters.csv'],
    'dirs': []
}

moduleclass = 'bio'
  • name and version are pretty clear
  • homepage and description; are recommended when releasing a future release of the tool.
  • toolchain; can be left like it is in the example (only when multiple tools need to be installed before this, you should use toolchain (see manual online of Easybuild).
  • easyblock; this is the type of data, tar.gz = ‘Tarball’ , executable = ‘Binary’ . All the different easyblocks are described ​here
  • If there any dependencies (in this case Molgenis-Compute), you put it in the name of your eb file (name will look like this: NGS_DNA-3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80)
  • Using variables instead of typing the same string 5 times is done with %s and then between () the name of the variable.
  • One necessary step is to set sanity_check_paths, this is a check whether the file is unpacked/installed correctly.
  • All the installed eb configs are put automatically in the /apps/modules/all folder, but with moduleclass you can specify an extra module path. N.B. when typing the command module avail on the cluster will only display the non-all modules. So specifying an extra moduleclass is necessary to find your module back in module avail

For more complicated installations like for example for R, please read the ​documentation online and have a look at existing EasyConfigs.

Find existing EasyConfig or create a new one

  • To search for existing EasyConfigs in both the default robot search path as well as in our dir for custom EasyConfigs: 
    $> module load EasyBuild
$> module list
$> eb --robot-paths=:/apps/sources/EasyBuild/custom/ -S MySearchTerm
  • When no suitable EasyConfig is present, create your own: 
    $> cd /apps/sources/EasyBuild/custom/
$> touch MyEasyConfig-1.2.3.eb
    Make sure to name the *.eb file exactly the same as the name of the eventually installed module and its version.

Installing the software

Enable the robot option for automatic dependency resolution and append our dir for custom EasyConfigs to the search path to make EasyBuild search for deps in first the default and second our custom dir in that order of precedence. 

$> module load EasyBuild
$> module list
$> eb --robot --robot-paths=:/apps/sources/EasyBuild/custom/ /path/to/MyEasyConfig-1.2.3.eb

Updating the Lmod caches and syncing installed software to nodes

Before you can use the installed software it needs to be synced to various places and the Lmod caches needs to be updated.

  • Switch to the envsync user: 
    $> sudo -u umcg-envsync bash
  • Now sync the module: 
    $> hpc-environment-sync.bash -m <modulename>/<module version>
  • For a full list of options use the commandline help: 
    hpc-environment-sync.bash -h

FAQ

error: The module file is already there

  • rerunning an .eb file that already is installed will result in this error . To overwrite the module file run with -f argument.

I try to rerun an already installed tool, but my sources are not updated

  • EasyBuild will first always check if there is source code in /apps/sources, when it is not there then it will try to download the file. Removing the source code will solve this problem