Changes between Version 28 and Version 29 of HPC_deploy


Ignore:
Timestamp:
2015-11-19T01:46:17+01:00 (9 years ago)
Author:
Pieter Neerincx
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • HPC_deploy

    v28 v29  
    99Responsibility for deploying and maintaining software is distributed over two teams:
    1010* Deployment of ''system'' software (packages from the repos of the Linux distros we use) is handled by our sys admins and beyond the scope of this SOP.
    11 * Deployment of bioinformatics software is handled by the bioinformaticians from the ''depad'' group.
    12 
    13 If you want to join the ''depad'' group, please [wiki:Contact contact the helpdesk].
     11* Deployment of bioinformatics software is handled by the bioinformaticians from the ''depad'' group.[[BR]]
     12  If you want to join the ''depad'' group, please [wiki:Contact contact the helpdesk].
     13
    1414The depad group uses [https://hpcugent.github.io/easybuild/ EasyBuild], which in turn uses !EasyConfigs as recipes to enforce consistent, reproducible installations.
    1515In a nutshell an !EasyConfig deployment recipe can handle the following steps:
    … …  
    24241. Generate a module file for use with a module system to configure the environment at runtime.
    2525
    26 The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the [wiki:HPC_storage#Software Storage SOP].
     26The locations where we store source code, deployed apps, their accompanying module files, etc. are documented in the [wiki:HPC_storage#Software Storage SOP]. We use the [https://www.tacc.utexas.edu/research-development/tacc-projects/lmod Lua based module system (Lmod)] to make software transparently available on all machines at runtime. Details on how to install and configure !EasyBuild, Lmod and our environment sync script on a new cluster can be found in our [https://github.com/molgenis/depad-utils/blob/master/hpc-2.x/README.md depad-utils GitHub repo].
     27
    2728!EasyBuild comes with !EasyConfigs for many of our apps of interest ''out of the box''; These files are stored
    2829* On our machines in a sub sub sub directory of where !EasyBuild was installed - Latest stable release we deployed.[[BR]]
    … …  
    115116}}}
    116117
     118=== Specifying the default version of an app (optional) ===
     119
     120When you load an app into your environment with the {{{module load}}} command without specifying explicitly which version you want to use, Lmod will load the highest version number. This is usually fine, but if you installed a new version that is not yet well tested, you may want to explicitly configure an older version as the default. This can simply be accomplished by creating a **relative** symlink named default. Our module files are located at {{{/apps/modules/}}}. For example let's look at our NGS_DNA analysis pipeline. It is part of the ''bio'' collection of apps, so the the module files are in:
     121{{{
     122$> ls -ahl /apps/modules/bio/NGS_DNA/
     123total 20K
     124drwxrwsr-x  2 umcg-gvdvries  umcg-depad 4.0K Nov 12 15:28 .
     125drwxrwsr-x 64 umcg-pneerincx umcg-depad 4.0K Nov 18 17:43 ..
     126lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Aug 25 10:52 3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 -> /apps/modules/all/NGS_DNA/3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
     127lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Nov  4 09:04 3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 -> /apps/modules/all/NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
     128lrwxrwxrwx  1 umcg-rkanninga umcg-depad   71 Nov 12 15:28 3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45 -> /apps/modules/all/NGS_DNA/3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45
     129lrwxrwxrwx  1 umcg-pneerincx umcg-depad   45 Sep 24 16:22 default -> 3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
     130}}}
     131Note that all module files for all apps are in {{{/apps/modules/all/}}} and we only see symlinks in {{{/apps/modules/bio/}}}. Version 3.2.1 is the latest, but 3.1.2 has been designated as default using a symlink. The symlink must be a relative one pointing to file or another symlink in the same directory. We can check with if the symlink was recognised using:
     132{{{
     133$> module avail NGS_DNA
     134
     135-------------------------- /apps/modules/bio --------------------------
     136   NGS_DNA/3.0.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
     137   NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80 (D)
     138   NGS_DNA/3.2.1-Molgenis-Compute-v15.11.1-Java-1.8.0_45
     139
     140  Where:
     141   (D):  Default Module
     142
     143$> module load NGS_DNA
     144$> module list
     145
     146Currently Loaded Modules:
     147  1) Java/1.7.0_80
     148  2) Molgenis-Compute/v15.04.1-Java-1.7.0_80
     149  3) NGS_DNA/3.1.2-Molgenis-Compute-v15.04.1-Java-1.7.0_80
     150
     151}}}
     152
     153=== Deprecating a previously installed older version of an app ===
     154
     155You can inform users that (a certain version of) an app is deprecated and will be removed in the near future by creating a custom message when an app is added to the environment with {{{module load}}}. Add your custom message to {{{/apps/modules/modules.admin}}}. For example with this modules.admin:
     156{{{
     157#
     158# The Lmod admin file consists of "key: value" pairs terminated with a blank line:
     159#
     160#    moduleName/version:  message
     161#    <blank line>
     162#
     163# Or
     164#
     165#    Full/PATH/to/Modulefile: message
     166#    <blank line>
     167#
     168# The message can be as many lines as you like and must be terminated with a blank line.
     169#
     170# Currently used by picard/1.102-Java-1.7.0_80
     171R/3.1.2-goolf-1.7.20: Deprecated incomplete installation. Will be removed in the near future.
     172}}}
     173loading this specific version of ''R'' will now result in:
     174{{{
     175$> module load R/3.1.2-goolf-1.7.20
     176--------------------------------------------------------------------------
     177There are messages associated with the following module(s):
     178--------------------------------------------------------------------------
     179R/3.1.2-goolf-1.7.20:
     180   Deprecated incomplete installation. Will be removed in the near future.
     181--------------------------------------------------------------------------
     182}}}
     183
    117184=== Updating the Lmod caches and syncing installed software to nodes ===
    118185
    … …  
    141208=== Q: !EasyBuild fails to download the source code. How can I continue the installation process? ===
    142209A: First check if the location where !EasyBuild tries to download the source code is still up-to-date. If not update the !EasyConfig. If the location is correct, but !EasyBuild cannot access this location directly for example because it is blocked by our firewall or because it requires authentication, you can try to download the source manually and put it in the cache directory for the app in {{{/apps/sources/[a-z]/NameOfTheApp/}}}. When !EasyBuild finds the cached source code it will skip the download step and continue.
    143 
    144 
    145 
    146  
    147