TracNav(xQTL)?

xQTL workbench - Setup job submission

TODO! content here

Configure your local system to accept and run jobs

• Use Settings -> Install R packages
• BioConductor package, manual install??
• Curl install??

Configure your computer cluster to accept and run jobs

• You have a LINUX/UNIX cluster, then:
• Install OpenPBS (torque+maoi)
• Install ​R
• Your scheduler (master/head..) node must a SSH keyboard interactive login mechanism
• Try to log in to the cluster from your application server. This will accept the SSH key and ensure your firewall isn't blocking outgoing SSH.

Curl install

Inside R: install.packages("RCurl",lib="~/libs")

might result in...

installing *source* package ‘RCurl’ ...
checking for curl-config... no
Cannot find curl-config
ERROR: configuration failed for package ‘RCurl’

Fix by... sudo apt-get install libcurl3-openssl-dev

inside, try again

*** installing help indices
** building package indices ...
* DONE (RCurl)

OLD STUFF

Installation

Overview

• Configure your local system to accept and run jobs
• Configure your computer cluster to accept and run jobs

Running on Tomcat

• NOTE: The Tomcat user is the 'special' user under which Tomcat is running and accessing the system. (ie. tomcat6)
• The Tomcat user must own the Tomcat home directory to install dependencies. (ie. /user/share/tomcat6)
• NOTE: If libraries are installed owned by root, they cannot be overwritten by Tomcat, meaning you could be stuck with an old version. If problems occur, delete the libraries and reinstall using the Dependency Manager plugin.
• The Tomcat user must own the Tomcat java working directory to be able to execute commands. (ie. /var/lib/tomcat6, use the tomcattmpdir helper servlet to find out)
• If 'RCurl' won't install under UNIX, try using the commandline as root and install the following package: 'libcurl3-openssl-dev'

Configuration for Windows

• Install RTools ​RTools
• Install Curl for Windows, use a generic binary. Put the install dir on your PATH. ​cURL
• ...

Continuation

• Optional: if your cluster uses a module system to load programs, load this set:

module initadd torque
module initadd maui
module initadd R

• Then log out from the cluster, then log back in.

• Create a libs folder for the dependencies using mkdir libs
• Start R and install a few things:

install.packages("RCurl",lib="~/libs")
install.packages("qtl",lib="~/libs")
install.packages("snow",lib="~/libs")

• Install our custom 'ClusterJobs' library by performing a 'checkout' and R install. From your home folder, do:

wget -r -l2 http://www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/ClusterJobs/
cd www.xgap.org/svn/xgap_1_4_distro/handwritten/java/plugins/cluster/R/
R CMD INSTALL ClusterJobs --library=~/libs 

• Make sure the firewall of your receiver (server that is running XGAP+Cluster+R/qtl) is not blocking port 8080 so the cluster can report back progress and save your results.

other old stuff

Adding new ComputationResources

It is possible to add new types of ComputationResource, places where jobs can be executed.

The Java interface ComputationResource is located at /xgap_1_4_distro/handwritten/java/plugins/cluster/interfaces/ComputationResource.java in your checked-out project and can be implemented.

In the folder /xgap_1_4_distro/handwritten/java/plugins/cluster/implementations/ you can find two implementations that we currently provide: ​LocalComputationResource.java and ​ClusterComputationResource.java.

They implement the following functions:

public boolean installDependencies()

Install the needed R-packages on this ComputationResource.

public boolean cleanupJob(int jobIs)

Remove job files and generated data from this ComputationResource.

public String executeCommand(Command command)

Execute a system command on this ComputationResource.

public List<String> executeCommands(List<Command> commands)

Execute multiple system commands. Not a wrapper for executeCommand, because it can behave differently.

At the moment, extending the system is not trivial because there are many ComputationResource specific parts in the core workings of ClusterPlugin.java. In further development, we will gradually make this more generic and easily extendable.