Job Manager Example1

The Job Manager logic is rather straightforward and can be easily adjusted for use in a specific cluster or server. After a job object is received by Job Manager, it is registered in the database and passed to the Worker nodes for execution.

This code example sends all jobs to one Resident Worker on the cluster for execution. For this the local node should excluded from the grid topology in ​GridStarter

GridBasicTopologySpi topSpi = new GridBasicTopologySpi();

// Exclude local node from topology.
topSpi.setLocalNode(false);

// Configure your own topology SPI.
cfg.setTopologySpi(topSpi);

A job consists of a number of steps which consists of a number of operations (​MCF Data Model). Every job is running in the separate thread (​PipelineThread) Operations are submitted to Worker on the cluster in a loop using the gridgain executor service.

ExecutorService exec = grid.newGridExecutorService();

int numberOfSteps = pipeline.getNumberOfSteps();
...

for (int i = 0; i < numberOfSteps; i++)
{
     Step step = pipeline.getStep(i);

     for (int j = 0; j < step.getNumberOfScripts(); j++)
     {
         Script script = step.getScript(j);

         Future<RemoteResult> future = exec.submit(new RemoteScriptSubmitter(script));

         RemoteResult back = null;
         try
         {
             back = future.get();
         }
...

The java code of ​RemoteScriptSubmitter is executed by Worker on the cluster side. It saves the transferred script and additional files on the cluster and submits the script to the cluster scheduler. The submission results are returned back to ​PipelineThread

//save script remotely
saveData(script);

//save additional files
if(script.isHasAdditionalFiles())
	for(int i = 0; i < script.getNumberFileToSaveRemotely(); i++)
	{
    	saveData(script.getFileToSaveRemotely(i));
    }
    
//sumbit script
SysCommandExecutor cmdExecutor = new SysCommandExecutor();
cmdExecutor.runCommand(SUB + script.getRemoteDir() + script.getRemotename());

//receive submission results
String cmdError = cmdExecutor.getCommandError();
String cmdOutput = cmdExecutor.getCommandOutput();
returnData.setErrString(cmdError);
returnData.setOutString(cmdOutput);

...
return returnData;

Back to ​Examples