Can parallel-ssh be used to run X jobs on Y servers, each server running Z parallel tasks?

[champagnealexandre]

Hi,

Just discovered parallel-ssh and would be interested in integrating that in my workflow. Currently using GNU parallel with the following scenario: let’s say I have 1000 simulations to run (compiled C executables located on a shared hard drive), and I want to run those onto a cluster of 25 servers, each server having between 8 and 24 CPU cores. I’m using Python as a wrapper for planning the simulations & analyzing their output, the only step where I’m not using Python is to launch said simulations.

Enter parallel-ssh: given that the number of simulations I have to launch exceeds the # of server times the # of cores, I have to wait for simulations to complete then launch another one (GNU parallel excels in doing that — i.e. using all available cores on a group of servers to run a list of tasks in parallel). I’ve searched parallel-ssh’s docs, but it’s not totally clear yet how easy it would be to wait for termination of one process (hence freeing up a core on one of the machines) and then launching another process on that same machine. Is something like this implemented in parallel-ssh?

Thanks!

[pkittenis]

Hi,

Thanks for the interest. Concurrency is controlled by the pool_size keyword parameter of the parallel client. If the host list size exceeds pool_size, the parallel client will run commands on pool_size number of hosts at a time, until all hosts have been used.

Waiting for a command to finish before running another on the same host can be done as part of the command/script being run. Ie something like run_command('step1 && step2').

Alternatively put in identical hosts in host list for up to the # of cores per host and make use of Per-Host Command substitution to run multiple individual commands on the same server.

Something like

hosts = ['host1', 'host1', 'host2', 'host2']
host_args = ('host1_cmd1', 'host1_cmd2', 'host2_cmd1', 'host2_cmd2')

client = ParallelSSHClient(hosts)

output = client.run_command('%s', host_args=host_args)
<..>

[champagnealexandre]

Hi,

Thanks for the reply. The problem I see, is that I don’t know in advance which command will have to be run on which computer ; i.e. I want to run the next command on the next available server core. So I can’t use expressions such as run_command('step1 && step2') that hard-code in advance what will be run.

In parallel, I use something like this: seq 1000 | parallel -j8 myjob {} so parallel simply assigns the next job to the next of 8 available slots/server as soon as jobs finish.

From what I see, this is not possible in parallel-ssh. Can you confirm?

[pkittenis]

As above, concurrency can be adjusted by the pool_size parameter. If there are multiple commands to run, run_command can be used multiple times.

Some example code of the specific use case would be helpful if this is not clear.

Something like:

my_cmds = [<..>]
for cmd in my_cmds:
    client.run_command(cmd)

The commands need to come from somewhere. The hard coding was an example. parallel is a also a shell program, it can still be used.

Parallel-ssh runs commands over SSH in parallel. What that command is is not relevant to the library.

[champagnealexandre]

Okay I’ll have to take a look closely at the documentation then. Thank you for the answers, appreciate it!