0

I need to run 2 different python scripts: script1 and script2 in parallel without any interaction. They are located as the following way:

dir0 contains the file jobcript.py and 2 directories named dir1, dir2. dir1 contains script1 and dir2 contains script2.

The jobscript.txt has the following lines in it.

#!/usr/bin/python
import subprocess

exit_code1 = subprocess.call(["python", "./dir1/script1", "-o"], shell=False)
exit_code2 = subprocess.call(["python", "./dir2/script2", "-o"], shell=False)

I ran the following command in linux:

$ python jobscript.py -o

But this runs in series. How can I run them in parallel? Solution is much appreciated!

Zaman
  • 37
  • 8
  • is sbatch installed? – shafeen Feb 21 '16 at 02:55
  • also why are you running a python script with the bash shebang line? – shafeen Feb 21 '16 at 02:56
  • @shafeen actually I am very new in linux. Trying to learn. Can you please correct the code? Thanks. – Zaman Feb 21 '16 at 02:59
  • the shebang line (`#!`) is the line at the top of your file telling it what kind of script it is, for python you should use `#!/bin/python` or `#!/usr/bin/env python` – shafeen Feb 21 '16 at 03:02
  • @ I got your point! I modified the code like this: #!/usr/bin/python import subprocess. This is working!! Thanks for the idea! Now can you give me idea how can I do the two jobs in different processors parallelly without interaction? – Zaman Feb 21 '16 at 03:06

1 Answers1

1

You can get shell to put the process in the background for you:

from subprocess import call
call(["python ./dir1/script1 -o &"], shell=True)
call(["python ./dir2/script2 -o &"], shell=True)

The "&" tells bash to put it in background. If you want python script to wait for result of each script, then you will need to create a thread. I suspect you want to use a bash script instead of python:

#!/bin/bash
python ./dir1/script1 -o &
python ./dir2/script2 -o &

P.S.: Why call python scripts from python via subprocess in the first place? You can just access that code directly. If you want it to run in parall then multithreading or multiprocessing is your friend.

Muposat
  • 1,476
  • 1
  • 11
  • 24
  • Thank you @Muposat! Can you tell me how I can distribute these 2 parallel processes to two different processors (nodes)? Much appreciated! – Zaman Feb 21 '16 at 03:53
  • It looks like bash does allow you to start on a particular cpu. I have never needed it as the OS usually does a good job. Here is your link: http://stackoverflow.com/questions/33994983/assigning-a-cpu-core-to-a-process-linux – Muposat Feb 21 '16 at 04:02
  • Thanks again @Muposat! – Zaman Feb 21 '16 at 04:27