Dear experts,
I am running 10 subjects in paralel, it runs 6 days and gives “time out error”.
not sure if it is an error in script of not optimal parameters, because it was working for one # # # subject before adding multiproc and so on part.
if you could spot the error, please let me know.
for subject in subject_list[0:9]: # set here the subjects to run, note zero indexing
cmd = “#!/bin/sh \nsingularity run
-B /imaging/morphology/bids/:/home/jovyan/work/
-B /imaging/morphology/my_mindboggle/:/home/jovyan/work/output/
-B /imaging/local/software/freesurfer/6.0.0/x86_64/license.txt:/config/license.txt
-e /imaging/local/software/singularity_images/mindboggle/v1.3.8/mindboggle.simg
mindboggle123 /home/jovyan/work/%s/anat/%s_T1w.nii.gz
–id %s
–out /home/jovyan/work/output/
–plugin MultiProc
–plugin_args ‘dict(n_procs=16)’
–fs_openmp 5 --ants_num_threads 5 --mb_num_threads 10” % (subject, subject, subject)
file = open(subject + ‘_mindboggle.sh’, ‘w’) # place the command within a .sh file
file.write(cmd)
file.close()
cmd = "sbatch --time=6-00:00:00 --mincpus=6 --mem-per-cpu=10GB " + subject + “_mindboggle.sh” # run command on the cluster, with the sbatch settings
call(cmd, shell=True)
###############
the last part of output file is:
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/opt/miniconda-latest/envs/mb/lib/python3.6/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/opt/miniconda-latest/envs/mb/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py”, line 149, in _async_callback
result = args.result()
File “/opt/miniconda-latest/envs/mb/lib/python3.6/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/opt/miniconda-latest/envs/mb/lib/python3.6/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
FileNotFoundError: /home/jovyan/work/mindboggle123_output/working/Mindboggle123/recon-all/result_recon-all.pklz
210602-15:04:15,21 nipype.workflow INFO:
[MultiProc] Running 2 tasks, and 0 jobs ready. Free memory (GB): 168.23/168.63, Free processors: 10/16.
Currently running:
* Mindboggle123.recon-all
* Mindboggle123.antsCorticalThickness
slurmstepd: error: *** JOB 1160096 ON node-i13 CANCELLED AT 2021-06-08T16:04:17 DUE TO TIME LIMIT ***
cheers,