Hello,
I’ve been having issues running fmriprep through singularity on our HPC.
#!/bin/bash
#PBS -l nodes=1:ppn=16,walltime=48:00:00,mem=128gb,vmem=356gb
#PBS -N fMRI-Prep-178950
#PBS -j oe
#PBS -o fMRI-Prep-178950.log
module load singularity-2.5.2
singularity run --cleanenv -B /scratch/kqc/fMRIPrep/sub-178950:/tmp /home/ccp_hcp/fmriprep/SingularityImages/fmriprep-1.1.7.simg --fs-license-file /tmp/.license/freesurfer/license.txt -w /tmp /tmp/fMRIPrep_input /tmp/fMRIPrep_output participant --participant_label 178950 -n-cpus 12 --omp-nthreads 8 --mem-mb 120000 --low-mem -vvvv
it seems like fmriprep will run for awhile then it will get caught
181028-15:46:16,670 nipype.workflow DEBUG:
Cannot allocate job 912 (40.65GB, 8 threads).
Needed files: /tmp/fMRIPrep_output/fmriprep/sub-178950/ses-01/func/sub-178950_ses-01_task-Cuedts_acq-MB42p4PA_run-02_bold_space-MNI152NLin2009cAsym_preproc.nii;/tmp/fMRIPrep_output/fmriprep/sub-178950/ses-01/func/sub-178950_ses-01_task-Cuedts_acq-MB42p4PA_run-02_bold_space-MNI152NLin2009cAsym_preproc.mat;/tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/_0x13e3b85e385d1ce67651aafcea4e3c80_unfinished.json;/tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/_inputs.pklz;/tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/_node.pklz
181027-07:33:37,330 nipype.workflow DEBUG:
Needed dirs: /tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/_report
181027-07:33:37,332 nipype.workflow DEBUG:
Removing files:
181027-07:33:37,335 nipype.workflow DEBUG:
saved results in /tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/result_ds_bold_mni.pklz
181027-07:33:37,336 nipype.workflow DEBUG:
[Node] Writing post-exec report to "/tmp/fmriprep_wf/single_subject_178950_wf/func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf/func_derivatives_wf/ds_bold_mni/_report/report.rst"
181027-07:33:37,339 nipype.workflow DEBUG:
Aggregate: False
181027-07:33:37,340 nipype.workflow INFO:
[Node] Finished "fmriprep_wf.single_subject_178950_wf.func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf.func_derivatives_wf.ds_bold_mni".
181027-07:33:37,340 nipype.workflow INFO:
[Job 852] Completed (fmriprep_wf.single_subject_178950_wf.func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf.func_derivatives_wf.ds_bold_mni).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 879 (43.58GB, 8 threads).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 883 (43.58GB, 8 threads).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 908 (40.65GB, 8 threads).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 912 (40.65GB, 8 threads).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 937 (40.77GB, 8 threads).
181027-07:33:37,673 nipype.workflow DEBUG:
Cannot allocate job 941 (40.77GB, 8 threads).
181027-07:33:38,986 nipype.workflow DEBUG:
Progress: 1287 jobs, 1234/1/6 (done/running/ready), 1/52 (pending_tasks/waiting).
181027-07:33:38,987 nipype.workflow DEBUG:
Tasks currently running: 1. Pending: 1.
181027-07:33:38,995 nipype.workflow INFO:
[MultiProc] Running 1 tasks, and 6 jobs ready. Free memory (GB): 73.70/117.19, Free processors: 4/12.
Currently running:
* fmriprep_wf.single_subject_178950_wf.func_preproc_ses_01_task_Cuedts_acq_MB42p4PA_run_02_wf.bold_t1_trans_wf.bold_to_t1w_transform
81027-07:33:39,307 nipype.workflow DEBUG:
Cannot allocate job 879 (43.58GB, 8 threads).
181027-07:33:39,307 nipype.workflow DEBUG:
Cannot allocate job 883 (43.58GB, 8 threads).
181027-07:33:39,307 nipype.workflow DEBUG:
Cannot allocate job 908 (40.65GB, 8 threads).
181027-07:33:39,307 nipype.workflow DEBUG:
Cannot allocate job 912 (40.65GB, 8 threads).
then it will continue to repeat
181027-07:33:39,307 nipype.workflow DEBUG:
Cannot allocate job 912 (40.65GB, 8 threads).
Cycling though the same set of job numbers. Until it hits the walltime and the job is killed.
Looking at the log files I’m thinking that this could be because I’m setting --omp_threads 8 while only allotting 12 cpus meaning that fmriprep could only have one 8-thread process running at a time. Is this the case?