Fmriprep stuck after apply_mask

1

The preprocessing of one subject stuck after the apply_mask step for ~24 hours on Sherlock. The --anat-only option has be run successfully before for this participant. I actually have two questions here:

(1) What step might make fmriprep running for so long? Usually fmriprep finishes running with 12 hours for other subjects in this project.

(2) I noticed that for apply masks the output file is not in a subject-specific folder (the output file is stored in the home directory of my scratch drive). Would this cause any confusion when preprocessing multiple subjects simultaneously?

Thanks!

Please find below the last a few entries in the output file:

180803-23:51:49,160 workflow INFO:
[Node] Running “fixhdr_unifize” (“niworkflows.interfaces.utils.CopyXForm”)
180803-23:51:49,280 workflow INFO:
[Node] Finished “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_unifize”.
180803-23:51:50,421 workflow INFO:
[Node] Setting-up “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass” in “/scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass”.
180803-23:51:50,462 workflow INFO:
[Node] Running “skullstrip_second_pass” (“niworkflows.nipype.interfaces.afni.preprocess.Automask”), a CommandLine Interface with command:
3dAutomask -apply_prefix uni_xform_masked.nii.gz -dilate 1 -prefix uni_xform_mask.nii.gz /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_second_pass/uni_xform.nii.gz
180803-23:51:51,410 workflow INFO:
[Node] Finished “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.skullstrip_second_pass”.
180803-23:51:54,473 workflow INFO:
[Node] Setting-up “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2” in “/scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2”.
180803-23:51:54,490 workflow INFO:
[Node] Running “fixhdr_skullstrip2” (“niworkflows.interfaces.utils.CopyXForm”)
180803-23:51:54,546 workflow INFO:
[Node] Finished “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.fixhdr_skullstrip2”.
180803-23:51:56,196 workflow INFO:
[Node] Setting-up “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks” in “/scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks”.
180803-23:51:56,224 workflow INFO:
[Node] Running “combine_masks” (“niworkflows.nipype.interfaces.fsl.maths.BinaryMaths”), a CommandLine Interface with command:
fslmaths /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/skullstrip_first_pass/ref_image_corrected_brain_mask.nii.gz -mul /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_skullstrip2/uni_xform_mask_xform.nii.gz /scratch/users/jfj/ref_image_corrected_brain_mask_maths.nii.gz
180803-23:51:56,410 workflow INFO:
[Node] Finished “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.combine_masks”.
180803-23:51:58,216 workflow INFO:
[Node] Setting-up “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask” in “/scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask”.
180803-23:51:58,256 workflow INFO:
[Node] Running “apply_mask” (“niworkflows.nipype.interfaces.fsl.maths.ApplyMask”), a CommandLine Interface with command:
fslmaths /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize/uni_xform.nii.gz -mas /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks/ref_image_corrected_brain_mask_maths.nii.gz /scratch/users/jfj/uni_xform_masked.nii.gz
180803-23:51:58,497 workflow INFO:
[Node] Finished “fmriprep_wf.single_subject_06_wf.func_preproc_task_loca_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask”.

2

Which version of fMRIPrep are you using? Prior to version 1.1.2, there was an issue that memory errors could sometimes cause crashes that could not be detected by the workflow engine, and thus it would wait on a task that would never be marked as complete or failed. In this case, you could kill fMRIPrep and re-run, using the same working directory. Hopefully the memory issue would not be hit, and you could finish your run.

You can also try adding a -v flag to increase the verbosity. That should cause the currently running jobs to be listed periodically, which may make it easier to identify the job that’s still running (or that fMRIPrep thinks is running, even if it’s crashed). You can also use htop, which makes it easy to identify processes that are consuming large amounts of CPU cycles or memory.

Could you give more details on this? I don’t see this reflected in your log.

1 Like
3

(1) I am using 1.0.8 so it may be the issue you mentioned. I am running it again using the ‘-v’ flag, hopefully will have more information to locate where it gets stuck.

(2) In the 2nd to last entry it says:
Node] Running “apply_mask” (“niworkflows.nipype.interfaces.fsl.maths.ApplyMask”), a CommandLine Interface with command:
fslmaths /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/fixhdr_unifize/uni_xform.nii.gz -mas /scratch/users/jfj/work/sub-06_1/fmriprep_wf/single_subject_06_wf/func_preproc_task_loca_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/combine_masks/ref_image_corrected_brain_mask_maths.nii.gz /scratch/users/jfj/uni_xform_masked.nii.gz

So the last file (output file?) is not saved on a subject-specific folder. Say if I am pre-processing sub-01 and sub-02 simultaneously, would it be possible that sub-01 writes on /scratch/users/jfj/uni_xform_masked.nii.gz first, but the file gets overwritten by sub-02 before sub-01 reads from it?

4

Unless you have a strong reason to continue using old versions, I would highly recommend upgrading to the most recent version. We are frequently fixing issues, and using the latest will help distinguish between new issues and those already resolved.

Good catch, I didn’t read carefully enough.

I believe this is another issue that was resolved since 1.0.8. 1.0.8 was released on Feb 22, and problems where filenames were not constructed in the correct directory were resolved in Nipype issue #2521, which would have made it into fmriprep in 1.0.12.