Fmriprep 1.0.15 quitting for a subject

imawla · October 3, 2018, 1:17pm

Hi all, we are trying to process some data with fmriprep 1.0.15, however running into errors with one subject in particular. All other subjects have run properly with no errors. The header info for this subject looks fine and we can view the images in fsleyes properly. Below is the error message, anyone have insights on what is going on?

Here’s the fmriprep command we used:

singularity run -B /scratch:/scratch $sing_img $dataDIR $dataDIR/output participant --participant-label $participant --template-resampling-grid /scratch/ERIC/MNI152_T1_2mm_brain.nii.gz --low-mem

And here is the error:

Running fMRIPREP version 1.0.15:
  * BIDS dataset path: /scratch/ERIC/micapp/BIDS_process.
  * Participant list: ['mic07002'].
  * Run identifier: 20181003-091255_f9178b74-50e2-4dc0-a4c5-16f9b231d27b.

181003-09:12:59,811 workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-cuff_bold.nii" (0.41 GB / 451 TRs). Memory resampled/largemem=1.63/3.47 GB.
181003-09:13:00,120 workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-cuff_bold.nii).
181003-09:13:00,587 workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181003-09:13:01,990 workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpost_bold.nii" (0.41 GB / 451 TRs). Memory resampled/largemem=1.63/3.47 GB.
181003-09:13:02,295 workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpost_bold.nii).
181003-09:13:02,760 workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181003-09:13:04,62 workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpre_bold.nii" (0.41 GB / 451 TRs). Memory resampled/largemem=1.63/3.47 GB.
181003-09:13:04,465 workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpre_bold.nii).
181003-09:13:04,930 workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181003-09:13:06,239 workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-thumb_bold.nii" (0.43 GB / 478 TRs). Memory resampled/largemem=1.73/3.80 GB.
181003-09:13:06,544 workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-thumb_bold.nii).
181003-09:13:07,9 workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181003-09:13:08,431 workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-visual_bold.nii" (0.23 GB / 251 TRs). Memory resampled/largemem=0.91/1.48 GB.
181003-09:13:08,732 workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-visual_bold.nii).
181003-09:13:09,198 workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181003-09:13:32,377 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_visual_wf.bold_reference_wf.validate" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_visual_wf/bold_reference_wf/validate".
181003-09:13:32,377 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_cuff_wf.bold_reference_wf.validate" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_cuff_wf/bold_reference_wf/validate".
181003-09:13:32,377 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpre_wf.bold_reference_wf.validate" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_restpre_wf/bold_reference_wf/validate".
181003-09:13:32,378 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_thumb_wf.bold_reference_wf.validate" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_thumb_wf/bold_reference_wf/validate".
181003-09:13:32,380 workflow INFO:
	 [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
181003-09:13:32,380 workflow INFO:
	 [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
181003-09:13:32,380 workflow INFO:
	 [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
181003-09:13:32,380 workflow INFO:
	 [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
181003-09:13:32,386 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_cuff_wf.bold_reference_wf.validate".
181003-09:13:32,386 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpre_wf.bold_reference_wf.validate".
181003-09:13:32,386 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_visual_wf.bold_reference_wf.validate".
181003-09:13:32,387 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_thumb_wf.bold_reference_wf.validate".
181003-09:13:32,396 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpost_wf.bold_reference_wf.validate" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_restpost_wf/bold_reference_wf/validate".
181003-09:13:32,399 workflow INFO:
	 [Node] Running "validate" ("fmriprep.interfaces.images.ValidateImage")
181003-09:13:32,406 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpost_wf.bold_reference_wf.validate".
181003-09:13:32,526 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.bidssrc" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/bidssrc".
181003-09:13:32,529 workflow INFO:
	 [Node] Running "bidssrc" ("fmriprep.interfaces.bids.BIDSDataGrabber")
181003-09:13:32,530 interface WARNING:
	 No 't2w' images found for sub-<undefined>
181003-09:13:32,531 interface WARNING:
	 No 'flair' images found for sub-<undefined>
181003-09:13:32,531 interface WARNING:
	 No 'fmap' images found for sub-<undefined>
181003-09:13:32,531 interface WARNING:
	 No 'sbref' images found for sub-<undefined>
181003-09:13:32,531 interface WARNING:
	 No 'roi' images found for sub-<undefined>
181003-09:13:32,535 workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_mic07002_wf.bidssrc".
181003-09:13:33,565 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_cuff_wf.bold_split" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_cuff_wf/bold_split".
181003-09:13:33,568 workflow INFO:
	 [Node] Running "bold_split" ("niworkflows.nipype.interfaces.fsl.utils.Split"), a CommandLine Interface with command:
fslsplit /scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-cuff_bold.nii -t
181003-09:13:33,722 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpost_wf.bold_split" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_restpost_wf/bold_split".
181003-09:13:33,725 workflow INFO:
	 [Node] Running "bold_split" ("niworkflows.nipype.interfaces.fsl.utils.Split"), a CommandLine Interface with command:
fslsplit /scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpost_bold.nii -t
181003-09:13:33,883 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_restpre_wf.bold_split" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_restpre_wf/bold_split".
181003-09:13:33,886 workflow INFO:
	 [Node] Running "bold_split" ("niworkflows.nipype.interfaces.fsl.utils.Split"), a CommandLine Interface with command:
fslsplit /scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-restpre_bold.nii -t
181003-09:13:34,16 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_thumb_wf.bold_split" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_thumb_wf/bold_split".
181003-09:13:34,18 workflow INFO:
	 [Node] Running "bold_split" ("niworkflows.nipype.interfaces.fsl.utils.Split"), a CommandLine Interface with command:
fslsplit /scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-thumb_bold.nii -t
181003-09:13:34,191 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_visual_wf.bold_split" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_visual_wf/bold_split".
181003-09:13:34,193 workflow INFO:
	 [Node] Running "bold_split" ("niworkflows.nipype.interfaces.fsl.utils.Split"), a CommandLine Interface with command:
fslsplit /scratch/ERIC/micapp/BIDS_process/sub-mic07002/func/sub-mic07002_task-visual_bold.nii -t
181003-09:13:34,230 workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_mic07002_wf.ds_report_about
	Interface: fmriprep.interfaces.bids.DerivativesDataSink
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 341, in _local_hash_check
cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 303, in is_cached
hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 497, in _get_hashval
self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 531, in _get_inputs
info[1][1], info[1][2], value)

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/utils.py", line 673, in evaluate_connect_function
output_value = func(first_arg, *list(args))

  File "<string>", line 14, in fix_multi_T1w_source_name

IndexError: list index out of range

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 339, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 417, in run
cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 303, in is_cached
hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 497, in _get_hashval
self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 531, in _get_inputs
info[1][1], info[1][2], value)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/utils.py", line 673, in evaluate_connect_function
output_value = func(first_arg, *list(args))
  File "<string>", line 14, in fix_multi_T1w_source_name
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
load_entry_point('fmriprep==1.0.15', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 314, in main
fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 595, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 190, in run
self._send_procs_to_workers(updatehash=updatehash, graph=graph)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 347, in _send_procs_to_workers
'traceback': traceback
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 339, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 417, in run
cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 303, in is_cached
hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 497, in _get_hashval
self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 531, in _get_inputs
info[1][1], info[1][2], value)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/utils.py", line 673, in evaluate_connect_function
output_value = func(first_arg, *list(args))
  File "<string>", line 14, in fix_multi_T1w_source_name
IndexError: list index out of range

Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
181003-09:13:34,512 workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_mic07002_wf.func_preproc_task_cuff_wf.bold_reference_wf.gen_ref" in "/scratch/ERIC/micapp/BIDS_process/work/fmriprep_wf/single_subject_mic07002_wf/func_preproc_task_cuff_wf/bold_reference_wf/gen_ref".
181003-09:13:34,518 workflow INFO:
	 [Node] Running "gen_ref" ("niworkflows.interfaces.registration.EstimateReferenceImage")

effigies · October 3, 2018, 1:44pm

Can you verify this problem still occurs with fMRIPrep 1.1.7?

ChrisGorgolewski · October 3, 2018, 3:56pm

It’s also worth checking if the input dataset passes the BIDS Validator.

imawla · October 3, 2018, 4:19pm

BIDS validator validates the input dataset. I can try using the latest version of FMRIPREP.
On a slightly related note, is there a version of FMRIPREP that is deprecated or no long plausible for use? At some point we will need to publish our analyses/results, so we don’t want to keep using an invalid version.

effigies · October 3, 2018, 6:29pm

I would suggest that, when you start a new project, use the latest fMRIPrep. As long as you don’t encounter errors and you don’t need new features, I would recommend continuing to use that version. However, if you encounter bugs that need to be fixed, move to the latest version once they’re fixed, and re-run your preprocessing on all your data.

imawla · October 5, 2018, 5:29pm

Downloaded the latest fmriprep 1.1.8 and ran it on a subject but it is now giving the error on grabbids, see below

Command that was run:

singularity run -B /scratch:/scratch $sing_img $dataDIR $dataDIR/output participant --participant-label sub-mic00000 --template-resampling-grid /scratch/ERIC/MNI152_T1_2mm_brain.nii.gz --low-mem

Error:

/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8

  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)

usage: fmriprep [-h] [--version]

                [--participant_label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]

                [-t TASK_ID] [--debug] [--nthreads NTHREADS]

                [--omp-nthreads OMP_NTHREADS] [--mem_mb MEM_MB] [--low-mem]

                [--use-plugin USE_PLUGIN] [--anat-only] [--boilerplate]

                [--ignore-aroma-denoising-errors] [-v]

                [--ignore {fieldmaps,slicetiming,sbref} [{fieldmaps,slicetiming,sbref} ...]]

                [--longitudinal] [--t2s-coreg] [--bold2t1w-dof {6,9,12}]

                [--output-space {T1w,template,fsnative,fsaverage,fsaverage6,fsaverage5} [{T1w,template,fsnative,fsaverage,fsaverage6,fsaverage5} ...]]

                [--force-bbr] [--force-no-bbr]

                [--template {MNI152NLin2009cAsym}]

                [--output-grid-reference OUTPUT_GRID_REFERENCE]

                [--template-resampling-grid TEMPLATE_RESAMPLING_GRID]

                [--medial-surface-nan] [--use-aroma]

                [--aroma-melodic-dimensionality AROMA_MELODIC_DIMENSIONALITY]

                [--skull-strip-template {OASIS,NKI}]

                [--skull-strip-fixed-seed] [--fmap-bspline] [--fmap-no-demean]

                [--use-syn-sdc] [--force-syn] [--fs-license-file PATH]

                [--no-submm-recon] [--cifti-output | --fs-no-reconall]

                [-w WORK_DIR] [--resource-monitor] [--reports-only]

                [--run-uuid RUN_UUID] [--write-graph] [--stop-on-first-crash]

                [--notrack]

                bids_dir output_dir {participant}

fmriprep: error: the following arguments are required: bids_dir, output_dir, analysis_level

ChrisGorgolewski · October 5, 2018, 5:59pm

The relevant error is

I suspect that $dataDir environment variable might not be set leading to invalid set of command line arguments.

imawla · October 8, 2018, 1:47am

$dataDIR doesn’t seem to be the issue, it is a valid directory

I spelled out the full directory in my command as follows:
singularity run -B /scratch:/scratch /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.1.8.simg /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant --participant-label sub-CuffFMCTRL005 --template-resampling-grid /scratch/Ishtiaq/MNI152_T1_2mm_brain.nii.gz --low-mem

However, it runs just fine when I use fmriprep 1.0.15 (what I have been using thus far) as follows:
singularity run -B /scratch:/scratch /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.0.15.simg /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant --participant-label sub-CuffFMCTRL005 --template-resampling-grid /scratch/Ishtiaq/MNI152_T1_2mm_brain.nii.gz --low-mem

ChrisGorgolewski · October 8, 2018, 6:25am

The same error? This is pretty bizarre. Could you try adding --clean-env after “singularity run”?

imawla · October 8, 2018, 2:00pm

Yes, the same error. Ran with --cleanenv flag as follows, still getting the same error
singularity run --cleanenv -B /scratch:/scratch /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.1.8.simg /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant --participant-label sub-CuffFMCTRL005 --template-resampling-grid /scratch/Ishtiaq/MNI152_T1_2mm_brain.nii.gz --low-mem

ChrisGorgolewski · October 8, 2018, 6:33pm

Could you try:

the same command but without the optional (--) flags
the same command but with compulsory flags (bids_dir output_dir participant) placed after the optional ones.

imawla · October 8, 2018, 10:56pm

Getting the same error when executing without optional flags as follows:
singularity run -B /scratch:/scratch /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.1.8.simg /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant

also tried putting compulsory flags at the end like this and still fails with same error:
singularity run -B /scratch:/scratch /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.1.8.simg --template-resampling-grid /scratch/Ishtiaq/MNI152_T1_2mm_brain.nii.gz --low-mem --participant-label sub-CuffFMCTRL005 /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant

ChrisGorgolewski · October 8, 2018, 11:40pm

@effigies @oesteban any ideas?

effigies · October 9, 2018, 1:55am

No… This is really bizarre. I’ve built a Singularity image from 1.1.8 and used it without problems.

@imawla is it possible for you to share your image? And what batch system are you running on?

imawla · October 9, 2018, 2:19am

@effigies Here is the simg for 1.1.8
https://drive.google.com/file/d/1s76RPS2sV0IPrm6yzPRITvkZq4byH6sX/view?usp=sharing
Let me know if you have any troubles downloading it.

I usually use shell scripts to execute the command, but for debugging this issue, I have been executing line-by-line in the terminal

effigies · October 9, 2018, 2:46pm

Hi @imawla, it looks like there’s something strange with your image. How did you generate it?

In any event, you can use exec to run fmriprep:

singularity exec -B /scratch:/scratch \
    /PROJECTS/REHARRIS/ISHTIAQ/singularity/fmriprep-1.1.8.simg \
    fmriprep /scratch/Ishtiaq/test /scratch/Ishtiaq/test/output participant \
    --participant-label sub-CuffFMCTRL005 \
    --template-resampling-grid /scratch/Ishtiaq/MNI152_T1_2mm_brain.nii.gz \
    --low-mem

imawla · October 14, 2018, 8:38pm

There was definitely something funky with the image I build, so I re-downloaded the image

Now getting a different error, namely ModuleNotFoundError: No module named 'niworkflows.nipype' (in detail below), which I have seen posted on other threads, seems to be an issue with version of niworkflows, but not sure if there’s a solution. I did the following:

singularity shell fmriprep-1.1.8.simg 
Singularity: Invoking an interactive shell within container...
Singularity fmriprep-1.1.8.simg:~> python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nipype
>>> import niworkflows
>>>

It seems that nipype and niworkflows are imported properly.

Details of the original error:

/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
181014-16:27:43,125 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.1.8:
      * BIDS dataset path: /scratch/Ishtiaq/test.
      * Participant list: ['CuffFMCTRL005'].
      * Run identifier: 20181014-162743_2ff02f0e-b65d-465d-9654-f513c4319b29.
    
181014-16:27:44,691 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/Ishtiaq/test/sub-CuffFMCTRL005/func/sub-CuffFMCTRL005_task-pain_bold.nii.gz" (0.03 GB / 180 TRs). Memory resampled/largemem=0.12/0.17 GB.
181014-16:27:44,691 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-CuffFMCTRL005_task-pain_bold.nii.gz
181014-16:27:44,899 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/Ishtiaq/test/sub-CuffFMCTRL005/func/sub-CuffFMCTRL005_task-pain_bold.nii.gz).
181014-16:27:45,260 nipype.workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181014-16:27:46,664 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/scratch/Ishtiaq/test/sub-CuffFMCTRL005/func/sub-CuffFMCTRL005_task-rest_bold.nii.gz" (0.03 GB / 180 TRs). Memory resampled/largemem=0.12/0.17 GB.
181014-16:27:46,665 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-CuffFMCTRL005_task-rest_bold.nii.gz
181014-16:27:46,869 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/scratch/Ishtiaq/test/sub-CuffFMCTRL005/func/sub-CuffFMCTRL005_task-rest_bold.nii.gz).
181014-16:27:47,228 nipype.workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
181014-16:27:50,568 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the following boilerplate:


Results included in this manuscript come from preprocessing
performed using *fMRIPprep* 1.1.8
(@fmriprep1; @fmriprep2; RRID:SCR_016216),
which is based on *Nipype* 1.1.3
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: The T1-weighted (T1w) image was corrected for intensity non-uniformity (INU)
using `N4BiasFieldCorrection` [@n4, ANTs 2.2.0],
and used as T1w-reference throughout the workflow.
The T1w-reference was then skull-stripped using `antsBrainExtraction.sh`
(ANTs 2.2.0), using OASIS as target template.
Brain surfaces were reconstructed using `recon-all` [FreeSurfer 6.0.1,
RRID:SCR_001847, @fs_reconall], and the brain mask estimated
previously was refined with a custom variation of the method to reconcile
ANTs-derived and FreeSurfer-derived segmentations of the cortical
gray-matter of Mindboggle [RRID:SCR_002438, @mindboggle].
Spatial normalization to the ICBM 152 Nonlinear Asymmetrical
template version 2009c [@mni, RRID:SCR_008796] was performed
through nonlinear registration with `antsRegistration`
[ANTs 2.2.0, RRID:SCR_004757, @ants], using
brain-extracted versions of both T1w volume and template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `fast` [FSL 5.0.9, RRID:SCR_002823,
@fsl_fast].


Functional data preprocessing

: For each of the 2 BOLD runs found per subject (across all
tasks and sessions), the following preprocessing was performed.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
The BOLD reference was then co-registered to the T1w reference using
`bbregister` (FreeSurfer) which implements boundary-based registration [@bbr].
Co-registration was configured with nine degrees of freedom to account
for distortions remaining in the BOLD reference.
Head-motion parameters with respect to the BOLD reference
(transformation matrices, and six corresponding rotation and translation
parameters) are estimated before any spatiotemporal filtering using
`mcflirt` [FSL 5.0.9, @mcflirt].
The BOLD time-series (including slice-timing correction when applied)
were resampled onto their original, native space by applying
a single, composite transform to correct for head-motion and
susceptibility distortions.
These resampled BOLD time-series will be referred to as *preprocessed
BOLD in original space*, or just *preprocessed BOLD*.
The BOLD time-series were resampled to MNI152NLin2009cAsym standard space,
generating a *preprocessed BOLD run in MNI152NLin2009cAsym space*.
Several confounding time-series were calculated based on the
*preprocessed BOLD*: framewise displacement (FD), DVARS and
three region-wise global signals.
FD and DVARS are calculated for each functional run, both using their
implementations in *Nipype* [following the definitions by @power_fd_dvars].
The three global signals are extracted within the CSF, the WM, and
the whole-brain masks.
Additionally, a set of physiological regressors were extracted to
allow for component-based noise correction [*CompCor*, @compcor].
Principal components are estimated after high-pass filtering the
*preprocessed BOLD* time-series (using a discrete cosine filter with
128s cut-off) for the two *CompCor* variants: temporal (tCompCor)
and anatomical (aCompCor).
Six tCompCor components are then calculated from the top 5% variable
voxels within a mask covering the subcortical regions.
This subcortical mask is obtained by heavily eroding the brain mask,
which ensures it does not include cortical GM regions.
For aCompCor, six components are calculated within the intersection of
the aforementioned mask and the union of CSF and WM masks calculated
in T1w space, after their projection to the native space of each
functional run (using the inverse BOLD-to-T1w transformation).
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file.
The BOLD time-series, were resampled to surfaces on the following
spaces: *fsaverage5*.
All resamplings can be performed with *a single interpolation
step* by composing all the pertinent transformations (i.e. head-motion
transform matrices, susceptibility distortion correction when available,
and co-registrations to anatomical and template spaces).
Gridded (volumetric) resamplings were performed using `antsApplyTransforms` (ANTs),
configured with Lanczos interpolation to minimize the smoothing
effects of other kernels [@lanczos].
Non-gridded (surface) resamplings were performed using `mri_vol2surf`
(FreeSurfer).


Many internal operations of *fMRIPrep* use
*Nilearn* 0.4.2 [@nilearn, RRID:SCR_001362],
mostly within the functional processing workflow.
For more details of the pipeline, see [the section corresponding
to workflows in *fMRIPrep*'s documentation](https://fmriprep.readthedocs.io/en/latest/workflows.html "FMRIPrep's documentation").


### References


[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
181014-16:28:05,79 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_CuffFMCTRL005_wf.func_preproc_task_pain_wf.bold_split
	Interface: nipype.interfaces.fsl.utils.Split
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

ModuleNotFoundError: No module named 'niworkflows.nipype'

181014-16:28:05,131 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_CuffFMCTRL005_wf.func_preproc_task_pain_wf.ds_report_validation
	Interface: fmriprep.interfaces.bids.DerivativesDataSink
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

ModuleNotFoundError: No module named 'niworkflows.nipype'

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 308, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
ModuleNotFoundError: No module named 'niworkflows.nipype'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 343, in main
    fmriprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 184, in run
    self._send_procs_to_workers(updatehash=updatehash, graph=graph)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 316, in _send_procs_to_workers
    'traceback': traceback
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 224, in _clean_queue
    raise RuntimeError("".join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 308, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
ModuleNotFoundError: No module named 'niworkflows.nipype'

Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
181014-16:28:07,939 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_CuffFMCTRL005_wf.func_preproc_task_pain_wf.bold_split" in "/scratch/Ishtiaq/work/fmriprep_wf/single_subject_CuffFMCTRL005_wf/func_preproc_task_pain_wf/bold_split".

oesteban · October 15, 2018, 5:12am

That looks a lot like your python environment has been leaked into the container.

Please unset PYTHONPATH, and export PYTHONNOUSERSITE=1

imawla · October 20, 2018, 2:57am

Thanks all for your help and for bearing with my naïveté, this is working successfully now.