MNI152Lin in fmriprep 20.0.6

1

Hi,

Wondering if v20.0.6 does not support MNI152Lin anymore?

We used the command below to run fmriprep : The fmriprep container was created on our cluster as a singularity image built directly from fmriprep’s docker image for 20.0.6.)

unset PYTHONPATH;
singularity run - --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.6.img
/data/dorit/data/conte_amy/BIDS
/data/dorit/data/conte_amy/BIDS/derivatives
participant --participant-label $S
–work-dir /data/dorit/data/conte_amy/work
–fs-license-file /data/dorit/license.txt
–output-spaces anat func MNI152Lin:res-2 T1w fsaverage6
-t rest
–ignore slicetiming
–cifti-output
–medial-surface-nan
–skip_bids_validation
–low-mem --mem-mb 24000 --nthreads 3 --notrack

When trying to run asking for the MNI152Lin 2mm output space above, we ran first into a permissions error when an attempted download of the template into the container failed (Read only file system error). The error does not crash the script though, but continues with a error/warning and produces all other output spaces.
Addressing this with a fix from previous versions, we tried to direct the template paths to the user’s home directory by mounting the TemplateFlow folder using the user directory (command below), but then got a different error (Error 1 below, crashes the script) mentioned in a similar context here (https://github.com/poldracklab/fmriprep/issues/1986). Just for the fun of it we checked and this errror also comes up for 20.0.5.

The container works fine with standard output space for 20.0.6 (e.g. FSL MNI space res2 and anat func).

Thanks,
Dorit

Command:
unset PYTHONPATH;
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
singularity run -B $HOME/.cache:/home/fmriprep/.cache --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.6.img
/data/dorit/data/conte_amy/BIDS
…
….
……
………

Error 1:
Process Process-2:
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/multiprocessing/process.py”, line 297, in _bootstrap
self.run()
File “/usr/local/miniconda/lib/python3.7/multiprocessing/process.py”, line 99, in run
self._target(*self._args, **self._kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py”, line 693, in build_workflow
bids_filters=bids_filters,
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py”, line 274, in init_fmriprep_wf
bids_filters=bids_filters,
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py”, line 638, in init_single_subject_wf
use_syn=use_syn,
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/bold/base.py”, line 946, in init_func_preproc_wf
repetition_time=metadata[‘RepetitionTime’])
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/bold/resampling.py”, line 743, in init_bold_grayords_wf
for hemi in ‘LR’
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py”, line 425, in validate
value = super(MultiObject, self).validate(objekt, name, newvalue)
File “/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_types.py”, line 2336, in validate
return TraitListObject( self, object, name, value )
File “/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py”, line 2313, in init
raise excp
File “/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py”, line 2305, in init
value = [ validate( object, name, val ) for val in value ]
File “/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py”, line 2305, in
value = [ validate( object, name, val ) for val in value ]
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py”, line 329, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py”, line 134, in validate
self.error(objekt, name, str(value))
File “/usr/local/miniconda/lib/python3.7/site-packages/traits/trait_handlers.py”, line 172, in error
value )
traits.trait_errors.TraitError: Each element of the ‘current_sphere’ trait of a DynamicTraitedSpec instance must be a pathlike object or string representing an existing file, but a value of ‘[]’ <class ‘str’> was specified.

3

(ops premature unfinished post previously, second try here:)

Brief update:
tried a woraround to set a custom template by copying/renaming files and folder from 'MNI152Lin" to custom (“test”). placing it either in a different location or user’s home location (outside of container) and definng templateflow_home in different ways as instructed here.

None worked successfull y either.

TEMPLATEFLOW_HOME=~/.cache/templateflow
singularity run -B $HOME/.cache:/home --cleanenv /data/singularity_images/ … --output-spaces test:res-2 T1w fsaverage6 \

  1. TEMPLATEFLOW_HOME=/data/dorit/data/conte_amy/scripts/templateflow
    singularity run --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.6.img \ … --output-spaces test:res-2 T1w fsaverage6 \

Error is always a complaint about the non-existing template so maybe my way of pointing to the directory is wrong (likely) or the way to point it to a custom template?

Traceback (most recent call last):
File “/usr/local/miniconda/bin/fmriprep”, line 10, in
sys.exit(main())
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py”, line 296, in main
opts = get_parser().parse_args()
File “/usr/local/miniconda/lib/python3.7/argparse.py”, line 1749, in parse_args
args, argv = self.parse_known_args(args, namespace)
File “/usr/local/miniconda/lib/python3.7/argparse.py”, line 1781, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File “/usr/local/miniconda/lib/python3.7/argparse.py”, line 1987, in _parse_known_args
start_index = consume_optional(start_index)
File “/usr/local/miniconda/lib/python3.7/argparse.py”, line 1927, in consume_optional
take_action(action, args, option_string)
File “/usr/local/miniconda/lib/python3.7/argparse.py”, line 1855, in take_action
action(self, namespace, argument_values, option_string)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/spaces.py”, line 667, in call
for sp in Reference.from_string(val):
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/spaces.py”, line 306, in from_string
return [cls(_args[0], s) for s in allspecs]
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/spaces.py”, line 306, in
return [cls(_args[0], s) for s in allspecs]
File “”, line 11, in init
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/spaces.py”, line 220, in _check_name
‘identifiers are: %s’ % (value, ', '.join(valid)))
ValueError: space identifier “test” is invalid.
Valid identifiers are: MNI152Lin, MNI152NLin2009cAsym, MNI152NLin2009cSym, MNI152NLin6Asym, MNI152NLin6Sym, MNIInfant, MNIPediatricAsym, NKI, OASIS30ANTs, PNC, WHS, fsLR, fsaverage, T1w, T2w, anat, fsnative, func, run, sbref, session
~
~

4

hey @dorit,

There were some modification to templateflow’s skeleton during the 20.0.x series, and that seems like what you experiencing here. We’ve worked on improving the user experience of templateflow, and following the next release, this problem should be extinct.

Here is a link to a gist where another user had a similar problem, and how we were able to resolve it.

Neither 1. or 2. would work since you are setting an environmental variable, but then running with --cleanenv. To get around that, you can take advantage of some singularity magic and add the SINGULARITYENV_ prefix when setting environmental variables that you want visible within the singularity environment.

HTH,
Mathias

5

Hi Mathias,
thanks - I tried the following then:

SINGULARITYENV_TEMPLATEFLOW_HOME=~/.cache/templateflow
SINGULARITYENV_TEMPLATEFLOW_USE_DATALAD=false

singularity run -B $HOME/.cache:/home/fmriprep/.cache --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.6.img \

Still getting the trait error (asking for both MNI152Lin or “test”)

traits.trait_errors.TraitError: Each element of the ‘current_sphere’ trait of a DynamicTraitedSpec instance must be a pathlike object or string representing an existing file, but a value of ‘[]’ <class ‘str’> was specified.

Since its within the container I assume I do not need to upgrade to the dev of templateflow?

Can you clarify what you mean with option2 in the post?

Thanks,
Dorit

6

@dorit

We’ve just released fmriprep 20.0.7, which includes the templateflow autoupdate. It should fix your problem, give it a try and let us know!

1 Like
7

@mgxd

Tried to run it but did not get successful MNI152Lin output spaces.
I can’t find an obvious error in the output file but only one line really of processing related to MNI152Lin:

 "[Node] Setting-up "fmriprep_wf.single_subject_CC0006_wf.func_preproc_ses_core2p2_task_rest_run_01_wf.bold_std_trans_wf.gen_ref" in "/data/dorit/data/fmriprep20.0.7/work/fmriprep_wf/single_subject_CC0006_wf/func_preproc_ses_core2p2_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152Lin.res2/gen_ref".

Our server error log however shows download of surface template spaces (not MNIxxx) and the magic words “fMRIPrep failed: Workflow did not execute cleanly. Check log for details”

The log folder shows crash logs all related to std space:

crash-20200508-004455-dorit-aparc_std_tfm.a1-ddf13dbc-f53f-4b65-9836-5d7474d85371.txt
crash-20200508-004518-dorit-aseg_std_tfm.a1-6c4c99e6-2934-4ee0-8d1e-100c13ac6a87.txt
crash-20200508-004539-dorit-mask_std_tfm.a1-cc6ebb11-8250-4365-8b21-00fec123e0aa.txt
crash-20200508-004701-dorit-aparc_std_tfm.a1-7d5fadfb-2f6d-45df-b839-e807b7d13dc7.txt
crash-20200508-004725-dorit-aseg_std_tfm.a1-4f55dec2-312b-4e5f-8d64-bf625e41826f.txt
crash-20200508-004745-dorit-mask_std_tfm.a1-21f6c4b5-12a4-4f10-b2b3-50b15405bd73.txt
crash-20200508-015157-dorit-bold_to_std_transform.a1-609d2ec0-3c11-4338-a3d0-fc70389ac61d.txt
crash-20200508-031118-dorit-bold_to_std_transform.a1-b12168b4-a85a-4d6d-a186-ec8908e99c47.txt

I post one here since the all show the same type of error about “reference file reading”

Node: fmriprep_wf.single_subject_CC0006_wf.func_preproc_ses_core2p2_task_rest_run_01_wf.bold_std_trans_wf.aparc_std_tfm
Working directory: /data/dorit/data/fmriprep20.0.7/work/fmriprep_wf/single_subject_CC0006_wf/func_preproc_ses_core2p2_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin6Asym.res2/aparc_std_tfm

Node inputs:

args =
default_value = 0.0
dimension =
environ = {‘NSLOTS’: ‘1’}
float = True
input_image =
input_image_type =
interpolation = MultiLabel
interpolation_parameters =
invert_transform_flags =
num_threads = 1
out_postfix = _trans
output_image =
print_out_composite_warp_file =
reference_image =
transforms =

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 397, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/fixes.py”, line 30, in _run_interface
runtime, correct_return_codes)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 792, in _run_interface
self.raise_exception(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 723, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
antsApplyTransforms --default-value 0 --float 1 --input /data/dorit/data/fmriprep20.0.7/work/fmriprep_wf/single_subject_CC0006_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aparc_aseg/tonii/aparc+aseg_vol_out.nii.gz --interpolation MultiLabel --output aparc+aseg_vol_out_trans.nii.gz --reference-image /home/fmriprep/.cache/templateflow/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-02_T1w.nii.gz --transform /data/dorit/data/fmriprep20.0.7/work/fmriprep_wf/single_subject_CC0006_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin6Asym/registration/ants_t1_to_mniComposite.h5
Standard output:

Standard error:
Exception caught during reference file reading

itk::ExceptionObject (0x2bea480)
Location: “unknown”
File: /tmp/ANTs/build/ITKv4/Modules/IO/PNG/src/itkPNGImageIO.cxx
Line: 107
Description: itk::ERROR: PNGImageIO(0x2ae2300): PNGImageIO failed to read header for file:
Reason: fread read only 0 instead of 8

file /home/fmriprep/.cache/templateflow/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-02_T1w.nii.gz
Segmentation fault (core dumped)
Return code: 139"

For the records here is the command I used to run fmriprep 20.0.7 in a singularity container.

SINGULARITYENV_TEMPLATEFLOW_HOME=~/.cache/templateflow
SINGULARITYENV_TEMPLATEFLOW_USE_DATALAD=false

singularity run -B $HOME/.cache:/home/fmriprep/.cache --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.7.img
/data/dorit/data/fmriprep20.0.7/BIDS
/data/dorit/data/fmriprep20.0.7/BIDS/derivatives
participant
-w /data/dorit/data/fmriprep20.0.7/work
–participant-label $S
–fs-license-file /data/dorit/license.txt
–output-spaces MNI152Lin:res-2 T1w fsaverage6
-t rest
–ignore slicetiming
–cifti-output
–medial-surface-nan
–skip_bids_validation
–low-mem --mem-mb 24000 --nthreads 3 --notrack

I’m surprised that the reference error is related to MNI152NLin now and not an issue about either finding the MNI152Lin space nor the MNI152NLin one.

Let me know if you have suggestions. Otherwise I will try to run it now with the MNI-Nonlin space and see whether I get the same issue or whether this error does go back to the MNILin space.

Thanks,
Dorit

8

Could you try removing the SINGULARITYENV variables and the -B $HOME/.cache:/home/fmriprep/.cache binding? That looks potentially problematic.

9

Sure thing. Will run another one.

10

Started it with the following command:

singularity run --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.7.img
/data/dorit/data/fmriprep20.0.7/BIDS
/data/dorit/data/fmriprep20.0.7/BIDS/derivatives
participant
-w /data/dorit/data/fmriprep20.0.7/work
–participant-label $S
–fs-license-file /data/dorit/license.txt
–output-spaces MNI152NLin6Asym MNI152Lin:res-2 T1w fsaverage6
-t rest
–ignore slicetiming
–cifti-output
–medial-surface-nan
–skip_bids_validation
–low-mem --mem-mb 24000 --nthreads 3 --notrack

Got the following error straight away (again not leading to crash):
OSError: [Errno 30] Read-only file system: ‘/home/fmriprep/.cache/templateflow/tpl-MNI152Lin/tpl-MNI152Lin_res-02_T1w.nii.gz’

Crash says: "
Node: fmriprep_wf.single_subject_CC0006_wf.anat_preproc_wf.anat_reports_wf.tf_select
Working directory: /data/dorit/data/fmriprep20.0.7/work/fmriprep_wf/single_subject_CC0006_wf/anat_preproc_wf/anat_reports_wf/_template_MNI152Lin/tf_select

Node inputs:

atlas =
resolution = [1]
template = MNI152Lin
template_spec = {‘resolution’: [1]}

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 344, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 397, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/smriprep/interfaces/templateflow.py”, line 59, in _run_interface
**specs
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 68, in get
_s3_get(filepath)
File “/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py”, line 217, in _s3_get
with filepath.open(“wb”) as f:
File “/usr/local/miniconda/lib/python3.7/pathlib.py”, line 1176, in open
opener=self._opener)
File “/usr/local/miniconda/lib/python3.7/pathlib.py”, line 1030, in _opener
return self._accessor.open(self, flags, mode)
OSError: [Errno 30] Read-only file system: ‘/home/fmriprep/.cache/templateflow/tpl-MNI152Lin/tpl-MNI152Lin_res-01_T1w.nii.gz’"

11

Oh, due to singularity’s permission issues you won’t be able to write to any directories owned by the fmriprep user. Are you setting the $HOME variable within the container to /home/fmriprep? I think the default singularity settings mount your local $HOME and use it as your home directory within.

12

@mgxd
Not sure what you mean. $HOME mounts to my user’s home, and with that fmriprep can write by user owned directories (I do get outputs for the other spaces and can write to the working dir as well).

13

Then maybe your $SINGULARITYENV_TEMPLATEFLOW_HOME is set to /home/fmriprep?

The crashes you are seeing:

OSError: [Errno 30] Read-only file system: ‘/home/fmriprep/.cache/templateflow/tpl-MNI152Lin/tpl-MNI152Lin_res-01_T1w.nii.gz’"

are occurring because there’s something wonky with the $TEMPLATEFLOW_HOME and/or $HOME environmental variables.

14

I haven’t explicitly set the $SINGULARITYENV_TEMPLATEFLOW_HOME this time (double-checked by running the same test again in a new clean terminal)…

15

We have tried a previous solution that had worked for earlier versions at some point and it was successful!

unset PYTHONPATH;
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
singularity run -B $HOME/.cache:/home/fmriprep/.cache --cleanenv /data/singularity_images/poldracklab_fmriprep_20.0.7.img …
…

1 Like