antsBrainExtraction failure? Freesurfer does fine

fmriprep
#10

Just ran into another subject whose segmentation has a similar problem (see .html report screenshot below). Still using fmriprep 1.2.6-1, for consistency with subjects run previously. This participant has 2 T1-weighted scans, one of which is significantly better than the other.

Freesurfer run separately on only the good T1-weighted image produces a fine brainmask (see images below). I can only assume this has to do with the custom ants/freesurfer method for creating a brainmask used in fmriprep?


#11

An even worse case!

Freesurfer on its own does fine with this person.

#12

Hello Ursula,

Did you figure out how to fix this problem. I’m having the similar problem. Please, let’s me know.

#13

Hi Thomas,

What I think worked, but I’m not totally positive (but looks like it from the .html output reports and digging into the output files) was running the person through Freesurfer separately, feeding only the good T1w image to Freesurfer, and then copying that person’s Freesurfer output into the fmriprep freesurfer directory, and running fmriprep with that pre-existing directory. Here’s the output from the person above when I do this instead. Maybe this will help you?

This leads me to think that it’s either a) the intensity normalization steps that are done with ANTs before freesurfer or b) feeding 2 images instead of 1 to freesurfer or c) the joint use of ANTs and FS to get a brainmask, I’ve been told that ANTs is great but that it’s built for perfect data, and our kid data is certainly not perfect.

#14

Hi all,

Could you try the new pediatric templates of TemplateFlow? We are testing this new feature on fMRIPrep version 1.4.1rc2

In principle, it should be as easy as adding the following arguments:

--skull-strip-template MNIInfant:cohort-3 --output-spaces MNIInfant:cohort-3:res-2 <other-spaces>

Please replace MNIInfant with MNIPediatricAsym if those fit better your age range (MNIInfant is 0-4.5yr and MNIPediatricAsym goes from 4.5yo through 18yo).

Please replace cohort-3 with the age range that better approximates your application (MNIInfant, MNIPediatricAsym)

1 Like
#15

Hmmm. I’m getting a problem with templateflow, error below. I feel like I’ve seen stuff like this on Neurostars, but searching didn’t yield anything useful.

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
190607-15:01:09,47 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.4.1rc2:
      * BIDS dataset path: /mnt.
      * Participant list: ['CBPD0142'].
      * Run identifier: 20190607-150108_7a1900d7-3648-40a8-af17-3351dc673405.
    
Downloading https://templateflow.s3.amazonaws.com/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 608, in build_workflow
    work_dir=str(work_dir),
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 259, in init_fmriprep_wf
    use_syn=use_syn,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 551, in init_single_subject_wf
    skull_strip_template=skull_strip_template,
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 230, in init_anat_preproc_wf
    normalization_quality='precise' if not debug else 'testing')
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/anat/ants.py", line 188, in init_brain_extraction_wf
    tpl_target_path = get_template(in_template, **template_spec)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 139, in _s3_get
    with filepath.open('wb') as f:
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1176, in open
    opener=self._opener)
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1030, in _opener
    return self._accessor.open(self, flags, mode)
OSError: [Errno 30] Read-only file system: '/opt/templateflow/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz'

Command is:


unset PYTHONPATH;
echo 'job is running'
singularity run --cleanenv -B ${BIDS_folder}:/mnt ${tools_dir}/fmriprep-1-4-1rc2.simg \
/mnt/ /mnt/derivatives/fmriprep_test \
participant \
-w /tmp/utooley${subject} \
--participant-label ${subject} \
--fs-license-file $HOME/license.txt \
--skull-strip-template MNIPediatricAsym:cohort-2 \
--output-spaces MNIPediatricAsym:cohort-2:res-2 T1w template fsaverage5 \
--nthreads 1 \
fMRIPrep: TemplateFlow fails with OSError: [Errno 30] Read-only file system
#16

Can you try adding (before the singularity run call):

export SINGULARITYENV_TEMPLATEFLOW_HOME=/mnt/derivatives/templateflow

Assuming that your singularity installation automatically binds $HOME, you probably want to use a different path other than/mnt/derivatives.

#17

@Ursula_Tooley did that work? I was writing on my cellphone, but now I can give some more context.

We have created a repository of templates called TemplateFlow to aid fMRIPrep (or any other tool, it’s totally open) with retrieving brain templates just giving an identifier. Templates can be navigated via the Python client (after pip install templateflow):

>>> from templateflow.api import templates, get
>>> templates()
... list of available template ids ...
>>> get('MNIInfant', suffix='T1w', resolution=1)
... list of T1w images of resolution 1mm^3 under the MNIInfant template ...

Or using datalad:

$ datalad install -r ///templateflow
$ cd templateflow/
$ datalad get tpl-MNIInfant/cohort-*/tpl-MNIInfant_cohort-*_res-1_T1w.nii.gz

Templates with several averages corresponding to age ranges or any other criteria are split by “cohorts”. The cohorts are annotated in the top template_description.json file (e.g., MNIInfant’s template_description.json).

The caveat is that TemplateFlow needs some location where all these templates are pulled (corresponding to the environment variable TEMPLATEFLOW_HOME), however, singularity images are -in general- read-only. In other words, we need users to prescribe that space. There are two ways of doing that:

  1. Overloading the default TEMPLATEFLOW_HOME, which points to /opt/templateflow, via binding:
singularity run -B $HOME/.cache/templateflow:/opt/templateflow
  1. Rewriting the TEMPLATEFLOW_HOME environment variable to some directory with write permissions:
export SINGULARITYENV_TEMPLATEFLOW_HOME=/scratch/templateflow
singularity run -B /host/workdir:/scratch fmriprep.simg /scratch/bids /scratch/bids/derivatives participant <...>

My previous post proposed solution 2, but solution 1 is equally valid. Please note how we can define environment variables that traverse the --cleanenv firewall and are thus defined in the container (i.e., using the SINGULARITYENV_ prefix).

1 Like
#18

Looking at releases on Github, looks like I should now be trying 1.4.1rc4, right? Sorry for the delay responding to this, was traveling, but I appreciate the detailed explanation!

Setting SINGULARITYENV_TEMPLATEFLOW_HOME beforehand didn’t seem to make a difference, I got the same read-only error, but it looks like you’ve debugged templateflow since the rc2 release. Do I need to do anything special using 1.4.1rc4?

#19

Does your singularity installation bind $HOME automatically?

If so, it should work out off the box, without any SINGULARITYENV_* set before hand.

If $HOME is not bound by default, then the SINGULARITYENV_TEMPLATEFLOW_HOME export is a must have.

You can see whether singularity is correctly setting it if you run:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/scratch/templateflow
singularity exec -B /host/workdir:/scratch fmriprep.simg env | grep TEMPLATEFLOW

All of this definitely requires 1.4.1rc4.

#20

Hmmm. Sorry again for the delay, I just got back to lab after two months of travel. I am still getting the same type of error with templateflow (below), running with 1.4.1rc4. I think this might be because our singularity installation doesn’t have a /scratch dir in the container. Not sure why, I assumed that was bundled with the container? Am I missing something? Some googling yielded that probably file system overlay is not enable on my Linux cluster, or the particular singularity configuration is different…?

Running the command above yields:

WARNING: Skipping user bind, non existent bind point (directory) in container: '/scratch'
TEMPLATEFLOW_HOME=/scratch/templateflow

The error:

 Running fMRIPREP version 1.4.1rc2:
      * BIDS dataset path: /mnt.
      * Participant list: ['CBPD0142'].
      * Run identifier: 20190716-191532_434bd2ab-c16e-46da-96d2-5ba907b1ce6d.
    
Downloading https://templateflow.s3.amazonaws.com/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 608, in build_workflow
    work_dir=str(work_dir),
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 259, in init_fmriprep_wf
    use_syn=use_syn,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 551, in init_single_subject_wf
    skull_strip_template=skull_strip_template,
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 230, in init_anat_preproc_wf
    normalization_quality='precise' if not debug else 'testing')
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/anat/ants.py", line 188, in init_brain_extraction_wf
    tpl_target_path = get_template(in_template, **template_spec)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 139, in _s3_get
    with filepath.open('wb') as f:
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1176, in open
    opener=self._opener)
  File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1030, in _opener
    return self._accessor.open(self, flags, mode)
OSError: [Errno 30] Read-only file system: '/opt/templateflow/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz'
#21

Alright, I found our singularity config file on the cluster! Attached below, I think it’s to do with BIND PATH not having /scratch?
singularity.txt (6.5 KB)

But, using export SINGULARITYENV_TEMPLATEFLOW_HOME=/opt/templateflow singularity run -B ${HOME}/.templateflow:/opt/templateflow did enable it to run. But I run into this:

      * BIDS dataset path: /mnt.
      * Participant list: ['CBPD0142'].
      * Run identifier: 20190716-194334_1d8cd3e6-a736-4ea7-b96c-9f33e358015d.
    
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 608, in build_workflow
    work_dir=str(work_dir),
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 259, in init_fmriprep_wf
    use_syn=use_syn,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 551, in init_single_subject_wf
    skull_strip_template=skull_strip_template,
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 305, in init_anat_preproc_wf
    template_list=vol_spaces)
  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/norm.py", line 103, in init_anat_norm_wf
    'This is embarrassing - custom templates are not (yet) supported.'
NotImplementedError: This is embarrassing - custom templates are not (yet) supported.Please make sure none of the options already available via TemplateFlow fit your needs.

Am I missing a template, or is this a variation on the same error? I’m using --skull-strip-template MNIPediatricAsym:cohort-2 \ --output-spaces MNIPediatricAsym:res-2:cohort-2 T1w template fsaverage5 \

#22

You seem to have HOME bound by default.

Then, it should be as easy as:

singularity exec --cleanenv fmriprep.simg python -c "from templateflow import api; print(api.get('MNIPediatricAsym', cohort=2, resolution=1, suffix='T1w'))"
#23

Does that imply that I shouldn’t need this?

export SINGULARITYENV_TEMPLATEFLOW_HOME=/opt/templateflow
singularity run -B ${HOME}/.templateflow:/opt/templateflow

Because it seems I do, otherwise I get the read-only error.

I see templates being downloaded when I use that, but I still get the NotImplementedError: This is embarrassing - custom templates are not (yet) . After some experimenting, I can get past that error by stripping down my requested output spaces: removing --skull-strip-template MNIPediatricAsym:cohort-2, which seemed to be problematic, and then taking out template from the list of output spaces. I just now realized that template has been replaced by MNI152NLin2009cAsym, and that I need --skull-strip-template MNIPediatricAsym:res-2:cohort-2 and not just --skull-strip-template MNIPediatricAsym:cohort-2. It seems like for skull-strip-template the resolution has to be specified, which is a bit tricky as I couldn’t find that in the docs. On purpose, I assume?

Anyways, I’m running now, will see if this fixes the skull-stripping issue!

#24

Alrighty, it’s working, but ran into a space error when downloading templateflow templates–not sure why. Although it’s not clear from the output, I assume they are being downloaded into /home/fmriprep/.cache/templateflow in the container. Is that true? It’d be nice to see where they were being written to-- they don’t seem to be written permanently anywhere outside the container that I can find (there is no /home/fmriprep folder outside the container), so I can’t figure out where they might be hitting mem limits…

190717-14:30:35,630 nipype.workflow WARNING:
	 Some nodes demand for more threads than available (1).
190717-14:30:36,317 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_CBPD0142_wf.func_preproc_task_rest_run_01_wf.func_derivatives_wf.raw_sources" in "/tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/func_preproc_task_rest_run_01_wf/func_derivatives_wf/raw_sources".
190717-14:30:36,319 nipype.workflow INFO:
	 [Node] Running "raw_sources" ("nipype.interfaces.utility.wrappers.Function")
190717-14:30:36,324 nipype.workflow INFO:
	 [Node] Finished "fmriprep_wf.single_subject_CBPD0142_wf.func_preproc_task_rest_run_01_wf.func_derivatives_wf.raw_sources".
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-brain_mask.nii.gz
156B [00:00, 7.59kB/s]                   
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_T1w.nii.gz
 77%|███████▋  | 10.3k/13.4k [00:01<00:00, 9.04kB/s]190717-14:30:39,550 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_CBPD0142_wf.anat_preproc_wf.anat_norm_wf.tpl_select" (/tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/tpl_select)
190717-14:30:39,553 nipype.workflow ERROR:
	 Node tpl_select.a1 failed to run on host compute-0-6.local.
190717-14:30:39,605 nipype.workflow ERROR:
	 Saving crash info to /mnt/derivatives/fmriprep_test/fmriprep/sub-CBPD0142/log/20190717-143008_b5a04d6c-21ff-4750-97dd-f55d1db7766d/crash-20190717-143039-utooley-tpl_select.a1-f33a502c-3fbd-41b5-b197-e8aab6698327.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 144, in _s3_get
    f.write(data)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 316, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 472, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 563, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 375, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 144, in _run_interface
    out = function_handle(**args)
  File "<string>", line 5, in _get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py", line 50, in get_template_specs
    tpl_target_path = get_template(in_template, **template_spec)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 144, in _s3_get
    f.write(data)
OSError: [Errno 28] No space left on device

And it goes on in that vein for a while then continues running.

#25

Release 1.4.1 comes with an very much improved error message. Upgrading will help diagnose this issue.

It shouldn’t be problematic, although it could be needing to add a :res-1 right after cohort-2 (i.e., MNIPediatricAsym:cohort-2:res-1).

yep, this is not allowed anymore in --output-spaces (please note the plural form of spaces).

I’m surprised it does not select res-1 automatically by default. If you want to quickly test, you can use res-2 here, but make sure you change to res-1 in production.

Yes, I was implying exactly that (although in combination with --cleanenv for the singularity run call and without the export SINGULARITYENV_TEMPLATEFLOW_HOME).

Without any export previously done, can you post the output of:

singularity exec --cleanenv fmriprep.simg python -c "from templateflow.conf import TF_HOME; print(TF_HOME)"
#26

Sorry for the confusion–the first couple times I tried this, I thought I was using 1.4.1rc4, but turns out underneath I was still actually using 1.4.1rc2… :woman_facepalming::woman_facepalming:t4: so very very sorry for the waste of time! Several of these errors above were fixed by that change, including the trouble with using --skull-strip-template MNIPediatricAsym:cohort-2 vs --skull-strip-template MNIPediatricAsym:res-2:cohort-2. It does seem to select res-1 by default now.

However, the error that persists is the OSError: [Errno 28] No space left on device, which happens after the boilerplate and after downloading some templates successfully at the beginning. Are they downloaded to somewhere else later in the workflow in tpl_select? I assume it doesn’t have anything to do with nodes needing more threads than are available?

singularity exec --cleanenv fmriprep.simg python -c "from templateflow.conf import TF_HOME; print(TF_HOME)"
yields
/home/fmriprep/.cache/templateflow
But that dir doesn’t exist outside the container that I can tell:

(base) [utooley@chead Ursula]$ ls /home/fmriprep/
ls: cannot access /home/fmriprep/: No such file or directory
(base) [utooley@chead Ursula]$ ls /home/utooley/fmriprep
ls: cannot access /home/utooley/fmriprep: No such file or directory

Full log trace just in case it’s helpful:

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
	1: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.


        Summary:                   Available Tasks:                       Available Modalities: 
        1175 Files, 32.59GB        rest                                   T1w                   
        105 - Subjects             number                                 T2w                   
        1 - Session                nback                                  bold                  
                                   TODO: full task name for rest          events                
                                   TODO: full task name for number                              


	If you have any questions, please post on https://neurostars.org/tags/bids.

190717-16:30:25,949 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.4.1rc4:
      * BIDS dataset path: /mnt.
      * Participant list: ['CBPD0142'].
      * Run identifier: 20190717-163025_cd31c52c-0946-4a90-8ff3-cb475f521884.
    
Downloading https://templateflow.s3.amazonaws.com/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz
4.44kB [00:00, 9.91kB/s]                                                                                                                
Downloading https://templateflow.s3.amazonaws.com/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_desc-brain_mask.nii.gz
160B [00:00, 7.58kB/s]                                                                                                                  
190717-16:30:29,818 nipype.workflow IMPORTANT:
	 Creating bold processing workflow for "/mnt/sub-CBPD0142/func/sub-CBPD0142_task-rest_run-01_bold.nii.gz" (0.07 GB / 67 TRs). Memory resampled/largemem=0.27/0.34 GB.
190717-16:30:29,828 nipype.workflow IMPORTANT:
	 No single-band-reference found for sub-CBPD0142_task-rest_run-01_bold.nii.gz
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz
1.71kB [00:00, 10.3kB/s]                                                                                                                
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
29.0B [00:00, 4.49kB/s]                                                                                                                 
190717-16:30:32,562 nipype.workflow IMPORTANT:
	 Slice-timing correction will be included.
190717-16:30:32,586 nipype.workflow WARNING:
	 SDC: no fieldmaps found or they were ignored (/mnt/sub-CBPD0142/func/sub-CBPD0142_task-rest_run-01_bold.nii.gz).
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-carpet_dseg.nii.gz
441B [00:00, 9.65kB/s]                                                                                                                  
190717-16:30:33,831 nipype.workflow IMPORTANT:
	 Creating BOLD surface-sampling workflow.
190717-16:30:39,108 nipype.workflow IMPORTANT:
	 Works derived from this fMRIPrep execution should include the following boilerplate:


Results included in this manuscript come from preprocessing
performed using *fMRIPrep* 1.4.1rc4
(@fmriprep1; @fmriprep2; RRID:SCR_016216),
which is based on *Nipype* 1.2.0
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: A total of 2 T1-weighted (T1w) images were found within the input
BIDS dataset.
All of them were corrected for intensity non-uniformity (INU)
with `N4BiasFieldCorrection` [@n4], distributed with ANTs 2.2.0 [@ants, RRID:SCR_004757].
The T1w-reference was then skull-stripped with a *Nipype* implementation of
the `antsBrainExtraction.sh` workflow (from ANTs), using MNIPediatricAsym
as target template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `fast` [FSL 5.0.9, RRID:SCR_002823,
@fsl_fast].
A T1w-reference map was computed after registration of
2 T1w images (after INU-correction) using
`mri_robust_template` [FreeSurfer 6.0.1, @fs_template].
Brain surfaces were reconstructed using `recon-all` [FreeSurfer 6.0.1,
RRID:SCR_001847, @fs_reconall], and the brain mask estimated
previously was refined with a custom variation of the method to reconcile
ANTs-derived and FreeSurfer-derived segmentations of the cortical
gray-matter of Mindboggle [RRID:SCR_002438, @mindboggle].
Volume-based spatial normalization to two standard spaces (MNIPediatricAsym, MNI152NLin2009cAsym) was performed through
nonlinear registration with `antsRegistration` (ANTs 2.2.0),
using brain-extracted versions of both T1w reference and the T1w template.
The following templates were selected for spatial normalization:
*MNI's unbiased standard MRI template for pediatric data from the 4.5 to 18.5y age range* [@mnipediatricasym, RRID:SCR_008796; TemplateFlow ID: MNIPediatricAsym], *ICBM 152 Nonlinear Asymmetrical template version 2009c* [@mni152nlin2009casym, RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym].

Functional data preprocessing

: For each of the 1 BOLD runs found per subject (across all
tasks and sessions), the following preprocessing was performed.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
The BOLD reference was then co-registered to the T1w reference using
`bbregister` (FreeSurfer) which implements boundary-based registration [@bbr].
Co-registration was configured with nine degrees of freedom to account
for distortions remaining in the BOLD reference.
Head-motion parameters with respect to the BOLD reference
(transformation matrices, and six corresponding rotation and translation
parameters) are estimated before any spatiotemporal filtering using
`mcflirt` [FSL 5.0.9, @mcflirt].
BOLD runs were slice-time corrected using `3dTshift` from
AFNI 20160207 [@afni, RRID:SCR_005927].
The BOLD time-series, were resampled to surfaces on the following
spaces: *fsaverage5*.
The BOLD time-series (including slice-timing correction when applied)
were resampled onto their original, native space by applying
a single, composite transform to correct for head-motion and
susceptibility distortions.
These resampled BOLD time-series will be referred to as *preprocessed
BOLD in original space*, or just *preprocessed BOLD*.
The BOLD time-series were resampled into several standard spaces,
correspondingly generating the following *spatially-normalized,
preprocessed BOLD runs*: MNIPediatricAsym, MNI152NLin2009cAsym.
First, a reference volume and its skull-stripped version were generated
using a custom methodology of *fMRIPrep*.
Several confounding time-series were calculated based on the
*preprocessed BOLD*: framewise displacement (FD), DVARS and
three region-wise global signals.
FD and DVARS are calculated for each functional run, both using their
implementations in *Nipype* [following the definitions by @power_fd_dvars].
The three global signals are extracted within the CSF, the WM, and
the whole-brain masks.
Additionally, a set of physiological regressors were extracted to
allow for component-based noise correction [*CompCor*, @compcor].
Principal components are estimated after high-pass filtering the
*preprocessed BOLD* time-series (using a discrete cosine filter with
128s cut-off) for the two *CompCor* variants: temporal (tCompCor)
and anatomical (aCompCor).
tCompCor components are then calculated from the top 5% variable
voxels within a mask covering the subcortical regions.
This subcortical mask is obtained by heavily eroding the brain mask,
which ensures it does not include cortical GM regions.
For aCompCor, components are calculated within the intersection of
the aforementioned mask and the union of CSF and WM masks calculated
in T1w space, after their projection to the native space of each
functional run (using the inverse BOLD-to-T1w transformation). Components
are also calculated separately within the WM and CSF masks.
For each CompCor decomposition, the *k* components with the largest singular
values are retained, such that the retained components' time series are
sufficient to explain 50 percent of variance across the nuisance mask (CSF,
WM, combined, or temporal). The remaining components are dropped from
consideration.
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file.
The confound time series derived from head motion estimates and global
signals were expanded with the inclusion of temporal derivatives and
quadratic terms for each [@confounds_satterthwaite_2013].
Frames that exceeded a threshold of 0.5 mm FD or 1.5 standardised DVARS
were annotated as motion outliers.
All resamplings can be performed with *a single interpolation
step* by composing all the pertinent transformations (i.e. head-motion
transform matrices, susceptibility distortion correction when available,
and co-registrations to anatomical and output spaces).
Gridded (volumetric) resamplings were performed using `antsApplyTransforms` (ANTs),
configured with Lanczos interpolation to minimize the smoothing
effects of other kernels [@lanczos].
Non-gridded (surface) resamplings were performed using `mri_vol2surf`
(FreeSurfer).


Many internal operations of *fMRIPrep* use
*Nilearn* 0.5.2 [@nilearn, RRID:SCR_001362],
mostly within the functional processing workflow.
For more details of the pipeline, see [the section corresponding
to workflows in *fMRIPrep*'s documentation](https://fmriprep.readthedocs.io/en/latest/workflows.html "FMRIPrep's documentation").


### References


pandoc-citeproc: reference mnipediatricasym not found
190717-16:30:52,989 nipype.workflow WARNING:
	 Some nodes demand for more threads than available (1).
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_desc-brain_mask.nii.gz
156B [00:00, 10.2kB/s]                                                                                                                  
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-01_T1w.nii.gz
 59%|███████████████████████████████████████████████████████▍                                      | 7.89k/13.4k [00:00<00:00, 11.3kB/s]
190717-16:30:56,566 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_CBPD0142_wf.anat_preproc_wf.anat_norm_wf.tpl_select" (/tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/tpl_select)
190717-16:30:56,569 nipype.workflow ERROR:
	 Node tpl_select.a1 failed to run on host chead.uphs.upenn.edu.
190717-16:30:56,624 nipype.workflow ERROR:
	 Saving crash info to /mnt/derivatives/fmriprep_test/fmriprep/sub-CBPD0142/log/20190717-163025_cd31c52c-0946-4a90-8ff3-cb475f521884/crash-20190717-163056-utooley-tpl_select.a1-ead924f9-336a-47bd-b588-a9ac70bd705a.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 144, in _s3_get
    f.write(data)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 316, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 472, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 563, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 375, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 144, in _run_interface
    out = function_handle(**args)
  File "<string>", line 5, in _get_template
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/misc.py", line 50, in get_template_specs
    tpl_target_path = get_template(in_template, **template_spec)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 144, in _s3_get
    f.write(data)
OSError: [Errno 28] No space left on device

190717-16:30:58,821 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_CBPD0142_wf.func_preproc_task_rest_run_01_wf.bold_reference_wf.gen_ref" in "/tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/func_preproc_task_rest_run_01_wf/bold_reference_wf/gen_ref".
190717-16:30:58,823 nipype.workflow INFO:
	 [Node] Running "gen_ref" ("niworkflows.interfaces.registration.EstimateReferenceImage")
190717-16:31:01,310 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.310071:++ 3dvolreg: AFNI version=Debian-16.2.07~dfsg.1-5~nd16.04+1 (Jun 12 2017) [64-bit]
190717-16:31:01,310 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.310071:++ Authored by: RW Cox
190717-16:31:01,328 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:*+ WARNING:   If you are performing spatial transformations on an oblique dset, 
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:  such as /tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/func_preproc_task_rest_run_01_wf/bold_reference_wf/gen_ref/slice.nii.gz,
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:  or viewing/combining it with volumes of differing obliquity,
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:  you should consider running: 
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:     3dWarp -deoblique 
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903:  on this and  other oblique datasets in the same session.
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.328903: See 3dWarp -help for details.
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.329352:++ Oblique dataset:/tmp/utooleyCBPD0142/fmriprep_wf/single_subject_CBPD0142_wf/func_preproc_task_rest_run_01_wf/bold_reference_wf/gen_ref/slice.nii.gz is 14.095265 degrees from plumb.
190717-16:31:01,329 nipype.interface INFO:
	 stderr 2019-07-17T16:31:01.329928:++ Coarse del was 10, replaced with 7
#27

No need for that, very glad you found out!

This is bad - it seems your singularity installation automatically binds HOME, but it does not overwrite the $HOME env variable.

Okay, let’s bind your home dir from the host:

mkdir -p $HOME/.cache/templateflow
singularity exec -B $HOME/.cache/templateflow:/home/fmriprep/.cache/templateflow --cleanenv fmriprep.simg python -c "from templateflow.conf import TF_HOME; print(TF_HOME)"

That should give you /home/fmriprep/.cache/templateflow.

Then try:

singularity exec -B $HOME/.cache/templateflow:/home/fmriprep/.cache/templateflow --cleanenv fmriprep.simg python -c "from templateflow import api; print(api.get('MNIPediatricAsym', cohort=2, resolution=1, suffix='T1w'))"

After that, you should be able to verify that $HOME/.cache/templateflow has pulled down that particular path (relative to your host’s $HOME, not the container’s).

#28

That gives me

WARNING: Skipping user bind, non existent bind point (directory) in container: '/home/fmriprep/.cache/templateflow'
/home/fmriprep/.cache/templateflow

However…using templateflow to get templates as below puts them there?

singularity exec -B $HOME/.cache/templateflow:/home/fmriprep/.cache/templateflow --cleanenv /data/picsl/mackey_group/tools/singularity/fmriprep-1-4-1rc4.simg python -c  "from templateflow import api; print(api.get('MNIPediatricAsym', cohort=2, resolution=1, suffix='T1w'))"
WARNING: Skipping user bind, non existent bind point (directory) in container: '/home/fmriprep/.cache/templateflow'
Downloading https://templateflow.s3.amazonaws.com/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz
4.44kB [00:00, 11.3kB/s]                                                                             
/home/fmriprep/.cache/templateflow/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz

But they don’t exist outside the container (which I’m not sure they ever should, but I am wondering how this relates to the mem error).

#29

Okay, true. Sorry about that. Let’s change that to (be careful with the rm in the beginning, make sure it doesn’t destroy anything you want to keep):

rm -fr $HOME/.cache/templateflow
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/templateflow
singularity exec -B $HOME/.cache:/home/fmriprep --cleanenv fmriprep.simg python -c "from templateflow.conf import TF_HOME; print(TF_HOME)"

Then the second part can be kept as is if you are still using the same bash session:

singularity exec -B $HOME/.cache:/home/fmriprep/ --cleanenv /data/picsl/mackey_group/tools/singularity/fmriprep-1-4-1rc4.simg python -c  "from templateflow import api; print(api.get('MNIPediatricAsym', cohort=2, resolution=1, suffix='T1w'))"

You should see that $HOME/.cache/templateflow/tpl-MNIPediatricAsym/cohort-2/tpl-MNIPediatricAsym_cohort-2_res-1_T1w.nii.gz exists on your host (outside the container)