This could be a memory issue. If you are running this via Singularity or natively make sure your system has at least 8GB of RAM. If you are running this via Docker on Windows or Mac OS X make sure Docker has access to enough of your local machine RAM (the default is to expose only 2GB of the host RAM which is way too little).
Thank you very much for your kind reply.
I run fmriprep via Docker on Ubuntu 14.04 (128GB RAM, 22C/44T CPU). I now retry the same analysis using -m option of docker run.
I also used --ants-nthreads as I can use all threads. Although I’m not sure the precise effect of this option, I delete this option this time.
Just to confirm - it’s the same dataset/subject that previously worked with FMRIPREP 0.2.0?
Yes, completely same dataset/subject and computing environment.
I retried, but it seems that I encountered the same error in probably different subject. I would appreciate if you could give me any suggestions.
Nothing comes up to my mind at the moment. Any chance you could share the data (probably t1s would be enough) of a failing subject so we could try to replicate the problem? chrisgor@stanford.edu
Thank you for your kind remark. But, I cannot send my own data due to ethical problem.
I’m reanalyzing the same data using bids/fmriprep now (previous errors were in poldracklab/fmriprep). Although I thought two docker containers are always same since I can confirm version in fmriprep/Dockerfile as 0.3.1 (https://github.com/BIDS-Apps/fmriprep/blob/master/Dockerfile), it seems that there are some differences (e.g. --no-freesurfer option). (I cannot confirm the version of bids/fmriprep using -v option. It showed only ‘vdev’ different from poldracklab/fmriprep.)
I hope it will work well.
The bids/fmriprep images have not been updated (are still at 0.2.0). Can you replicate this error on any public data? For example ds000005 or ds000114 from OpenfMRI.org?
I understand bids/fmriprep is still 0.2.0.
I’m trying to confirm the error using ds001 now.
Same error happened in ds001/sub-05, sub-14. Since I cannot attach crash.pklz file, I email it directly to you.
I wonder if the issue is with ants multithreading (you have a very powerful system with 44 cores!) Could you try running the analysis with --ants-nthreads 12 or --ants-nthreads 1?
I second this issue, as I am getting similar error messages in two other fMRI datasets, when I run the bids/fmriprep app with Docker.
I am running this on a single thread, on Mac OS (Version: 10.11.16; 16GB Ram, 8T CPU)
Node Name: workflow_enumerator.03.t1w_preprocessing.ANTsBrainExtraction.Ants_T1_Brain_Extraction
Working Directory: /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/ANTsBrainExtraction/Ants_T1_Brain_Extraction
Inputs:
anatomical_image: /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/CorrectINU/sub-03_T1w_corrected.nii.gz
args:
brain_probability_mask: /niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumProbabilityMask.nii.gz
brain_template: /niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz
debug: False
dimension: 3
environ: {‘NSLOTS’: ‘4’}
extraction_registration_mask: /niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumRegistrationMask.nii.gz
generate_report: True
ignore_exception: False
image_suffix: nii.gz
keep_temporary_files:
num_threads: 4
out_prefix: highres001_
out_report: report.html
terminal_output: stream
use_floatingpoint_precision: 1
use_random_seeding:
Traceback:
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/pipeline/plugins/multiproc.py”, line 52, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py”, line 366, in run
self._run_interface()
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py”, line 476, in _run_interface
self._result = self._run_command(execute)
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/pipeline/engine/nodes.py”, line 607, in _run_command
result = self._interface.run()
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/interfaces/base.py”, line 1085, in run
runtime = self._run_wrapper(runtime)
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/interfaces/base.py”, line 1728, in _run_wrapper
runtime = self._run_interface(runtime)
File “/src/niworkflows/niworkflows/common/report.py”, line 45, in _run_interface
ReportCapableInterface, self).run_interface(runtime)
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/interfaces/base.py”, line 1762, in run_interface
self.raise_exception(runtime)
File “/usr/local/miniconda/lib/python3.5/site-packages/nipype/interfaces/base.py”, line 1686, in raise_exception
**runtime.dictcopy()))
RuntimeError: Command:
antsBrainExtraction.sh -a /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/CorrectINU/sub-03_T1w_corrected.nii.gz -m /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/ANTsBrainExtraction/Ants_T1_Brain_Extraction/T_template0_BrainCerebellumProbabilityMask.nii.gz -e /niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz -d 3 -f /niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumRegistrationMask.nii.gz -s nii.gz -o highres001 -q 1
Standard output:
The output directory "highres001" does not exist. Making it.
…
ERROR: command exited with nonzero status 137
Command: /opt/ants/antsRegistration -d 3 -u 1 -w [0.025,0.975] -o highres001_BrainExtractionPrior -r highres001_BrainExtractionInitialAffine.mat -z 1 --float 1 -x [/niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumRegistrationMask.nii.gz] -m MI[/niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz,highres001_N4Corrected0.nii.gz,1,32,Regular,0.25] -c [1000x500x250x100,1e-8,10] -t Rigid[0.1] -f 8x4x2x1 -s 4x2x1x0 -m MI[/niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz,highres001_N4Corrected0.nii.gz,1,32,Regular,0.25] -c [1000x500x250x100,1e-8,10] -t Affine[0.1] -f 8x4x2x1 -s 4x2x1x0 -m CC[/niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz,highres001_N4Corrected0.nii.gz,0.5,4] -m CC[highres001_BrainExtractionTemplateLaplacian.nii.gz,highres001_BrainExtractionLaplacian.nii.gz,0.5,4] -c [50x10x0,1e-9,15] -t SyN[0.1,3,0] -f 4x2x1 -s 2x1x0
Standard error:
Using antsBrainExtraction with the following arguments:
image dimension = 3
anatomical image = /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/CorrectINU/sub-03_T1w_corrected.nii.gz
extraction template = /niworkflows_data/ants_oasis_template_ras/T_template0.nii.gz
extraction reg. mask = /niworkflows_data/ants_oasis_template_ras/T_template0_BrainCerebellumRegistrationMask.nii.gz
extraction prior = /root/src/fmriprep/work/workflow_enumerator/03/t1w_preprocessing/ANTsBrainExtraction/Ants_T1_Brain_Extraction/T_template0_BrainCerebellumProbabilityMask.nii.gz
output prefix = highres001_
output image suffix = nii.gz
N4 parameters (pre brain extraction):
convergence = [50x50x50x50,0.0000001]
shrink factor = 4
B-spline parameters = [200]
Atropos parameters (extraction):
convergence = [3,0.0]
likelihood = Gaussian
initialization = kmeans[3]
mrf = [0.1,1x1x1]
use clock random seed = 1
bad det -1 v 1 u -1
bad det -1 v 1 u -1 new 1
/opt/ants/antsBrainExtraction.sh: line 105: 15386 Killed $cmd
Return code: 1
Interface BrainExtractionRPT failed to run.
How many cores are available on your system?
This is a MBP with 4 cores and 16GB RAM
In your case the process was Killed, but yuki1209 reported Segmentation fault - those can be unrelated. Are you assigning enough memory to Docker on your Mac? See Fmriprep 0.3.0 or 0.3.1, error in t1w_preprocessing.ANTsBrainExtraction.Ants_T1_Brain_Extraction
Although I haven’t tried --ants-nthreads 12 or --ants-nthreads 1, I reanalyze ds001/sub-05 completely same settings. It worked well this time. Therefore, I’m not sure but some uncertain factors seem to be related with this matter in my case.
Strange. I recommend running some RAM diagnostics if you keep seeing this isssue.
I see. I’ll check it immediately.
Thanks Chris. You were right, it was a memory issue - which can be solved by allocating more memory to Docker.
1 Like
Although I haven’t conducted RAM diagnostics, fmriprep 0.3.1 with --ntreads 44 --ants-nthreads 12 options worked perfectly. I’m sorry that my error was only due to the resource balance between CPU and RAM.