Bus error - FMRIPREP

sofie_vdb · July 17, 2019, 12:39pm

Dear all,

Recently, I tried to run FMRIPREP (v1.1.8) on our cluster, and I ran into a bus error while running “bold_transform” (fmri.interfaces.itk.MultiApplyTransforms). Even when increasing vmem to over 150GB and using the --use-plugin argument to specify Nipype’s old MultiProc, the error persevered.

More concretely, the command and the produced error file (using 50GB vmem and --use-plugin are the following:

fmriprep $DATADIR $OUTDIR participant --fs-license-file $FS -w $WORKDIR --participant_label $LABEL -v -vvv --write-graph --fs-no-reconall --use-plugin “$HOMEDIR/mem_error3.yaml”

mem_error3.yaml

plugin: LegacyMultiProc
plugin_args: {maxtasksperchild: 1, memory_gb: 50, n_procs: 4, raise_insufficient: false}

/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
Fatal Python error: Bus error

Thread 0x00007f453efa3700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/selectors.py”, line 376 in select
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 843 in communicate
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 899 in get_dependencies
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1034 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f453f7a4700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/selectors.py”, line 376 in select
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 843 in communicate
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/utils/filemanip.py”, line 899 in get_dependencies
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1034 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00007f453ffa5700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/volumeutils.py”, line 820 in _write_data
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/volumeutils.py”, line 766 in array_to_file
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/arraywriters.py”, line 562 in to_fileobj
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/analyze.py”, line 1092 in to_file_map
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/filebasedimages.py”, line 334 in to_filename
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 271 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f459dd23740 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 295 in wait
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/_base.py”, line 427 in result
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/_base.py”, line 586 in result_iterator
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 143 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 635 in _run_command
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 555 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 471 in run
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/plugins/legacymultiproc.py”, line 68 in run_node
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/pool.py”, line 119 in worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/process.py”, line 93 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/process.py”, line 258 in _bootstrap
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/spawn.py”, line 118 in _main
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/forkserver.py”, line 231 in _serve_one
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/forkserver.py”, line 196 in main
File “”, line 1 in
slurmstepd: error: *** JOB 356291 ON node3113.skitty.os CANCELLED AT 2019-07-16T19:27:23 ***

I would appreciate your feedback and help solving this error.

Best,

Sofie

effigies · July 17, 2019, 12:48pm

Hi,

This is an issue resulting from attempting to overwrite an uncompressed file with its own data… For the moment, the fix is to compress your input NIfTI files in your original dataset, e.g.,

find /path/to/bids/data -name "*.nii" -exec gzip -n "{}" ";"

We should prioritize fixing this so users don’t need to compress their data.

Related: https://github.com/poldracklab/fmriprep/issues/1555

Best,
Chris

sofie_vdb · July 18, 2019, 10:39am

Dear Chris,

Thank you for your fast reply!
However, the original input dataset (e.g. sub-03/ses-1/func/*.nii.gz) was already compressed.
Is there another possible reason for the error?
The previous step in the FMRIPREP pipeline was “bold_split”, with as output separate .nii files (900 volumes/session; two sessions in total).

The same error occurs when running only 1 session:

/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/bids/grabbids/init.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
warnings.warn(“grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8”, FutureWarning)
Fatal Python error: Bus error

Thread 0x00007f586f7fe700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/selectors.py”, line 376 in select
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 843 in communicate
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 839 in run_command
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1035 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f586ffff700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/selectors.py”, line 376 in select
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 1514 in _communicate
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py”, line 843 in communicate
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 839 in run_command
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 1035 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/niworkflows/interfaces/fixes.py”, line 24 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 262 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Current thread 0x00007f5874e5e700 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/volumeutils.py”, line 820 in _write_data
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/volumeutils.py”, line 766 in array_to_file
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/arraywriters.py”, line 562 in to_fileobj
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/analyze.py”, line 1092 in to_file_map
File “/apps/gent/CO7/skylake-ib/software/NiBabel/2.3.0-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nibabel/filebasedimages.py”, line 334 in to_filename
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 271 in _applytfms
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 56 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/thread.py”, line 69 in _worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 864 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 916 in _bootstrap_inner
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 884 in _bootstrap

Thread 0x00007f58d2bdc740 (most recent call first):
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/threading.py”, line 295 in wait
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/_base.py”, line 427 in result
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/concurrent/futures/_base.py”, line 586 in result_iterator
File “/apps/gent/CO7/skylake-ib/software/FMRIprep/1.1.8-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/fmriprep/interfaces/itk.py”, line 143 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/interfaces/base/core.py”, line 522 in run
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 635 in _run_command
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 555 in _run_interface
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py”, line 471 in run
File “/apps/gent/CO7/skylake-ib/software/Nipype/1.1.3-foss-2018b-Python-3.6.6/lib/python3.6/site-packages/nipype/pipeline/plugins/legacymultiproc.py”, line 68 in run_node
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/pool.py”, line 119 in worker
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/process.py”, line 93 in run
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/process.py”, line 258 in _bootstrap
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/spawn.py”, line 118 in _main
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/forkserver.py”, line 231 in _serve_one
File “/apps/gent/CO7/skylake-ib/software/Python/3.6.6-foss-2018b/lib/python3.6/multiprocessing/forkserver.py”, line 196 in main
File “”, line 1 in

Best,

Sofie

effigies · July 18, 2019, 6:57pm

Ah, thanks, I see the problem. It still exists now, so I’ll need to submit a fix, and then once we release you can upgrade.

If you or your sysadmin is willing to patch it live, what you’ll need is to apply this patch:

diff --git a/fmriprep/interfaces/itk.py b/fmriprep/interfaces/itk.py
index 46f737fe..3dfcb470 100644
--- a/fmriprep/interfaces/itk.py
+++ b/fmriprep/interfaces/itk.py
@@ -262,7 +262,7 @@ def _applytfms(args):
     runtime = xfm.run().runtime
 
     if copy_dtype:
-        nii = nb.load(out_file)
+        nii = nb.load(out_file, mmap=False)
         in_dtype = nb.load(in_file).get_data_dtype()
 
         # Overwrite only iff dtypes don't match

sofie_vdb · July 19, 2019, 12:18pm

Thank you. I will get in touch with the admin, and I’ll keep you up to date.