Hello! I’m trying to use the lovely and useful fMRIPrep today, but I’m having trouble. I’m running fMRIPrep within a singularity container on a cluster, and I’m launching a separate container for each subject in my dataset. I’ve successfully used this same setup for 2 other datasets, so I’m unsure why I’m getting errors for my current dataset. I’ve tried to preprocess 3 subjects so far, and each gave me the same error.
- Here’s the crash report for one of these 3 subjects, subject 106:
Node: fmriprep_wf.single_subject_106_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /blue/akeil/veliebm/files/bopscanner/bids_attempt-1_logs/sub-106_work/work/fmriprep_wf/single_subject_106_wf/anat_preproc_wf/surface_recon_wf/autorecon1Node inputs:
FLAIR_file =
T1_files =
T2_file =
args =
big_ventricles =
brainstem =
directive = autorecon1
environ = {}
expert =
flags =
hemi =
hippocampal_subfields_T1 =
hippocampal_subfields_T2 =
hires =
mprage =
mri_aparc2aseg =
mri_ca_label =
mri_ca_normalize =
mri_ca_register =
mri_edit_wm_with_aseg =
mri_em_register =
mri_fill =
mri_mask =
mri_normalize =
mri_pretess =
mri_remove_neck =
mri_segment =
mri_segstats =
mri_tessellate =
mri_watershed =
mris_anatomical_stats =
mris_ca_label =
mris_fix_topology =
mris_inflate =
mris_make_surfaces =
mris_register =
mris_smooth =
mris_sphere =
mris_surf2vol =
mrisp_paint =
openmp = 1
parallel =
steps =
subject_id = recon_all
subjects_dir =
talairach =
use_FLAIR =
use_T2 =
xopts =Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 419, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 814, in _run_interface
self.raise_exception(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 745, in raise_exception
).format(**runtime.dictcopy())
RuntimeError: Command:
recon-all -autorecon1 -i /blue/akeil/veliebm/files/bopscanner/bids_attempt-1_logs/sub-106_work/work/fmriprep_wf/single_subject_106_wf/anat_preproc_wf/anat_validate/sub-106_T1w_ras_valid.nii -noskullstrip -openmp 1 -subjid sub-106 -sd /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer
Standard output:
Subject Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
Current Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
INFO: SUBJECTS_DIR is /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer
Actual FREESURFER_HOME /opt/freesurfer
Linux c25a-s1.ufhpc 3.10.0-1062.18.1.el7.x86_64 #1 SMP Wed Feb 12 14:08:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
‘/opt/freesurfer/bin/recon-all’ -> ‘/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/scripts/recon-all.local-copy’
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106mri_convert /blue/akeil/veliebm/files/bopscanner/bids_attempt-1_logs/sub-106_work/work/fmriprep_wf/single_subject_106_wf/anat_preproc_wf/anat_validate/sub-106_T1w_ras_valid.nii /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig/001.mgz
mri_convert.bin /blue/akeil/veliebm/files/bopscanner/bids_attempt-1_logs/sub-106_work/work/fmriprep_wf/single_subject_106_wf/anat_preproc_wf/anat_validate/sub-106_T1w_ras_valid.nii /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig/001.mgz
Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp
reading from /blue/akeil/veliebm/files/bopscanner/bids_attempt-1_logs/sub-106_work/work/fmriprep_wf/single_subject_106_wf/anat_preproc_wf/anat_validate/sub-106_T1w_ras_valid.nii…
TR=0.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (1, 0, 0)
j_ras = (0, 1, 0)
k_ras = (0, 0, 1)
writing to /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig/001.mgz…
#--------------------------------------------
#@# MotionCor Sun Nov 22 15:00:43 EST 2020
Found 1 runs
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig/001.mgz
Checking for (invalid) multi-frame inputs…
WARNING: only one run found. This is OK, but motion
correction cannot be performed on one run, so I’ll
copy the run to rawavg and continue.cp /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig/001.mgz /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/rawavg.mgz
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106
mri_convert /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/rawavg.mgz /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig.mgz --conform
mri_convert.bin /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/rawavg.mgz /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig.mgz --conform
Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp
reading from /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/rawavg.mgz…
TR=0.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (1, 0, 0)
j_ras = (0, 1, 0)
k_ras = (0, 0, 1)
changing data type from float to uchar (noscale = 0)…
MRIchangeType: Building histogram
Reslicing using trilinear interpolation
writing to /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig.mgz…mri_add_xform_to_header -c /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/transforms/talairach.xfm /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig.mgz /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/orig.mgz
INFO: extension is mgz
#--------------------------------------------
#@# Talairach Sun Nov 22 15:00:50 EST 2020
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mrimri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri
/opt/freesurfer/bin/mri_nu_correct.mni
–no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50
nIters 1
Id: mri_nu_correct.mni,v 1.27 2016/02/26 16:19:49 mreuter Exp
Linux c25a-s1.ufhpc 3.10.0-1062.18.1.el7.x86_64 #1 SMP Wed Feb 12 14:08:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Sun Nov 22 15:00:50 EST 2020
Program nu_correct, built from:
Package MNI N3, version 1.12.0, compiled by nicks@terrier (x86_64-unknown-linux-gnu) on 2015-06-19 at 01:25:34
/usr/bin/bc
tmpdir is ./tmp.mri_nu_correct.mni.15823
/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri
mri_convert orig.mgz ./tmp.mri_nu_correct.mni.15823/nu0.mnc -odt float
mri_convert.bin orig.mgz ./tmp.mri_nu_correct.mni.15823/nu0.mnc -odt float
Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp
reading from orig.mgz…
TR=0.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (-1, 0, 0)
j_ras = (0, 0, -1)
k_ras = (0, 1, 0)
changing data type from uchar to float (noscale = 0)…
writing to ./tmp.mri_nu_correct.mni.15823/nu0.mnc…
Iteration 1 Sun Nov 22 15:00:51 EST 2020
nu_correct -clobber ./tmp.mri_nu_correct.mni.15823/nu0.mnc ./tmp.mri_nu_correct.mni.15823/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.15823/0/ -iterations 1000 -distance 50
[veliebm@c25a-s1.ufhpc:/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/mri/] [2020-11-22 15:00:51] running:
/opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.15823/0/ ./tmp.mri_nu_correct.mni.15823/nu0.mnc ./tmp.mri_nu_correct.mni.15823/nu1.impAssertion failed at line 742 in file templates/CachedArray.cc
nu_estimate_np_and_em: crashed while running volume_stats (termination status=256)
nu_correct: crashed while running nu_estimate_np_and_em (termination status=256)
ERROR: nu_correct
Linux c25a-s1.ufhpc 3.10.0-1062.18.1.el7.x86_64 #1 SMP Wed Feb 12 14:08:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linuxrecon-all -s sub-106 exited with ERRORS at Sun Nov 22 15:00:52 EST 2020
For more details, see the log file /blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106/freesurfer/sub-106/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReportingStandard error:
Return code: 1
- And here’s the Slurm script I ran to launch fMRIPrep for sub-106:
#! /bin/bash
/blue/akeil/veliebm/files/scripts/submitjob_fmriprep.py generated this script on 11.16.2020 at 12.32.32.
Then, /blue/akeil/veliebm/files/scripts/submitjob_fmriprep.py ran this script, which kindly asks fMRIPrep to preprocess sub-106 inside a
Singularity container using the great might of HiPerGator.
#SBATCH --job-name=sub-106 # Job name
#SBATCH --ntasks=1 # Run a single task
#SBATCH --cpus-per-task=2 # Number of CPU cores per task
#SBATCH --mem=8gb # Job memory request. You should never need to change this from 8gb for fMRIPrep.
#SBATCH --time=4-00:00:00 # Walltime in hh:mm:ss or d-hh:mm:ss
#SBATCH --qos=akeil-b # QOS level to use. Can be investment (akeil) or burst (akeil-b).Outputs ----------------------------------
#SBATCH --mail-type=ALL # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=veliebm@ufl.edu # Where to send mail
#SBATCH --output=sub-106_fmriprep_date-11.16.2020_time-12.32.32.log # Standard output log
#SBATCH --error=sub-106_fmriprep_date-11.16.2020_time-12.32.32.err # Standard error log
pwd; hostname; date # Useful things we’ll want to see in the log------------------------------------------
DERIVS_DIR="/blue/akeil/veliebm/files/bopscanner/bids_attempt-1/derivatives/preprocessing/sub-106"
LOCAL_FREESURFER_DIR="$DERIVS_DIR/freesurfer"Make sure FS_LICENSE is defined in the container.
export SINGULARITYENV_FS_LICENSE="/blue/akeil/.licenses/freesurfer.txt"
Prepare derivatives folder.
mkdir -p “$DERIVS_DIR”
mkdir -p “$LOCAL_FREESURFER_DIR”Compose command to start singularity.
SINGULARITY_CMD=“singularity run --cleanenv /blue/akeil/veliebm/files/images/fmriprep_version-20.2.0.sig”
Remove IsRunning files from FreeSurfer.
find “$LOCAL_FREESURFER_DIR/sub-$subject”/ -name “IsRunning” -type f -delete
Compose the command line.
cmd="$SINGULARITY_CMD /blue/akeil/veliebm/files/bopscanner/bids_attempt-1 $DERIVS_DIR participant --participant-label 106 -vv --resource-monitor --write-graph --nprocs 2 --mem_mb 8000"
Setup done, run the command.
echo Running task “$SLURM_ARRAY_TASK_ID”
echo Commandline: “$cmd”
eval "cmd" exitcode=?echo Finished processing subject “$subject” with exit code $exitcode
exit $exitcode
- And last but not least, this is the structure of my dataset:
.
|-- derivatives
|-- sourcedata
|-- sub-101
| |-- anat
| |-- eeg
| |-- func
|-- sub-102
| |-- anat
| |-- eeg
| |-- func
|-- sub-106
| |-- anat
| |-- eeg
| |-- func
|-- sub-107
| |-- anat
| |-- eeg
| |-- func
|-- sub-108
| |-- anat
| |-- eeg
| |-- func
|-- sub-109
| |-- anat
| |-- eeg
| |-- func
|-- sub-111
| |-- anat
| |-- eeg
| |-- func
|-- sub-112
| |-- anat
| |-- eeg
| |-- func
|-- sub-113
| |-- anat
| |-- eeg
| |-- func
|-- sub-114
| |-- anat
| |-- eeg
| |-- func
|-- sub-115
| |-- anat
| |-- eeg
| |-- func
|-- sub-116
| |-- anat
| |-- eeg
| |-- func
|-- sub-117
| |-- anat
| |-- eeg
| |-- func
|-- sub-118
| |-- anat
| |-- eeg
| |-- func
|-- sub-119
| |-- anat
| |-- eeg
| |-- func
|-- sub-120
| |-- anat
| |-- eeg
| |-- func
|-- sub-121
| |-- anat
| |-- eeg
| |-- func
|-- sub-122
| |-- anat
| |-- eeg
| |-- func
|-- sub-123
| |-- anat
| |-- eeg
| |-- func
|-- sub-124
| |-- anat
| |-- eeg
| |-- func
I’m unsure what other information I should include. I appreciate you for helping me 