Fmriprep crashing during freesurfer nu_correct

I’m having some issues running fmriprep 1.1.2 on data from a couple studies (both of which pass the BIDS-validator). I am processing this on the local HCP.

First off here is what I’m attempting to run (with all directory variables defined in code above this):
env -i which singularity run -e
-B /dartfs-hpc/rc/lab/H/HuckinsJ:/mnt
-B $SCRATCH_DIR:/scratch
-H /my/home/directory
$IMAGE
–participant_label sid000444
–nthreads $NTHREADS --omp-nthreads $OMP_NTHREADS
–output-space template fsaverage fsaverage6 fsaverage5
–template-resampling-grid /mnt/templates/MNI152_T1_1mm_to_3x3x3.nii.gz
–fs-license-file /home/jhuckins/license.txt
–no-submm-recon
–ignore fieldmaps
–write-graph
–cifti-output
–skull-strip-template OASIS -w /scratch/working/$DATASET
/mnt/data/bids/$DATASET $OUT_DIR participant

This start running and then crashes saying it can’t load nu0.mnc although the file exists and is located exactly where fmriprep/freesurfer thinks it is. The same error happens when running with or without the submm flag and with multiple procs/threads or 1 each. Thanks for any help on this.
Jeremy

Tail of FreeSurfer output log:
mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50

/mnt/data/fmriprep/1030_ThisIsUs17F/freesurfer/sub-sid000444/mri
/opt/freesurfer/bin/mri_nu_correct.mni
–no-rescale --i orig.mgz --o orig_nu.mgz --n 1 --proto-iters 1000 --distance 50
nIters 1
$Id: mri_nu_correct.mni,v 1.27 2016/02/26 16:19:49 mreuter Exp $
Linux m03.hpcc.dartmouth.edu 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Mon Jul 16 18:01:54 UTC 2018
Program nu_correct, built from:
Package MNI N3, version 1.12.0, compiled by nicks@terrier (x86_64-unknown-linux-gnu) on 2015-06-19 at 01:25:34
/usr/bin/bc
tmpdir is ./tmp.mri_nu_correct.mni.28172
/mnt/data/fmriprep/1030_ThisIsUs17F/freesurfer/sub-sid000444/mri
mri_convert orig.mgz ./tmp.mri_nu_correct.mni.28172/nu0.mnc -odt float
mri_convert.bin orig.mgz ./tmp.mri_nu_correct.mni.28172/nu0.mnc -odt float
$Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp $
reading from orig.mgz…
TR=2300.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (-1, -5.47152e-09, 9.31323e-10)
j_ras = (9.31323e-10, 1.49012e-08, -1)
k_ras = (1.33878e-08, 1, -1.49012e-08)
changing data type from uchar to float (noscale = 0)…
writing to ./tmp.mri_nu_correct.mni.28172/nu0.mnc…


Iteration 1 Mon Jul 16 18:01:55 UTC 2018
nu_correct -clobber ./tmp.mri_nu_correct.mni.28172/nu0.mnc ./tmp.mri_nu_correct.mni.28172/nu1.mnc -tmpdir ./tmp.mri_nu_correct.mni.28172/0/ -iterations 1000 -distance 50
Cannot read input file: ./tmp.mri_nu_correct.mni.28172/nu0.mnc
[d28603y@m03.hpcc.dartmouth.edu:/mnt/data/fmriprep/1030_ThisIsUs17F/freesurfer/sub-sid000444/mri/] [2018-07-16 18:01:55] running:
/opt/freesurfer/mni/bin/nu_estimate_np_and_em -parzen -log -sharpen 0.15 0.01 -iterations 1000 -stop 0.001 -shrink 4 -auto_mask -nonotify -b_spline 1.0e-7 -distance 50 -quiet -execute -clobber -nokeeptmp -tmpdir ./tmp.mri_nu_correct.mni.28172/0/ ./tmp.mri_nu_correct.mni.28172/nu0.mnc ./tmp.mri_nu_correct.mni.28172/nu1.imp

nu_correct: crashed while running nu_estimate_np_and_em (termination status=512)
ERROR: nu_correct
Linux m03.hpcc.dartmouth.edu 3.10.0-862.6.3.el7.x86_64 #1 SMP Tue Jun 26 16:32:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-sid000444 exited with ERRORS at Mon Jul 16 18:01:56 UTC 2018

To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Not sure what is going on here - this thread seems relevent: https://mail.nmr.mgh.harvard.edu/pipermail//freesurfer/2017-February/050008.html

Could you try running it with --clearenv and without -H singularity flags?

Thanks for the quick response! Previously I was running it with -e but have also tested it with --cleanenv and also removed the home directory mapping and am getting the same results. Similar to the user in the thread you suggested I should have plenty of space in the output and scratch directories (>0.5TG).

For kicks I deleted the previous fmriprep output for that subject and tried using the image available here to see if there was something wrong in how I had made the image: https://singularity-hub.org/collections/895

Even running with an image made by someone else I’m getting the same error message:
Cannot read input file: ./tmp.mri_nu_correct.mni.4292/nu0.mnc

As a note, the cluster I’m using runs Singularity version 2.4.2-master.g09fb002.
Cheers

Does the problem only exist for that subject?

I have tried a couple subjects and all crash during nu_correct because they can’t read the temp nu0.mnc file even though it is present.