Hello fellow Neurostars,
I have been attempting to run fMRIprep on 27 subjects for almost a month now, however it has only successfully finished running for one subject. Every time I run it on another subject it takes a very long time but still ends up failing. All the subject folders have also been BIDS validated and the FreeSurfer license key I am using is updated and valid. Additionally, I have allocated 34GB of memory to Docker and 10 CPUs. The error log on the .html file usually shows both or either of the following errors:
Node Name: _autorecon_surfs0
Node Name: _autorecon_surfs1
The command I am running is:
fmriprep-docker subject-directory subject-output participant --participant-label 26 --fs-license-file /Applications/freesurfer/7.1.1/license.txt --ignore slicetiming
The error in the terminal for the most recent subject I ran gave:
recon-all -s sub-25 exited with ERRORS at Mon Jul 5 20:48:19 UTC 2021
For more details, see the log file /out/freesurfer/sub-25/scripts/recon-all-rh.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
Standard error:
Return code: 1
210705-20:48:23,310 nipype.workflow ERROR:
could not run node: fmriprep_wf.single_subject_25_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon_surfs
210705-20:48:23,318 nipype.workflow ERROR:
could not run node: fmriprep_wf.single_subject_25_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon_surfs
210705-20:48:23,349 nipype.workflow CRITICAL:
fMRIPrep failed: Workflow did not execute cleanly. Check log for details
210705-20:48:24,386 cli ERROR:
Preprocessing did not finish successfully. Errors occurred while processing data from participants: 25 (2). Check the HTML reports for details.
fMRIPrep: Please report errors to Issues · nipreps/fmriprep · GitHub
Your help is very much appreciated.
Can you open the log file and report the error listed there? Go to your output directory, and in the subject folder, you should be able to find a crash report in a folder called logs.
Hi Steven,
Thank you for your response.
There are two crash files that the outputs gave. The final error at the end of the crash files are:
File 1:
recon-all -s sub-25 exited with ERRORS at Mon Jul 5 20:47:27 UTC 2021
For more details, see the log file /out/freesurfer/sub-25/scripts/recon-all-lh.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
Standard error:
Return code: 1
File 2:
recon-all -s sub-25 exited with ERRORS at Mon Jul 5 20:48:19 UTC 2021
For more details, see the log file /out/freesurfer/sub-25/scripts/recon-all-rh.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
Standard error:
Return code: 1
Neurostars will not let me attach the full files since I am a new user. Is there another way I can share the full files with you?
Thanks.
I’ll send you a dm with my email.
I have a similar issue.
Was there a resolution to this?
Hi I’m new to fMRIprep, but running into a similar issue. FWIW Single subject runs finish fine with free surfer switched off via the “ fs-no-reconall” flag. but including freesurfer in the pipeline is preferred. Using the v20.2.6 via docker on a redhat 7 machine
When FreeSurfer fails, it directs you to logs where you can view a detailed report of the error. In most cases it will either be a memory issue or a data quality issue.
Thank you for your reply.
Recon-all.log reports no errors.
Recon-all-lh.log reports errors.
Recon-all-rh.log reports errors.
The forum is preventing me from attaching error log.
The errors seem to refer to:
== Number of threads available to mris_sphere for OpenMP = 8 ==
scaling brain by 0.326…
inflating to sphere (rms error < 2.00)
000: dt: 0.0000, rms radial error=174.727, avgs=0
spherical inflation complete.
epoch 1 (K=10.0), pass 1, starting sse = 19433.96
taking momentum steps…
taking momentum steps…
taking momentum steps…
pass 1 complete, delta sse/iter = 0.00/10 = 0.00016
epoch 2 (K=40.0), pass 1, starting sse = 3255.13
taking momentum steps…
taking momentum steps…
taking momentum steps…
pass 1 complete, delta sse/iter = 0.00/10 = 0.00001
epoch 3 (K=160.0), pass 1, starting sse = 329.28
taking momentum steps…
taking momentum steps…
taking momentum steps…
pass 1 complete, delta sse/iter = 0.09/11 = 0.00861
epoch 4 (K=640.0), pass 1, starting sse = 17.39
taking momentum steps…
taking momentum steps…
taking momentum steps…
pass 1 complete, delta sse/iter = 0.28/19 = 0.01478
final distance error %21.97
From Freesurfer forum:
There are lots of internal “error” metrics we use to optimize things that are not errors in the programatic sense - they are functionals we are minimizing/maximizing.
This could be the case here.
Any insights appreciated.
Thanks.
-Tom.
PS look in scripts folder for error logs.
Generally the final thing in the log will indicate an error. Could you paste the last 50 lines or so?
Thanks for looking at this.
From Recon-all-lh.log (last 100 lines):
mris_inflate -rusage /out/freesurfer/sub-003/touch/rusage.mris_inflate.lh.dat -n 50 ../surf/lh.smoothwm ../surf/lh.inflated
niterations = 50
Reading ../surf/lh.smoothwm
avg radius = 44.9 mm, total surface area = 76046 mm^2
writing inflated surface to ../surf/lh.inflated
writing sulcal depths to ../surf/lh.sulc
step 000: RMS=0.162 (target=0.015)
step 005: RMS=0.116 (target=0.015)
step 010: RMS=0.086 (target=0.015)
step 015: RMS=0.071 (target=0.015)
step 020: RMS=0.059 (target=0.015)
step 025: RMS=0.050 (target=0.015)
step 030: RMS=0.042 (target=0.015)
step 035: RMS=0.036 (target=0.015)
step 040: RMS=0.032 (target=0.015)
step 045: RMS=0.028 (target=0.015)
step 050: RMS=0.025 (target=0.015)
step 055: RMS=0.024 (target=0.015)
step 060: RMS=0.022 (target=0.015)
step 065: RMS=0.021 (target=0.015)
step 070: RMS=0.020 (target=0.015)
step 075: RMS=0.019 (target=0.015)
step 080: RMS=0.019 (target=0.015)
step 085: RMS=0.018 (target=0.015)
step 090: RMS=0.018 (target=0.015)
step 095: RMS=0.018 (target=0.015)
step 100: RMS=0.017 (target=0.015)
step 105: RMS=0.017 (target=0.015)
step 110: RMS=0.017 (target=0.015)
step 115: RMS=0.017 (target=0.015)
step 120: RMS=0.017 (target=0.015)
step 125: RMS=0.017 (target=0.015)
step 130: RMS=0.016 (target=0.015)
step 135: RMS=0.016 (target=0.015)
step 140: RMS=0.016 (target=0.015)
step 145: RMS=0.016 (target=0.015)
step 150: RMS=0.016 (target=0.015)
step 155: RMS=0.016 (target=0.015)
step 160: RMS=0.016 (target=0.015)
step 165: RMS=0.016 (target=0.015)
step 170: RMS=0.016 (target=0.015)
step 175: RMS=0.016 (target=0.015)
step 180: RMS=0.016 (target=0.015)
step 185: RMS=0.016 (target=0.015)
step 190: RMS=0.016 (target=0.015)
step 195: RMS=0.015 (target=0.015)
step 200: RMS=0.015 (target=0.015)
step 205: RMS=0.015 (target=0.015)
step 210: RMS=0.015 (target=0.015)
step 215: RMS=0.015 (target=0.015)
step 220: RMS=0.015 (target=0.015)
step 225: RMS=0.015 (target=0.015)
step 230: RMS=0.015 (target=0.015)
step 235: RMS=0.015 (target=0.015)
step 240: RMS=0.015 (target=0.015)
step 245: RMS=0.015 (target=0.015)
step 250: RMS=0.015 (target=0.015)
step 255: RMS=0.015 (target=0.015)
step 260: RMS=0.015 (target=0.015)
step 265: RMS=0.015 (target=0.015)
step 270: RMS=0.015 (target=0.015)
step 275: RMS=0.015 (target=0.015)
step 280: RMS=0.015 (target=0.015)
step 285: RMS=0.015 (target=0.015)
step 290: RMS=0.015 (target=0.015)
step 295: RMS=0.015 (target=0.015)
step 300: RMS=0.015 (target=0.015)
inflation complete.
inflation took 1.0 minutes
mris_inflate utimesec 280.498084
mris_inflate stimesec 0.873651
mris_inflate ru_maxrss 262516
mris_inflate ru_ixrss 0
mris_inflate ru_idrss 0
mris_inflate ru_isrss 0
mris_inflate ru_minflt 64466
mris_inflate ru_majflt 0
mris_inflate ru_nswap 0
mris_inflate ru_inblock 0
mris_inflate ru_oublock 12704
mris_inflate ru_msgsnd 0
mris_inflate ru_msgrcv 0
mris_inflate ru_nsignals 0
mris_inflate ru_nvcsw 105838
mris_inflate ru_nivcsw 945
#--------------------------------------------
#@# Curv .H and .K lh Wed Dec 1 04:04:38 UTC 2021
/out/freesurfer/sub-003/surf
mris_curvature -w lh.white.preaparc
total integrated curvature = 18.848*4pi (236.849) --> -18 handles
ICI = 190.6, FI = 1869.5, variation=29563.506
writing Gaussian curvature to ./lh.white.preaparc.K...done.
writing mean curvature to ./lh.white.preaparc.H...done.
rm -f lh.white.H
ln -s lh.white.preaparc.H lh.white.H
Linux eeff23e566a8 5.4.0-91-generic #102~18.04.1-Ubuntu SMP Thu Nov 11 14:46:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s sub-003 exited with ERRORS at Wed Dec 1 04:04:41 UTC 2021
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
From Recon-all-rh.log:
erasing segment 8 (vno[0] = 124916)
erasing segment 9 (vno[0] = 124918)
erasing segment 10 (vno[0] = 125911)
erasing segment 11 (vno[0] = 126962)
erasing segment 12 (vno[0] = 126965)
erasing segment 13 (vno[0] = 126966)
writing cortex label to /out/freesurfer/sub-003/label/rh.cortex.label...
writing curvature file /out/freesurfer/sub-003/surf/rh.curv
writing smoothed area to rh.area
writing curvature file /out/freesurfer/sub-003/surf/rh.area
vertex spacing 0.78 +- 0.22 (0.03-->4.20) (max @ vno 113046 --> 113073)
face area 0.25 +- 0.12 (0.00-->2.82)
refinement took 3.1 minutes
#--------------------------------------------
#@# Smooth2 rh Wed Dec 1 04:01:14 UTC 2021
/out/freesurfer/sub-003/scripts
mris_smooth -n 3 -nw -seed 1234 ../surf/rh.white.preaparc ../surf/rh.smoothwm
smoothing for 3 iterations
setting seed for random number generator to 1234
smoothing surface tessellation for 3 iterations...
smoothing complete - recomputing first and second fundamental forms...
#--------------------------------------------
#@# Inflation2 rh Wed Dec 1 04:01:18 UTC 2021
/out/freesurfer/sub-003/scripts
mris_inflate -rusage /out/freesurfer/sub-003/touch/rusage.mris_inflate.rh.dat -n 50 ../surf/rh.smoothwm ../surf/rh.inflated
niterations = 50
Reading ../surf/rh.smoothwm
avg radius = 45.4 mm, total surface area = 76662 mm^2
writing inflated surface to ../surf/rh.inflated
writing sulcal depths to ../surf/rh.sulc
step 000: RMS=0.159 (target=0.015)
step 005: RMS=0.113 (target=0.015)
step 010: RMS=0.084 (target=0.015)
step 015: RMS=0.069 (target=0.015)
step 020: RMS=0.057 (target=0.015)
step 025: RMS=0.048 (target=0.015)
step 030: RMS=0.040 (target=0.015)
step 035: RMS=0.034 (target=0.015)
step 040: RMS=0.029 (target=0.015)
step 045: RMS=0.026 (target=0.015)
step 050: RMS=0.023 (target=0.015)
step 055: RMS=0.022 (target=0.015)
step 060: RMS=0.021 (target=0.015)
step 065: RMS=0.020 (target=0.015)
step 070: RMS=0.019 (target=0.015)
step 075: RMS=0.018 (target=0.015)
step 080: RMS=0.017 (target=0.015)
step 085: RMS=0.017 (target=0.015)
step 090: RMS=0.017 (target=0.015)
step 095: RMS=0.017 (target=0.015)
step 100: RMS=0.016 (target=0.015)
step 105: RMS=0.016 (target=0.015)
step 110: RMS=0.016 (target=0.015)
step 115: RMS=0.016 (target=0.015)
step 120: RMS=0.016 (target=0.015)
step 125: RMS=0.016 (target=0.015)
step 130: RMS=0.015 (target=0.015)
step 135: RMS=0.015 (target=0.015)
step 140: RMS=0.015 (target=0.015)
step 145: RMS=0.015 (target=0.015)
step 150: RMS=0.015 (target=0.015)
inflation complete.
inflation took 0.6 minutes
mris_inflate utimesec 176.692003
mris_inflate stimesec 0.798914
mris_inflate ru_maxrss 262672
mris_inflate ru_ixrss 0
mris_inflate ru_idrss 0
mris_inflate ru_isrss 0
mris_inflate ru_minflt 64574
mris_inflate ru_majflt 1
mris_inflate ru_nswap 0
mris_inflate ru_inblock 232
mris_inflate ru_oublock 12752
mris_inflate ru_msgsnd 0
mris_inflate ru_msgrcv 0
mris_inflate ru_nsignals 0
mris_inflate ru_nvcsw 99027
mris_inflate ru_nivcsw 791
#--------------------------------------------
#@# Curv .H and .K rh Wed Dec 1 04:01:54 UTC 2021
/out/freesurfer/sub-003/surf
mris_curvature -w rh.white.preaparc
total integrated curvature = 21.119*4pi (265.393) --> -20 handles
ICI = 183.5, FI = 1836.1, variation=28848.006
writing Gaussian curvature to ./rh.white.preaparc.K...done.
writing mean curvature to ./rh.white.preaparc.H...done.
rm -f rh.white.H
ln -s rh.white.preaparc.H rh.white.H
Linux eeff23e566a8 5.4.0-91-generic #102~18.04.1-Ubuntu SMP Thu Nov 11 14:46:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
recon-all -s sub-003 exited with ERRORS at Wed Dec 1 04:01:57 UTC 2021
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting
Curiously, the only explicit references to error are in the previous post (not at the end).
Thanks.
-Tom.
Interesting. You can try posting the whole logs to something like https://gist.github.com/ or https://pastebin.com/.
I am beginning to suspect this is not a grievous error in freesurfer world but for some reason fmriprep flags it as a critical error.
Thanks.
-Tom.
Scratch that.
Closer inspection of the output files reveals there are several missing.
There is an error.
I just don’t know what it is.
Yeah, with no error reported, it’s going to be hard to diagnose. I would try re-running fMRIPrep with -vv flags and then report:
- Your command
- A full log of the terminal output
Will do.
Thanks for the help.
I will try some other subjects too.
I think I have identified the problem.
I specified the output file path as a remote server.
For some reason this broke the pipeline.
It worked locally.
Thanks again.