[Errno 39] crashes with (potentially) freesurfer

hanqiuishere · January 19, 2023, 10:56pm

Summary of what happened:

Hi! When running fmriprep on my data, it crashes at some point during processing anatominal run. From the error message in the log file, it says OSError: [Errno 39] Directory not empty: 'label'
At first I thought it means the /label folders inside of /fmriprep/sourcedata/freesurfer should be empty, so I deleted the whole freesurfer folder and reran it. It still show this error. Wierd thing is at the end of the log file (attached below), it says run successfully. While in the output there is only sub-007_desc-preproc_T1w.nii in the /anat folder, without the /func folder.

I’m running with --ignore fieldmaps, but it also happen when running with fieldmaps.

Here is the output file in HPC. Hope it helps. I can’t really understand where it goes wrong from these output messages. Hope you can help me figure out what should I do. Many thanks!
fmriprep_sub-007.o48433628.txt (2.3 MB)

Command used (and if a helper script was used, a link to the helper script or the command generated):

/opt/conda/bin/fmriprep bids bids/derivatives/fmriprep participant -w /scratch/qiuhan/48433628.dccn-l029.dccn.nl/sub-007 --participant-label 007 --skip-bids-validation --fs-license-file /opt_host/fmriprep/license.txt --mem_mb 20000 --omp-nthreads 2 --nthreads 2 --ignore fieldmaps

Version:

fmripre 22.0.1

Environment (Docker, Singularity, custom installation):

pre-loaded in HPC

Data formatted according to a validatable standard? Please provide the output of the validator:

Warning 1: [Code 25] EVENTS_TSV_MISSING

Relevant log outputs (up to 20 lines):

Node: fmriprep_22_0_wf.fsdir_run_20230119_162714_a9e9c7f1_fd5a_45b3_a00f_3fb3a5fad703
Working directory: /scratch/qiuhan/48433628.dccn-l029.dccn.nl/sub-007/fmriprep_22_0_wf/fsdir_run_20230119_162714_a9e9c7f1_fd5a_45b3_a00f_3fb3a5fad703

Node inputs:

derivatives = /project/3018068.01/bids/derivatives/fmriprep
freesurfer_home = /opt/freesurfer
minimum_fs_version = 7.0.0
overwrite_fsaverage = False
spaces = []
subjects_dir = /project/3018068.01/bids/derivatives/fmriprep/sourcedata/freesurfer

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsdir_run_20230119_162714_a9e9c7f1_fd5a_45b3_a00f_3fb3a5fad703.

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/bids.py", line 923, in _run_interface
    shutil.rmtree(dest)
  File "/opt/conda/lib/python3.9/shutil.py", line 732, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/conda/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/opt/conda/lib/python3.9/shutil.py", line 669, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'label'

Screenshots / relevant information:

Steven · January 19, 2023, 11:35pm

Hi @hanqiuishere,

Can you try doing the following?

update to 22.1.1 (most recent at this time)
try specifying a fresh work directory with the -w argument.
make sure no pre-existing Freesurfer outputs are being used
if you can, try using a container (Docker or Singularity) to run the workflow. Using your own anaconda environment introduces the possibilities of different software versions being used. Of note, recent versions of fmriprep expect FreeSurfer 7.2 so they may not work with earlier versions.

Thanks,
Steven

hanqiuishere · January 20, 2023, 9:24am

Hi Steven,
Thank you very much for the suggestions!

It is already ran in a container. Most importantly, these data have been run successfully before with fmriprep 22.1.0 and freesufer 7.3.2, so these might not be the problem. I also noticed some new information, could you have a look at if it help to locate the error?

The error message above is from sub-007, which was newly added and never been processed with fmriprep before. However, for the other subs, which have been run before (but I deleted the whole bids folder before re-running them), the error is different:


230120-01:12:46,602 nipype.workflow INFO:
	 [Node] Finished "_parcstats1", elapsed time 310.528993s.
230120-01:12:50,130 nipype.workflow ERROR:
	 could not run node: fmriprep_22_0_wf.single_subject_005_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.parcstats
230120-01:12:50,183 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _parcstats0.

here is the full log
fmriprep_sub-005.o48433627.txt (2.4 MB)

Besides, after running, in the bids/derivatives/fmriprep/sourcedata/freesurfer folder, sub-007 (the new one) is missing.
I guess the new sub broke at an earlier stage than the ones that have been run before. But still even for the new one (which for sure has no previous freesurfer output) it didn’t work.

Could you perhaps tell more from this? I really appreaciate any help. Thanks!

Steven · January 20, 2023, 1:47pm

Hi,

The text log you attached is using fmriprep 22.0.1. I am confused as you said you were using 22.1.0. If different versions are mixed using the same work directory that can lead to problems.

Please (1) update to the newest version (22.1.1 at this time). (2) provide the full command you use to run fmriprep (including the docker run or singularity run arguments), making sure to specify a work directory that has not been used. (3) make sure there are no pre-existing Freesurfer outputs that fMRIprep can be using.

Also, can you confirm that the only bids validator output was the events.tsv warning?

Best,
Steven

hanqiuishere · January 23, 2023, 9:50pm

Hi Steven,

Thanks a lot!

I got new results: using a newly created folder as working directory, and tried with both 22.1.1 and 22.1.0, the results are the same: Now some subjects are running successfully, while two subs still get the same error as before (the first one in this stream)

cNode: fmriprep_22_1_wf.fsdir_run_20230123_160029_51ab4d24_aaae_43f7_95cc_46a469a65209
Working directory: /project/3018068.01/fmriprep_work2/sub-004/fmriprep_22_1_wf/fsdir_run_20230123_160029_51ab4d24_aaae_43f7_95cc_46a469a65209

Node inputs:

derivatives = /project/3018068.01/bids/derivatives/fmriprep_sub001007
freesurfer_home = /opt/freesurfer
minimum_fs_version = 7.0.0
overwrite_fsaverage = False
spaces = []
subjects_dir = /project/3018068.01/bids/derivatives/fmriprep_sub001007/sourcedata/freesurfer

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsdir_run_20230123_160029_51ab4d24_aaae_43f7_95cc_46a469a65209.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/bids.py", line 936, in _run_interface
	    shutil.rmtree(dest)
	  File "/opt/conda/lib/python3.9/shutil.py", line 732, in rmtree
	    _rmtree_safe_fd(fd, path, onerror)
	  File "/opt/conda/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd
	    onerror(os.rmdir, fullname, sys.exc_info())
	  File "/opt/conda/lib/python3.9/shutil.py", line 669, in _rmtree_safe_fd
	    os.rmdir(entry.name, dir_fd=topfd)
	OSError: [Errno 39] Directory not empty: 'label'

for (2):

unset PYTHONPATH; export PYTHONNOUSERSITE=1; singularity run --cleanenv --env TMP=\$TMPDIR,TMPDIR=\$TMPDIR,TEMP=\$TMPDIR,TEMPDIR=\$TMPDIR /opt/fmriprep/22.1.1/fmriprep-22.1.1.simg bids bids/derivatives/fmriprep_sub001007 participant -w fmriprep_work2/sub-004 --participant-label 004  --fs-license-file /opt_host/fmriprep/license.txt --mem_mb 20000 --omp-nthreads 2 --nthreads 2  --skip-bids-validation --ignore fieldmaps

for (3):
I deleted the whole /sourcedata/freesurfer folder before rerunning it, and also changed the output directory of fmriprep. Can this make sure no pre-existing Freesurfer outputs that fMRIprep can be using?

It’s really good that with a working directory some subs run well, but this also makes the problem more confusing, since the subs were handled in the same way. Could you think of other possible reasons? Really appreaciate any help! Thank you!

Steven · January 23, 2023, 10:01pm

Hi,

Glad that some subjects are working!

For the subjects that do not work, was it the same subjects that were failing before? If you repeat the process with these two subjects (e.g., delete their freesurfer data, restart job with new working directory), does it still fail? What is in the contents of these subjects’ label folders?

Deleting the data in sourcedata/freesurfer should do the trick, as that is the default place fMRIPrep will look for freesurfer outputs.

Your command looks fine, but I note two things. First, I would add a bind string to the singularity part of the command, e.g.,singularity run --cleanenv -B /FULL/PATH/TO/bids --env ...... The second would be that I would not put have a separate output directory for each subject, which you seem to be doing here based on your output directory being called bids/derivatives/fmriprep_sub001007. I also imagine this could just be something you are doing while testing, but just make sure that when you are ready to run for real, that your output directory is just bids/derivatives.

Best,
Steven

hanqiuishere · February 1, 2023, 8:43am

Hi ,

Sorry for the late reply. I waited for some new data to confirm the issue was solved.

Here is what happened: I increased the memory allocated to each fmriprep job from 20 to 40 GB, suddenly they can all run well. I don’t really understand how, since 20 GB used to be enough for my data and still enough for others’ data who are using the same cluster. But it did solve all the issues for all the subjects.

Besides, in the job log file, it seems fmriprep will use up the allocated memory no matter how much:

Asked resources:   nodes=1:ppn=6,walltime=48:00:00,mem=40000mb,file=50gb,neednodes=1:ppn=6
Used resources:	   cput=26:40:57,walltime=07:59:03,mem=41943056384b

Asked resources:   nodes=1:ppn=2,walltime=48:00:00,mem=20000mb,neednodes=1:ppn=2
Used resources:	   cput=26:15:44,walltime=17:00:20,mem=20971536384b

which makes it seem fmriprep is flexible in terms of memory usage, so less likely to crush due to memory shortage.
I put this information here in case someone else encounter the same problem, although I’m not so sure that the memory caused the error, maybe it’s worth trying.

Thanks a lot!