FileExistsError: [Errno 17] File exists: ‘/out/freesurfer’

Adam_Raikes · May 23, 2019, 10:53pm

I attempted using the method you described for reusing Freesurfer outputs. I previously ran fMRIPrep 1.3.2 and wanted to see the new outputs from 1.4.0. So moved my 1.3.2 Freesurfer outputs as described. All appears well and good, because when running 1.4.0, it detects the Freesurfer folder.

However, fMRIPrep crashes before getting to the BOLD images and I’m hoping maybe you have a thought. The crash log gives me:

Node: fmriprep_wf.fsdir_run_20190523_213307_3f8ef33e_1e56_4014_bafd_9f5d28c81cf9
Working directory: /scratch/fmriprep_wf/fsdir_run_20190523_213307_3f8ef33e_1e56_4014_bafd_9f5d28c81cf9

Node inputs:

derivatives = /out
freesurfer_home = /opt/freesurfer
overwrite_fsaverage = False
spaces = [‘fsaverage5’, ‘fsnative’]
subjects_dir = freesurfer

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 472, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 563, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 643, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 375, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/bids.py”, line 633, in _run_interface
os.makedirs(subjects_dir, exist_ok=True)
File “/usr/local/miniconda/lib/python3.7/os.py”, line 221, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: ‘/out/freesurfer’

I’m running this through Docker on our local cluster using the following (for general compatibility with the code you provided above):

#!/bin/bash
#
#SBATCH -J fmriprep
#SBATCH --array=1-35%5
#SBATCH --time=48:00:00
#SBATCH --partition verylong.q
#SBATCH -w hyperion
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
# Outputs --------
#SBATCH -o /data/BL2_clean/slurm_logs/fmriprep-1.4.0/%x-%A-%a.out
#SBATCH -e /data/BL2_clean/slurm_logs/fmriprep-1.4.0/%x-%A-%a.err
# -----------------------------------------


BASE_DIR="/data/BL2_clean"
BIDS_DIR="$BASE_DIR/nifti"
OUTPUT_DIR="$BASE_DIR/derivatives/fmriprep-1.4.0"
DOCKER_CMD="docker run -i --rm -v $BIDS_DIR:/data:ro -v $OUTPUT_DIR:/out -v $BASE_DIR/code/license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:1.4.0"
unset PYTHONPATH
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*" | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
cmd="${DOCKER_CMD} /data /out participant --participant-label $subject -w /scratch --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --task-id "rest" --output-space T1w template fsnative fsaverage5 --use-syn-sdc --notrack --cifti-output"

echo Running task ${SLURM_ARRAY_TASK_ID}
echo Commandline: $cmd
eval $cmd
exitcode=$?

if [ "$exitcode" -ne "0" ]; then
	echo "$subject" >> ${BASE_DIR}/slurm_logs/fmriprep-1.4.0/failed_subjects.${SLURM_ARRAY_JOB_ID}
	echo "${SLURM_ARRAY_TASK_ID}" >> ${BASE_DIR}/slurm_logs/fmriprep-1.4.0/failed_taskids.${SLURM_ARRAY_JOB_ID}
fi
a
echo Finished task ${SLURM_ARRAY_TASK_ID} with exit code $exitcode

oesteban · May 23, 2019, 11:15pm

I presume that the OS within the container cannot access the freesurfer folder because it is one level up from the folder you are mounting as /out.

I’d suggest the following modification:

#!/bin/bash
#
#SBATCH -J fmriprep
#SBATCH --array=1-35%5
#SBATCH --time=48:00:00
#SBATCH --partition verylong.q
#SBATCH -w hyperion
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=4G
# Outputs --------
#SBATCH -o /data/BL2_clean/slurm_logs/fmriprep-1.4.0/%x-%A-%a.out
#SBATCH -e /data/BL2_clean/slurm_logs/fmriprep-1.4.0/%x-%A-%a.err
# -----------------------------------------


BASE_DIR="/data/BL2_clean"
BIDS_DIR="$BASE_DIR/nifti"
OUTPUT_DIR="$BASE_DIR/derivatives"
DOCKER_CMD="docker run -i --rm -v $BIDS_DIR:/data:ro -v $OUTPUT_DIR:/out -v $BASE_DIR/code/license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:1.4.0"
unset PYTHONPATH
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*" | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
cmd="${DOCKER_CMD} /data /out/fmriprep-1.4.0 participant --participant-label $subject -w /scratch --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --task-id "rest" --output-space T1w template fsnative fsaverage5 --use-syn-sdc --notrack --cifti-output"

echo Running task ${SLURM_ARRAY_TASK_ID}
echo Commandline: $cmd
eval $cmd
exitcode=$?

if [ "$exitcode" -ne "0" ]; then
	echo "$subject" >> ${BASE_DIR}/slurm_logs/fmriprep-1.4.0/failed_subjects.${SLURM_ARRAY_JOB_ID}
	echo "${SLURM_ARRAY_TASK_ID}" >> ${BASE_DIR}/slurm_logs/fmriprep-1.4.0/failed_taskids.${SLURM_ARRAY_JOB_ID}
fi
a
echo Finished task ${SLURM_ARRAY_TASK_ID} with exit code $exitcode

Adam_Raikes · May 24, 2019, 4:40pm

Hi @oesteban

Came in this morning with high hopes but no such luck. Same error message. Only change is instead of

FileExistsError: [Errno 17] File exists: ‘out/freesurfer’

now it’s

FileExistsError: [Errno 17] File exists: ‘out/fmriprep-1.4.0/freesurfer’

effigies · May 24, 2019, 6:34pm

Can you check whether $OUTPUT_DIR/freesurfer exists, and whether it’s a directory? I have a suspicion that it may be a symlink. If so, then it’s probably the case that it points to a valid location outside the container, but that location does not exist inside the container.

Adam_Raikes · May 24, 2019, 9:29pm

It is indeed a symlinked folder (created as in FMRIPREP confounds error (copied columns for multiple subjects)).

I modified my Docker call as below and it appears to be working. Thanks for helping me figure out where I was going wrong.

New code:

BASE_DIR="/data/BL2_clean"
BIDS_DIR="$BASE_DIR/nifti"
OUTPUT_DIR="$BASE_DIR/derivatives"
DOCKER_CMD="docker run -i --rm -v $BASE_DIR:$BASE_DIR -v /data/BL2/BIDS_derivatives/code/license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:1.4.0"
unset PYTHONPATH
subject=$( find ${BIDS_DIR} -maxdepth 1 -type d -name "sub-*" | sort | sed "${SLURM_ARRAY_TASK_ID}q;d" | sed "s/sub-//g" | sed 's!.*/!!' )
cmd="${DOCKER_CMD} $BIDS_DIR $OUTPUT_DIR/fmriprep-1.4.0 participant --participant-label $subject -w /scratch --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --task-id "rest" --output-space T1w template fsnative fsaverage5 --use-syn-sdc --notrack --cifti-output"

oesteban · May 24, 2019, 9:50pm

Oh, right. The previous readonly mount of the root BIDS folder was creating the problem.

Please note that now your data are not protected by the read-only mount. I would consider adding an extra -v flag to isolate the BIDS data root with read only permissions.

Adam_Raikes · May 24, 2019, 10:01pm

Just clarifying, do something like:

DOCKER_CMD="docker run -i --rm  -v $BIDS_DIR:/nifti:ro -v $OUTPUT_DIR:$OUTPUT_DIR -v /data/BL2/BIDS_derivatives/code/license.txt:/opt/freesurfer/license.txt poldracklab/fmriprep:1.4.0"

cmd="${DOCKER_CMD} /nifti $OUTPUT_DIR/fmriprep-1.4.0 participant --participant-label $subject -w /scratch --skip_bids_validation --omp-nthreads 8 --nthreads 12 --mem_mb 30000 --task-id "rest" --output-space T1w template fsnative fsaverage5 --use-syn-sdc --notrack --cifti-output"