No space left for bold_transform [Errno28]

[hcp4715]

Dear all,

I encountered Errno 28 No space left on device when using fmriprep.

I used the following script:

sudo docker run --rm -it \
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti: /data:ro\
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti/derivatives:/out \
-v /media/hcp4715/OS/license.txt \
poldracklab/fmriprep:latest \
/data /out/out \
participant --participant-label 001 \
--fs-license-file 
--ignore slicetiming

Here is my docker information:

Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 3
Server Version: 18.09.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-45-generic
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.53GiB
Name: hcp4715-…
ID: 6KSY…
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Below are part of the crash log:

Node: fmriprep_wf.single_subject_001_wf.func_preproc_ses_d3_task_exp_wf.bold_bold_trans_wf.bold_transform
Working directory: /tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_bold_trans_wf/bold_transform

Node inputs:

args =
copy_dtype = True
default_value = 0.0
dimension =
environ = {}
float = True
input_image = [’/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0000.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0001.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0002.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0003.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0004.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0005.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0006.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0007.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0008.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0009.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0010.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0011.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0012.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0013.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0014.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0015.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf
…
‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol1029.nii.gz’, ‘/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol1030.nii.gz’]
input_image_type =
interpolation = LanczosWindowedSinc
interpolation_parameters =
invert_transform_flags =
num_threads = 7
out_postfix = _trans
output_image =
print_out_composite_warp_file =
reference_image = /tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_split/vol0000.nii.gz
save_cmd = True
transforms = [’/tmp/work/fmriprep_wf/single_subject_001_wf/func_preproc_ses_d3_task_exp_wf/bold_hmc_wf/fsl2itk/mat2itk.txt’]

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 637, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 369, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/itk.py”, line 143, in _run_interface
for i, (in_file, in_xfm) in enumerate(zip(in_files, xfms_list))]
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 586, in result_iterator
yield fs.pop().result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 432, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/thread.py”, line 57, in run
result = self.fn(*self.args, **self.kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/itk.py”, line 262, in _applytfms
runtime = xfm.run().runtime
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 369, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/fixes.py”, line 28, in _run_interface
self.class.name, version))
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utils.py”, line 182, in _copyxform
newimg.to_filename(out_image)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/filebasedimages.py”, line 334, in to_filename
self.to_file_map()
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/analyze.py”, line 1090, in to_file_map
arr_writer.to_fileobj(imgf)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arraywriters.py”, line 562, in to_fileobj
nan2zero=self._needs_nan2zero())
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/volumeutils.py”, line 660, in array_to_file
pre_clips=pre_clips)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/volumeutils.py”, line 833, in _write_data
fileobj.write(dslice.tostring())
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/openers.py”, line 220, in write
return self.fobj.write(*args, **kwargs)
File “/usr/local/miniconda/lib/python3.7/gzip.py”, line 264, in write
self.fileobj.write(self.compress.compress(data))
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/gzip.py”, line 309, in close
fileobj.write(self.compress.flush())
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 69, in run_node
result[‘result’] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 473, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 557, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 641, in _run_command
_save_resultfile(result, outdir, self.name)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py”, line 303, in save_resultfile
savepkl(resultsfile, result)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py”, line 732, in savepkl
pkl_file.close()
File “/usr/local/miniconda/lib/python3.7/gzip.py”, line 319, in close
myfileobj.close()
OSError: [Errno 28] No space left on device

[jdkent]

This looks like your hard drive (or whatever storage device fmirprep is writing intermediate files to) is full.
you can type the following in a terminal to make sure this is the case: df -h (the -h makes the size information in gigabytes instead of kilobytes). The place where /tmp is mounted (probably your primary drive) should show 100% under the Use% column if the message OSError: [Errno 28] No space left on device is suggesting the storage device is full.

If that is the case, the solutions I can see are to either point the working directory to a place with more storage space with the -w option in fmriprep or get more space on the drive /tmp is located (either by deleting files or buying/getting more storage).

Hope this helps!
James

[hcp4715]

Hi, James,

Thanks a lot for your help!
You are right, the /tmp need much more space than I expected (I had only 12 GB at that time)

I changed my code as:

sudo docker run --rm -it \
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti: /data:ro\
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti/derivatives:/out \
-v /media/hcp4715/OS/license.txt \
-v /media/hcp4715/tmp:/tmp \
poldracklab/fmriprep:latest \
/data /out/out \
participant --participant-label 001 \
--fs-license-file 
--ignore slicetiming

Now I have ~50GB for one subject with 2.5 GB raw data. Hope it will work.
PS: I am still not very sure that I useed the -w --work_dir properly (as you can see from my updated code, there is no -w or --work_dir, but seems that the temporary files were written in /tmp as I assigned. Would appreciate if you can provide a more correct way to write the code.

[jdkent]

Glad you found a solution! Your solution works just as well (I just didn’t think of it ). A -w solution would look something like:

sudo docker run --rm -it \
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti: /data:ro\
-v /media/hcp4715/USB1/Data/BIDS_test/Nifti/derivatives:/out \
-v /media/hcp4715/OS/license.txt \ # <- isn't this file supposed to be mapped somewhere in the container?
-v /media/hcp4715/tmp:/work \
poldracklab/fmriprep:latest \
/data /out/out \
participant --participant-label 001 \
-w /work \
--fs-license-file \  # <- isn't there supposed to be a file here?
--ignore slicetiming

as to which solution is “more correct” is a matter of preference. With that said, here is my preference (i.e. opinion, not a rule)

I like to be explicit where any files are being written so no one has to know that fmriprep by default writes to /tmp. By mounting the /tmp directory to another location you solve the problem, but someone else (or you several years from now) reading the code has to infer why you are mounting /tmp to a separate location. Leveraging the -w option (or even more explicitly the --work-dir option) makes it a bit easier (in my mind) to see where all the files are going.

Best,
James

[hcp4715]

Hi, James,

Thanks a lot for your help.

My current code just an trial, didn’t expect that it would work ;-).
It’s really nice that you provided a more explicit way, actually I prefer your way too. I didn’t figure it out because I am still struggling to understand the commands of docker .

Tue 26 update: I tried to write the code as your suggested, but always have an error (sorry for my stupid question): docker: invalid reference format.

I tried to write the following code and other variations:

sudo docker run --rm -it
-v /media/hcp4715/USB1/Data/RepDopa/BIDS_test/Nifti:/data:ro
-v /media/hcp4715/USB1/Data/RepDopa/BIDS_test/Nifti/derivatives:/out
-v /media/hcp4715/OS/host/freesurf_license/license.txt:/opt/freesurfer \ # <- I tried to use a location/folder
-v /media/hcp4715/study/fmriprep_tmp:/work
poldracklab/fmriprep:latest
/data /out/out
participant --participant-label 001
-w /work
–fs-license-file /opt/freesurfer/license.txt
–ignore slicetiming

but when I run the above code, there’s an error about docker:

Maybe I missed something in docker command?
Thanks in advance.

[jdkent]

oops, I think my comment above may have steered you astray, you can bind directories to a docker container, but not files: (you can bind files)

sudo docker run --rm -it 
-v /media/hcp4715/USB1/Data/RepDopa/BIDS_test/Nifti:/data:ro 
-v /media/hcp4715/USB1/Data/RepDopa/BIDS_test/Nifti/derivatives:/out 
-v /media/hcp4715/OS/host/freesurf_license/:/license \ # <- I tried to use a location/folder
-v /media/hcp4715/study/fmriprep_tmp:/work 
poldracklab/fmriprep:latest 
/data /out/out 
participant --participant-label 001 
-w /work 
–fs-license-file /license/license.txt 
–ignore slicetiming

You may not want to bind the directory where freesurfer is installed either since when you bind your local directory to an existing directory in the container, the contents of the container directory are “overwritten” by the contents in your local directory, so I made up a root directory called license (but you can call this whatever you want as long as it doesn’t overwrite any directories in the container)

I think that will fix your error, but I could not replicate it, so I am unsure if that is the root of the issue, but it is an issue.

-James

[effigies]

Just to chime in, you can definitely mount individual files using -v in Docker. However, rather than fiddling with that, I would recommend using the fmriprep-docker wrapper, which will construct the needed mount options for you. See the documentation.

[jdkent]

my bad, I must typed something incorrectly last night when trying to do that, I tried again and I was able to mount individual files.

I echo @effigies comment that using fmriprep-docker is a good way to go since he has already put in a lot of work to make that an easier interface to work with than docker.

[hcp4715]

Hi, James,
Thanks for your help again!
I tested your code again, and found that the error as I showed by the screen shot might resulted from an additional space after the \, i.e., it should be -v /media/hcp4715/OS/host/freesurf_license/:/license \ , instead of -v /media/hcp4715/OS/host/freesurf_license/:/license \[one additional space here].

You’re right, I didn’t use the directory from the freesurfer’s folder.

[lukasvo76]

Hi James,

Thanks a ton for this interesting info!

I am experiencing a similar problem when running fMRIprep locally using the fMRIprep-docker wrapper using Docker desktop and the command prompt on my Windows 10 machine (using Linux containers), especially when trying to run even though it has worked fine in the past.

I have plenty of space left on my hard drive, so surprising to get this error?!

Is there a way to figure out where tmp is physically located on my mounted C-drive so I can check how full that is/gets?

Or can I simply choose define any path I like using the -w or --work-dir options (what is the difference)?

Would adding --clean-workdir help?

Thanks a lot in advance!

Best wishes,

Lukas

[jdkent]

Hi Lukas,

Windows allocates a certain amount of space to their running docker containers so you can try increasing “disk image size” in the advanced setting of docker.

Or as you pointed out, you can use -w or --work-dir option to have all temporary files be written somewhere on your actual C drive. (there is no difference between the -w and --work-dir options, -w is just the abbreviated form of --work-dir).

Hope that helps!
James

[lukasvo76]

Thanks a lot James, very helpful!

I will check out disk image size after my current process finishes!

I tried using the --work-dir option (followed by full Windows notation path to a directory I created first on my C-drive, but that did not work (“no such file or directory” error), any idea why this may be? fMRIprep seemed to call that directory “scratch” internally?

Is there a way to figure out where my current /tmp is mounted?

And would specifying the --clean-workdir option help when attempting to run multiple subjects in one command? The usage notes say “Use of this flag is notrecommended when running concurrent processes of fMRIPrep.”, but not sure what concurrent processes of fMRIPrep actually means?

Sorry for the naive questions, I am new to Docker and fMRIprep, and thanks a ton in advance once again!

Best wishes,

Lukas

[jdkent]

Hi Lukas,

Docker is a complex (but very learnable) tool so don’t apologize for “naive” questions

I tried using the --work-dir option (followed by full Windows notation path to a directory I created first on my C-drive, but that did not work (“no such file or directory” error), any idea why this may be?

If you could share the command you are using, I believe that would shed light on the situation.

Is there a way to figure out where my current /tmp is mounted?

My understanding of how docker works on windows means that /tmp exists on a
virtual machine with a set amount of space that runs on top of windows.
I don’t not know of an easy way of the top of my head to access that directory, but increasing the disk image space for docker should help.

And would specifying the --clean-workdir option help when attempting to run multiple subjects in one command?

I’m actually not very familiar with this option, I would caution against it until we’ve exhausted other solutions.

As a final suggestion, if you are testing fmriprep, I would use the --participant-label flag to select one participant first to make sure everything works before trying to run fmriprep on multiple participants.

Best,
James

[lukasvo76]

Hi James,

Many thanks for the quick and detailed response!

One subject at a time works perfectly fine so far, so this is obviously an option!

I use the following command line for one subject

fmriprep-docker C:\Users\lukas\proj-SERT-fMRI\rawdata1 C:\Users\lukas\proj-SERT-fMRI\derivatives participant --participant-label 01 --fs-license-file C:\Users\lukas\freesurfer.txt --fd-spike-threshold 1 --dvars-spike-threshold 3

When adding the following argument, I got the error I mentioned earlier (“no such file or directory”)
-w C:\Users\lukas\proj-SERT-fMRI\temp

However, when I can allocate more disk space to where /tmp is mounted by default, there is no need to know where it is physically, nor to change its location from the default, so will try that first after my second subject finished running (no errors so far!), before looking into the --clean-workdir option.

FYI, this is what I tried for multiple subjects at once
fmriprep-docker C:\Users\lukas\proj-SERT-fMRI\rawdata1 C:\Users\lukas\proj-SERT-fMRI\derivatives participant --participant-label 01 02 03 04 05 06 08 09 10 11 12 13 14 15 16 17 18 19 20 --fs-license-file C:\Users\lukas\freesurfer.txt --fd-spike-threshold 1 --dvars-spike-threshold 3

Thanks again!

Lukas

[lukasvo76]

Hi again James,

I just checked and my disk image size was set to 256 GB, with 55.7 GB being used according to the Resources/Advanced tab of Docker Desktop.

I can of course increase this further but it already seems quite a lot and lots of space left?

Not sure what is taking up those 55.7 GB, the fmriprep image (which only takes 13.1 GB when I check using docker image ls --all), my tmp folder, or something else? Seems like a good idea to clean up tmp every now and then anyway I guess?

Best wishes,

Lukas