Ubuntu Server Usage vs Personal Computer

vpalacios · January 25, 2021, 12:48am

When I attempt to run a single participant through fmriprep on my ubuntu server which has docker fmriprep, I receive an error that I do not receive when I perform the exact same operation locally on my mac. I am new to fmriprep, docker, and only have a little experience with a server and terminal. Here is the final part of the error message (the entire message is extremely long):

 [Node] Finished "fmriprep_wf.single_subject_10159_wf.func_preproc_task_scap_wf.bold_t1_trans_wf.bold_reference_wf.enhance_and_skullstrip_bold_wf.apply_mask".

210125-00:31:27,503 nipype.workflow WARNING:
Storing result file without outputs
210125-00:31:27,504 nipype.workflow WARNING:
[Node] Error on “fmriprep_wf.single_subject_10159_wf.func_preproc_task_rest_wf.bold_std_trans_wf.bold_to_std_transform” (/tmp/work/fmriprep_wf/single_subject_10159_wf/func_preproc_task_rest_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.res2/bold_to_std_transform)
exception calling callback for <Future at 0x7f350f883ef0 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback:
“”"
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 741, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 419, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/itk.py”, line 172, in _run_interface
zip(in_files, xfms_list)
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 586, in result_iterator
yield fs.pop().result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 432, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/thread.py”, line 57, in run
result = self.fn(*self.args, **self.kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/itk.py”, line 234, in _applytfms
runtime = xfm.run().runtime
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 419, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/fixes.py”, line 48, in _run_interface
message="%s (niworkflows v%s)" % (self.class.name, version),
File “/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/interfaces/utils.py”, line 239, in _copyxform
newimg.to_filename(out_image)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/filebasedimages.py”, line 333, in to_filename
self.to_file_map()
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/analyze.py”, line 1007, in to_file_map
data = np.asanyarray(self.dataobj)
File “/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/numeric.py”, line 553, in asanyarray
return array(a, dtype, copy=False, order=order, subok=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arrayproxy.py”, line 391, in array
arr = self._get_scaled(dtype=dtype, slicer=())
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arrayproxy.py”, line 358, in _get_scaled
scaled = apply_read_scaling(self._get_unscaled(slicer=slicer), scl_slope, scl_inter)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/arrayproxy.py”, line 337, in _get_unscaled
mmap=self._mmap)
File “/usr/local/miniconda/lib/python3.7/site-packages/nibabel/volumeutils.py”, line 529, in array_from_file
raise IOError(f"Expected {n_bytes} bytes, got {n_read} bytes from "
OSError: Expected 4328140 bytes, got 1276628 bytes from object

could the file be damaged?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 67, in run_node
result[“result”] = node.run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 516, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 635, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 750, in _run_command
rebase=str2bool(self.config[“execution”][“use_relative_paths”]),
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py”, line 235, in save_resultfile
savepkl(resultsfile, result)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py”, line 726, in savepkl
pkl_file.write(content)
File “/usr/local/miniconda/lib/python3.7/gzip.py”, line 319, in close
myfileobj.close()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py”, line 232, in process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 70, in run_node
result[“result”] = node.result
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 217, in result
op.join(self.output_dir(), "result%s.pklz" % self.name)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py”, line 291, in load_resultfile
raise FileNotFoundError(results_file)
FileNotFoundError: /tmp/work/fmriprep_wf/single_subject_10159_wf/func_preproc_task_rest_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.res2/bold_to_std_transform/result_bold_to_std_transform.pklz
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 324, in _invoke_callbacks
callback(self)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 159, in _async_callback
result = args.result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 425, in result
return self.__get_result()
File “/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py”, line 384, in __get_result
raise self._exception
FileNotFoundError: /tmp/work/fmriprep_wf/single_subject_10159_wf/func_preproc_task_rest_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.res2/bold_to_std_transform/result_bold_to_std_transform.pklz

Steven · January 25, 2021, 4:43pm

I would first try to address the “No space left on device error”. BIDS-apps create a lot of files (both in terms of final outputs and intermediate files in the work directory). Without knowing the command you used to run fmriprep (which you could add in a reply here), I am guessing that the work directory (where intermediate files are stored) may have been created by default in a drive that does not have a lot of room. You should use the -w argument to specify the work drive.

Steven

vpalacios · January 28, 2021, 3:11am

Thank you Steven, this was indeed the issue. As I am new to servers, I did not know that the partition in which Docker lived only had 20 GB of memory.

I followed roughly these steps to move Docker to a partition that has about 500GB and that resolved the issue:

EVERY COMMAND MUST START WITH `sudo` on the server

Stop docker:

$ service docker stop

Verify no docker process is running

$ ps faux

Double check docker really isn’t running.

Take a look at the current docker directory:

$ ls [PATH TO WHERE DOCKER LIVES]/docker/

Move the docker directory to a larger partition:

$ mv [PATH TO WHERE DOCKER LIVES]/docker [PATH TO WHERE DOCKER SHOULD LIVE]/docker

Make a symlink - tell system(?) to look for docker where I want it to look:

$ ln -s [PATH TO WHERE DOCKER NOW LIVES]/docker [PATH TO WHERE DOCKER USED TO LIVE]/docker

Take a peek at the directory structure to make sure it looks like it did before the mv:

$ ls [PATH TO WHERE DOCKER USED TO LIVE]/docker/

(note the trailing slash to resolve the symlink)

Start docker back up

$ service docker start

restart your containers

There is probably a more elegant a solution, but this works for now. Thank you, Steven!