Qsiprep fills up disk space

Software Support
,
1

Summary of what happened:

HI, I am new to QSIprep and Docker. I am running qsiprep with docker container for a single HCP subject. But I get EOS: No space left on device. Which would be the best practice to solve the problem in this case?

Command used

docker run -ti --rm -v $HOME/100307:/data -v $HOME/dti-preproc:/out -v \
licence.txt:/opt/freesurfer/license.txt pennlinc/qsiprep:latest /data /out \
 participant --fs-license-file licence.txt --output-resolution 1.2  \
 --nthreads 64 --omp-nthreads 2 \
--anat-modality T1w --denoise-after-combining \
--distortion-group-merge average --hmc-model eddy

Version:

qsiprep latest version (1.0.0rc2)

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Relevant log outputs (up to 20 lines):

241130-04:45:04,744 nipype.workflow WARNING:
         [Node] Error on "qsiprep_1_0_wf.sub_100307_wf.dwi_preproc_acq_dir97lr_wf.pre_hmc_wf.merge_and_denoise_wf.merged_denoise.denoiser" (/ssd1/work/qsiprep_1_0_wf/sub_100307_wf/dwi_preproc_acq_dir97lr_wf/pre_hmc_wf/merge_and_denoise_wf/merged_denoise/denoiser)
exception calling callback for <Future at 0x7f2d02447430 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 66, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 743, in _run_command
    _save_resultfile(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 234, in save_resultfile
    savepkl(resultsfile, result)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/utils/filemanip.py", line 697, in savepkl
    with pkl_open(tmpfile, "wb") as pkl_file:
  File "/opt/conda/envs/qsiprep/lib/python3.10/gzip.py", line 344, in close
    myfileobj.close()
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result["result"] = node.result
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 221, in result
    return _load_resultfile(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/utils.py", line 290, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /ssd1/work/qsiprep_1_0_wf/sub_100307_wf/dwi_preproc_acq_dir97lr_wf/pre_hmc_wf/merge_and_denoise_wf/merged_denoise/denoiser/result_denoiser.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks
    callback(self)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 158, in _async_callback
    result = args.result()
  File "/opt/conda/envs/qsiprep/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/opt/conda/envs/qsiprep/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
FileNotFoundError: /ssd1/work/qsiprep_1_0_wf/sub_100307_wf/dwi_preproc_acq_dir97lr_wf/pre_hmc_wf/merge_and_denoise_wf/merged_denoise/denoiser/result_denoiser.pklz

Other relevant information

This is the disk that gets filled up to 100% during the process.
image

2

Hi @cmatti and welcome to neurostars!

A few things:

  1. It looks like you are working with an HCP subject. Qsirecon (a separate but related software) allows you to ingress minimally preprocessed hcp data.
  2. It’s possible it was a temp drive that is filling up. I’d recommend explicitly setting your own working directory (-w flag).
  3. What kind of system are you on? If you plan on moving to a high performance compute cluster, I recommend doing it now rather than troubleshooting on a personal device.

Best,
Steven

3

Hi @Steven,

Thank you for your answer.

  1. I didn’t know QSIrecon could have HCP minimally preprocessed data as input, that is super helpful and I will look into it!

However, I am also interested in learning to preprocess the raw data with QSIprep.

  1. I am on a cluster already, it’s a Debian-based linux system, but as a user I have permission access only to the aforementioned /dev/sdc1 disk. I am added as a user to the docker group. After creating the docker image, the /dev/sdc1 disk has about 22 GB left. Since I am not familiar with the amount of space qsiprep will need, I don’t know if this is enough and the problem lies elsewhere.
  2. Is it possible to set the -w flag in a different disk ?

Best,
Chiara

4

Hi @cmatti,

Hm most clusters don’t allow docker (and use singularity instead), interesting.

Especially for dense hcp data, 22gb is likely not enough.

You can set the -w anywhere (just don’t put it in the BIDS directory), and be sure to mount it in the docker container too.

Also, I recommend doing some form of Gibbs deringing. Use rpg since the DWI is uses a partial fourier acquisition.

Best,
Steven

5

Hi @Steven,

Thank you so much, specifying the working directory worked!

I am now trying to run qsirecon on a single subject with the following command but the pipeline gets stuck with no error for hours. It doesn’t always happen at the same point (sometimes its at node Finished “plot_peaks”, sometimes at “tractography”, or at “estimate_fod”).
Do you know why this might happen? I can open another topic if necessary.

$docker run -ti --rm -v $HOME/dti-preproc/:/input \
-v $HOME/qsirecon/:/output -v "/work-qsir":/work -v license.txt:/opt/freesurfer/license.txt pennlinc/qsirecon:latest /input /output participant \
--fs-license-file /opt/freesurfer/license.txt \
 --n-cpus 8 --omp-nthreads 8 --mem-mb 32000\
 --output-resolution 1.2 --recon-spec mrtrix_multishell_msmt_noACT \
--atlases 4S456Parcels -w /work

Thank you again for your help,
Best,
Chiara

6

Hi @cmatti,

Good to hear!

Tractography always takes a long time, and with dense hcp data, FODs can take long to estimate too. As long as there’s not a memory overload, I’d recommend just being patient and letting it run.

I do not recommend the noACT pipelines. You should have freesurfer from the data release ans SDC should have been performed by qsiprep given you have two reverse phase encoded acquisitions. Thus I recommend the multishell HSVS workflow.

Best,
Steven

7

Hi @Steven,

I will wait then. I have a last question about QSIrecon. I have seen that QSIprep can run on GPU, is it the same for QSIrecon? If I put the flag --gpus all it doesn’t look like it’s recruting them.

Best,
Chiara

8

Hi @cmatti,

I do not think QSIRecon has any GPU optimization for GPUs.

Best,
Steven