MRIQC failing with OSError: [Errno 28] No space left on device

Software Support
1

Summary of what happened:

I am using MRIQC on an apptainer image on Alliance Can Beluga cluster. I have started to run it with the workdir as the local scratch of the compute node (called $SLURM_TMPDIR on this cluster) to avoid overcrowding my local scratch ($SCRATCH) and also it seems to be improving performance quite a bit. It worked once, but when I re-run it on the same participant and some others and now it seems (at least to me) like MRIQCs config is setup with the former local scratch of the node, which is now not available in further jobs. I have tried to debug this with AllianceCan staff without success so far.

This is the main error identified (se below for further detailed output and error log)

  File "/opt/conda/lib/python3.11/site-packages/nipype/utils/filemanip.py", line 719, in
savepkl
    with pkl_open(tmpfile, "wb") as pkl_file:
  File "/opt/conda/lib/python3.11/gzip.py", line 344, in close
    myfileobj.close()
OSError: [Errno 28] No space left on device

Any help would be appreciated, thanks a lot!

Command used (this is a debug script, normally I run it with more cpus, time and RAM):

#!/bin/bash
#SBATCH --job-name=MRIQC
#SBATCH --output=MRIQC_%j_%a.out
#SBATCH --error=MRIQC_%j_%a.err
#SBATCH --time=0:30:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=40G

 
echo "Loading the necessary module..."
module load apptainer/1.3.5
 
export SUBJECT_NAME='sub-RCX136'
# Ensure SUBJECT_NAME is provided
if [ -z "$SUBJECT_NAME" ]; then
  echo "Error: SUBJECT_NAME environment variable is not set."
  exit 1
fi
 

 
# Define paths
input_dir=/home/pagag24/projects/def-amichaud/share/GutBrain/data2
output_dir=/home/pagag24/projects/def-amichaud/share/GutBrain/derivatives/mriqc
CONTAINER_IMAGE=/home/pagag24/myimages/mriqc-24.0.2.sif
 
# Echo all of the directories
echo "Input directory: $input_dir"
echo "Output directory: $output_dir"
echo "Container image: $CONTAINER_IMAGE"
 
# Print the PID of the script
echo "Script PID: $$"
 
env | grep -i scratch
env | grep -i tmp
 
echo "About to run MRIQC with:"
echo "Container: $CONTAINER_IMAGE"
echo "Input: $input_dir"
echo "Output: $output_dir"
echo "Subject: $SUBJECT_NAME"
echo "SLURM_TMPDIR: $SLURM_TMPDIR"
ls -ld $input_dir $output_dir $SLURM_TMPDIR
 
date
 
mkdir $SLURM_TMPDIR/HZ_tmp
apptainer run -C -W $SLURM_TMPDIR \
     -B /home/pagag24/projects/def-amichaud/share/GutBrain:/GutBrain \
     -B $SLURM_TMPDIR \
     -B $SLURM_TMPDIR/HZ_tmp:/tmp \
     $CONTAINER_IMAGE \
     /GutBrain/data2 $SLURM_TMPDIR  participant \
     --participant-label $SUBJECT_NAME \
     --n_procs 4  \
     --mem_gb 40 \
     --no-sub \
     --bids-database-wipe
sleep  10000
 
echo "MRIQC processing complete."

Version:

Running MRIQC version 24.1.0.dev0+gd5b13cb5.d20240826

Environment (Docker, Singularity / Apptainer, custom installation):

Running it with apptainer on beluga (alliance can cluster)

Data formatted according to a validatable standard? Please provide the output of the validator:

I have issues with running the validator on the cluster recently.

Relevant log outputs (up to 20 lines):

(nilearn_ENV) [pagag24@beluga1 scripts]$ cat MRIQC_56377923_4294967294.err
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main
    mriqc_wf.run(**_plugin)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 196, in run
    notrun.append(self._clean_queue(jobid, graph, result=result))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue
    raise RuntimeError(''.join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node PlotMosaicNoise.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/nireports/interfaces/mosaic.py", line 143, in _run_interface
            plot_mosaic(
          File "/opt/conda/lib/python3.11/site-packages/nireports/reportlets/mosaic.py", line 762, in plot_mosaic
            fig.savefig(out_file, format="svg", dpi=300, bbox_inches="tight")
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/figure.py", line 3390, in savefig
            self.canvas.print_figure(fname, **kwargs)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backend_bases.py", line 2193, in print_figure
            result = print_method(
                     ^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backend_bases.py", line 2043, in <lambda>
            print_method = functools.wraps(meth)(lambda *args, **kwargs: meth(
                                                                         ^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backends/backend_svg.py", line 1339, in print_svg
            self.figure.draw(renderer)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/artist.py", line 95, in draw_wrapper
            result = draw(artist, renderer, *args, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
            return draw(artist, renderer)
                   ^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/figure.py", line 3154, in draw
            mimage._draw_list_compositing_images(
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
            a.draw(renderer)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
            return draw(artist, renderer)
                   ^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/axes/_base.py", line 3070, in draw
            mimage._draw_list_compositing_images(
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/image.py", line 157, in _draw_list_compositing_images
            a.draw(renderer)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
            return draw(artist, renderer)
                   ^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/image.py", line 647, in draw
            renderer.draw_image(gc, l, b, im, trans)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backends/backend_svg.py", line 990, in draw_image
            self.writer.element(
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backends/backend_svg.py", line 241, in element
            self.start(tag, attrib, **extra)
          File "/opt/conda/lib/python3.11/site-packages/matplotlib/backends/backend_svg.py", line 165, in start
            self.__write(f' {k}={v}')
        OSError: [Errno 28] No space left on device


slurmstepd: error: *** JOB 56377923 ON bc12047 CANCELLED AT 2025-06-10T18:28:44 ***

Screenshots / relevant information:

this is the script I ran with success in the past, but want to move away from because using the compute local tmp directory is best practice.

#!/bin/bash

#SBATCH --job-name=MRIQC

#SBATCH --output=MRIQC_%j_%a.out

#SBATCH --error=MRIQC_%j_%a.err

#SBATCH --time=3:00:00

#SBATCH --ntasks=1

#SBATCH --cpus-per-task=1

#SBATCH --mem-per-cpu=40G

#SBATCH --mail-user=patrick.gagnon.21@ulaval.ca

#SBATCH --mail-type=BEGIN,FAIL,END

#SBATCH --array=6

 

echo "Loading the necessary module..."

module load apptainer/1.3.5

 

# To run locally for a test, uncomment the following line

#SLURM_ARRAY_TASK_ID=1

 

if [ "$SLURM_ARRAY_TASK_ID" -ge 0 ] && [ "$SLURM_ARRAY_TASK_ID" -le 9 ]; then

  name="RGC90${SLURM_ARRAY_TASK_ID}"

else

  name="RGC9${SLURM_ARRAY_TASK_ID}"

fi

 

# Define paths

input_dir=${HOME}/projects/def-amichaud/share/GutBrain/data

output_dir=${HOME}/projects/def-amichaud/share/GutBrain/data/derivatives/mriqc

SCRATCH_DIR=${SCRATCH}/GutBrain/mriqc

CONTAINER_IMAGE=${HOME}/myimages/mriqc-24.0.2.sif

 

# Echo all of the directories

echo "Input directory: $input_dir"

echo "Output directory: $output_dir"

echo "Scratch directory: $SCRATCH_DIR"

echo "Container image: $CONTAINER_IMAGE"

 

# export APPTAINER_BIND=${input_dir},${output_dir},${SCRATCH_DIR}

 

# Print the PID of the script

echo "Script PID: $$"

 

apptainer run $CONTAINER_IMAGE $input_dir $output_dir participant \

    --participant-label $name \

    --n_procs 1 \

    --mem_gb 40 \

    --work-dir $SCRATCH_DIR \

    --no-sub \

 

echo "MRIQC processing complete."
2

Hi @pganon32,

Please specify a MRIQC work space with -w (separate from apptainer preamble), ideally in non space limited temporary compute space.

Please add --writable-tmpfs to your apptainer run preamble.

You can also try replacing -C with -e in apptainer preamble.

Best,
Steven

3

Hi @Steven Steven,

Thanks a lot, and that fixed it!

A bit out of topic for this thread, but is it normal that I get a synthstrip OOM error with 8 cpus and 80 GB of RAMS specified?

I have listed below my participant tree, error log and output log.

Thanks again!

For context here is a tree of a test participant i am running:

../data2/sub-RCX136
├── ses-1
│   ├── anat
│   │   ├── sub-RCX136_ses-1_T1w.json
│   │   └── sub-RCX136_ses-1_T1w.nii.gz
│   ├── fmap
│   │   ├── sub-RCX136_ses-1_run-1_fieldmap.json
│   │   ├── sub-RCX136_ses-1_run-1_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-1_run-1_magnitude.json
│   │   ├── sub-RCX136_ses-1_run-1_magnitude.nii.gz
│   │   ├── sub-RCX136_ses-1_run-2_fieldmap.json
│   │   ├── sub-RCX136_ses-1_run-2_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-1_run-2_magnitude.json
│   │   └── sub-RCX136_ses-1_run-2_magnitude.nii.gz
│   └── func
│       ├── sub-RCX136_ses-1_task-BDM_run-1_bold.json
│       ├── sub-RCX136_ses-1_task-BDM_run-1_bold.nii.gz
│       ├── sub-RCX136_ses-1_task-BDM_run-2_bold.json
│       ├── sub-RCX136_ses-1_task-BDM_run-2_bold.nii.gz
│       ├── sub-RCX136_ses-1_task-BDM_run-3_bold.json
│       └── sub-RCX136_ses-1_task-BDM_run-3_bold.nii.gz
├── ses-2
│   ├── anat
│   │   ├── sub-RCX136_ses-2_T1w.json
│   │   └── sub-RCX136_ses-2_T1w.nii.gz
│   ├── fmap
│   │   ├── sub-RCX136_ses-2_run-1_fieldmap.json
│   │   ├── sub-RCX136_ses-2_run-1_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-2_run-1_magnitude.json
│   │   ├── sub-RCX136_ses-2_run-1_magnitude.nii.gz
│   │   ├── sub-RCX136_ses-2_run-2_fieldmap.json
│   │   ├── sub-RCX136_ses-2_run-2_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-2_run-2_magnitude.json
│   │   └── sub-RCX136_ses-2_run-2_magnitude.nii.gz
│   └── func
│       ├── sub-RCX136_ses-2_task-BDM_run-1_bold.json
│       ├── sub-RCX136_ses-2_task-BDM_run-1_bold.nii.gz
│       ├── sub-RCX136_ses-2_task-BDM_run-2_bold.json
│       ├── sub-RCX136_ses-2_task-BDM_run-2_bold.nii.gz
│       ├── sub-RCX136_ses-2_task-BDM_run-3_bold.json
│       └── sub-RCX136_ses-2_task-BDM_run-3_bold.nii.gz
├── ses-3
│   ├── anat
│   │   ├── sub-RCX136_ses-3_T1w.json
│   │   └── sub-RCX136_ses-3_T1w.nii.gz
│   ├── fmap
│   │   ├── sub-RCX136_ses-3_run-1_fieldmap.json
│   │   ├── sub-RCX136_ses-3_run-1_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-3_run-1_magnitude.json
│   │   ├── sub-RCX136_ses-3_run-1_magnitude.nii.gz
│   │   ├── sub-RCX136_ses-3_run-2_fieldmap.json
│   │   ├── sub-RCX136_ses-3_run-2_fieldmap.nii.gz
│   │   ├── sub-RCX136_ses-3_run-2_magnitude.json
│   │   └── sub-RCX136_ses-3_run-2_magnitude.nii.gz
│   └── func
│       ├── sub-RCX136_ses-3_task-BDM_run-1_bold.json
│       ├── sub-RCX136_ses-3_task-BDM_run-1_bold.nii.gz
│       ├── sub-RCX136_ses-3_task-BDM_run-2_bold.json
│       ├── sub-RCX136_ses-3_task-BDM_run-2_bold.nii.gz
│       ├── sub-RCX136_ses-3_task-BDM_run-3_bold.json
│       └── sub-RCX136_ses-3_task-BDM_run-3_bold.nii.gz
└── ses-4
    ├── anat
    │   ├── sub-RCX136_ses-4_T1w.json
    │   └── sub-RCX136_ses-4_T1w.nii.gz
    ├── fmap
    │   ├── sub-RCX136_ses-4_run-1_fieldmap.json
    │   ├── sub-RCX136_ses-4_run-1_fieldmap.nii.gz
    │   ├── sub-RCX136_ses-4_run-1_magnitude.json
    │   ├── sub-RCX136_ses-4_run-1_magnitude.nii.gz
    │   ├── sub-RCX136_ses-4_run-2_fieldmap.json
    │   ├── sub-RCX136_ses-4_run-2_fieldmap.nii.gz
    │   ├── sub-RCX136_ses-4_run-2_magnitude.json
    │   └── sub-RCX136_ses-4_run-2_magnitude.nii.gz
    └── func
        ├── sub-RCX136_ses-4_task-BDM_run-1_bold.json
        ├── sub-RCX136_ses-4_task-BDM_run-1_bold.nii.gz
        ├── sub-RCX136_ses-4_task-BDM_run-2_bold.json
        ├── sub-RCX136_ses-4_task-BDM_run-2_bold.nii.gz
        ├── sub-RCX136_ses-4_task-BDM_run-3_bold.json
        └── sub-RCX136_ses-4_task-BDM_run-3_bold.nii.gz

See error log below:

/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide
  bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread
/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide
  bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread
/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide
  bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread
/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide
  bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main
    mriqc_wf.run(**_plugin)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 196, in run
    notrun.append(self._clean_queue(jobid, graph, result=result))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue
    raise RuntimeError(''.join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthstrip.

Cmdline:
        synthstrip -b 1 -i /localscratch/pagag24.56406317.0/mriqc_wf/funcMRIQC/synthstrip_wf/aa27be3db20927618ae36462adf8bd61418e1909/pre_n4/clipped_corrected.nii.gz --model /opt/freesurfer/models/synthstrip.1.pt -n 1 -o clipped_corrected_desc-brain.nii.gz -m clipped_corrected_desc-brain_mask.nii.gz
Stdout:

Stderr:
        /opt/conda/lib/python3.11/site-packages/mriqc/synthstrip/cli.py:119: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
          checkpoint = torch.load(modelfile, map_location=device)
        Killed
Traceback:
        RuntimeError: subprocess exited with code 137.

slurmstepd: error: *** JOB 56406317 ON bc11952 CANCELLED AT 2025-06-11T13:02:16 ***
slurmstepd: error: Detected 1 oom_kill event in StepId=56406317.batch. Some of the step tasks have been OOM Killed.

and output log :

(i realise my outdir is set as my workdir also here, I will re-run with a proper output dir also.

Wed Jun 11 07:15:26 EDT 2025
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+gd5b13cb5.d20240826
  ----------------------------------------------------------------

  NOTICE
  Copyright © The NiPreps Developers.

  This product includes software developed by
  the NiPreps Community (https://nipreps.org/).

  Portions of this software were developed at the Department of
  Psychology at Stanford University, Stanford, CA, US.

  This software contains code ultimately derived from the
  PCP Quality Assessment Protocol (QAP;
  http://preprocessed-connectomes-project.org/quality-assessment-protocol)
  by C. Craddock, S. Giavasis, D. Clark, Z. Shezhad, and J. Pellman.

  This software is also distributed as a Docker container image.
  The bootstrapping file for the image ("Dockerfile") is licensed
  under the MIT License.

  ----------------------------------------------------------------

  * BIDS dataset path: /GutBrain/data2.
  * Output folder: /localscratch/pagag24.56406317.0.
  * Analysis levels: ['participant'].
------------------------------------------------------------------

2025-06-11 11:15:38 | IMPORTANT | mriqc            | Extracting metadata and entities for 4 input runs of modality 't1w'...
2025-06-11 11:15:38 | IMPORTANT | mriqc            | File size ('t1w'): 0.01|0.01 GB [maximum|average].
2025-06-11 11:15:38 | IMPORTANT | mriqc            | Extracting metadata and entities for 12 input runs of modality 'bold'...
2025-06-11 11:15:41 | IMPORTANT | mriqc            | File size ('bold'): 0.05|0.04 GB [maximum|average].
2025-06-11 11:15:53 | IMPORTANT | mriqc            | Building MRIQC's workflows...
2025-06-11 11:16:02 | IMPORTANT | mriqc            | Workflow building finished (exit code 0).
2025-06-11 11:26:33 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2025-06-11 11:26:35 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2025-06-11 11:26:35 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2025-06-11 11:26:38 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2025-06-11 11:27:56 | INFO     | niworkflows      | Terminal outputs of initialization saved (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/stderr.nipype-init).
2025-06-11 11:27:56 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json.
2025-06-11 11:27:56 | INFO     | niworkflows      | Retry #1, commandline:
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2025-06-11 11:27:56 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json.
2025-06-11 11:27:56 | INFO     | niworkflows      | Retry #1, commandline:
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2025-06-11 11:27:57 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json.
2025-06-11 11:27:57 | INFO     | niworkflows      | Retry #1, commandline:
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2025-06-11 11:28:02 | INFO     | niworkflows      | Terminal outputs of initialization saved (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/stderr.nipype-init).
2025-06-11 11:28:02 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json.
2025-06-11 11:28:02 | INFO     | niworkflows      | Retry #1, commandline:
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/fixed_masked.nii.gz, /localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2025-06-11 11:28:14 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2025-06-11 11:28:14 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2025-06-11 11:28:16 | INFO     | niworkflows      | Report - setting fixed (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/fixed_masked.nii.gz) and moving (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/e622eb31005bdf75ce1dac3e5a69907eba6b6632/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images
2025-06-11 11:28:16 | INFO     | niworkflows      | Generating visual report
2025-06-11 11:28:16 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2025-06-11 11:28:16 | INFO     | niworkflows      | Report - setting fixed (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/fixed_masked.nii.gz) and moving (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/f07679cddf7a9431e0cdcc353afef00efb2cbfe7/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images
2025-06-11 11:28:16 | INFO     | niworkflows      | Generating visual report
2025-06-11 11:28:17 | INFO     | niworkflows      | Report - setting fixed (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/fixed_masked.nii.gz) and moving (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/2baae387d0561701a63d0ec722a4ded0207c94b2/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images
2025-06-11 11:28:17 | INFO     | niworkflows      | Generating visual report
2025-06-11 11:28:21 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2025-06-11 11:28:22 | INFO     | niworkflows      | Report - setting fixed (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/fixed_masked.nii.gz) and moving (/localscratch/pagag24.56406317.0/mriqc_wf/anatMRIQC/SpatialNormalization/bd06287f57769cda6439b953bd1aa59def881fde/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images
2025-06-11 11:28:22 | INFO     | niworkflows      | Generating visual report
2025-06-11 11:31:05 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=5.7996)
2025-06-11 11:31:05 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=5.7996)
2025-06-11 11:31:05 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=5.7996)
2025-06-11 11:31:11 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.1482)
2025-06-11 11:31:11 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.1482)
2025-06-11 11:31:11 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.1482)
2025-06-11 11:31:19 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.5765)
2025-06-11 11:31:19 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.5765)
2025-06-11 11:31:19 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.5765)
2025-06-11 11:31:21 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.9607)
2025-06-11 11:31:21 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.9607)
2025-06-11 11:31:21 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=4.9607)
2025-06-11 11:35:55 | WARNING  | nipype.workflow  | [Node] Error on "mriqc_wf.funcMRIQC.synthstrip_wf.synthstrip" (/localscratch/pagag24.56406317.0/mriqc_wf/funcMRIQC/synthstrip_wf/aa27be3db20927618ae36462adf8bd61418e1909/synthstrip)
2025-06-11 11:35:56 | ERROR    | nipype.workflow  | Node synthstrip.a00 (taskid=305) crashed: /localscratch/pagag24.56406317.0/logs/crash-20250611-113556-pagag24-synthstrip.a00-e6564139-7806-47b4-bada-57072dfde9f6.txt```
4

Hi @pganon32,

Synthstrip does use a lot of memory, but that is still above normal. You can try limiting omp nthreads to something like 2.

Best,
Steven

5

Hi @Steven

So I ended-up just reducing my number of cores to 4 cpus and ran 4 sessions (4 t1s and 12 bold runs) in 49 minutes. Max RAM hit 51 GBs.

Everything seemed to run pretty smoothly!

Thanks a lot for your help, I have a few tweaks to do on the script but once I am sure it batches well I will post it back here for reference.

Cheers,

Patrick

1 Like