QSIPrep 1.0.2 SynthSeg failure

Summary of what happened:

Yesterday, I downloaded and used the newly released QSIPrep version 1.0.2. Only the first dataset was processed successfully. Starting from the second dataset, each one failed with the error: “Node synthseg failed to run on host 65d6ab86fcaa.” The same datasets did not have this problem when processed with the previous version of QSIPrep. My system has 40 CPUs and 251.5 GiB of total memory.

Command used (and if a helper script was used, a link to the helper script or the command generated):

docker run --rm -m 50GB\
            -v "$INPUT_DIR:/input" \
            -v "$WORK_DIR:/work" \
            -v "$OUTPUT_DIR:/output" \
            pennlinc/qsiprep:1.0.2 \
            /input /output participant \
            --participant-label KW73020200929 \
            --b0-threshold 10 \
            --output-resolution 2 \
            --hmc-model eddy \
            --skip-bids-validation \
            --anatomical-template MNI152NLin2009cAsym \
            --work-dir /work \
            --nprocs 12 \
            --omp-nthreads 2

Version:

QSIPrep 1.0.2

Environment (Docker, Singularity / Apptainer, custom installation):

Docker

Data formatted according to a validatable standard? Please provide the output of the validator:

Relevant log outputs (up to 20 lines):

251113-07:37:26,636 nipype.workflow WARNING:
         Storing result file without outputs
251113-07:37:26,637 nipype.workflow WARNING:
         [Node] Error on "qsiprep_1_0_wf.sub_KW99220190809_wf.anat_preproc_wf.synthseg_wf.synthprep_1_0_wf/sub_KW99220190809_wf/anat_preproc_wf/synthseg_wf/synthseg)
251113-07:37:26,729 nipype.workflow ERROR:
         Node synthseg failed to run on host 29d9a3046556.
251113-07:37:26,745 nipype.workflow ERROR:
         Saving crash info to /output/sub-KW99220190809/log/20251113-073555_13a6319a-2e58-4b0a-1c/crash-20251113-073726-root-synthseg-c8a2f8c3-df80-4a67-9628-e2c2e0ca168a.txt
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.n run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", n
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", un_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", un_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg

Cmdline:
        mri_synthseg --cpu --i /work/qsiprep_1_0_wf/sub_KW99220190809_wf/anat_preproc_wf/dl_prekulled_to_reference/sub-KW99220190809_T1w_lps_trans.nii.gz --threads 1 --post sub-KW99220190809post.nii.gz --qc sub-KW99220190809_T1w_lps_trans_qc.csv --o sub-KW99220190809_T1w_lps_trans_ase
Stdout:
        SynthSeg 2.0
        using CPU, hiding all CUDA_VISIBLE_DEVICES
        using 1 thread
Stderr:
        Traceback (most recent call last):
          File "/opt/freesurfer/bin/mri_synthseg", line 2581, in <module>
            main()
          File "/opt/freesurfer/bin/mri_synthseg", line 123, in main
            predict(
          File "/opt/freesurfer/bin/mri_synthseg", line 195, in predict
            labels_segmentation, _ = get_list_labels(label_list=labels_segmentation)
          File "/opt/freesurfer/bin/mri_synthseg", line 1446, in get_list_labels
            label_list = np.array(reformat_to_list(label_list, load_as_numpy=True, dtype='int')
          File "/opt/freesurfer/bin/mri_synthseg", line 1534, in reformat_to_list
            if isinstance(var, (int, float, np.int, np.int32, np.int64, np.float, np.float32, n
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/numpy/__init__.py", line 3r__
            raise AttributeError(__former_attrs__[attr])
        AttributeError: module 'numpy' has no attribute 'int'.
        `np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing  by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you me.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use,ase note link for additional information.
        The aliases was originally deprecated in NumPy 1.20; for more details and guidance see lease note at:
            https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'in
Traceback:
        Traceback (most recent call last):
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/cor, in aggregate_outputs
            setattr(outputs, key, val)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/tray", line 325, in validate
            value = super().validate(objekt, name, value, return_pathlike=True)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/tray", line 135, in validate
            self.error(objekt, name, str(value))
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/traits/base_trait_handler.n error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_post' trait of a _SynthSegOutputSpec instance ike object or string representing an existing file, but a value of '/work/qsiprep_1_0_wf/sub_KWanat_preproc_wf/synthseg_wf/synthseg/sub-KW99220190809_T1w_lps_trans_post.nii.gz' <class 'str'>

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/cor, in run
            outputs = self.aggregate_outputs(runtime)
          File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/interfaces/base/cor, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/work/qsiprep_1_0_wf/sub_KW99220190809_wff/synthseg_wf/synthseg/sub-KW99220190809_T1w_lps_trans_post.nii.gz' for output 'out_post' of a ace

Screenshots / relevant information:


Hi @HanyuShao,

Can you try raising the -m docker memory limit? And also set the —mem flag in qsiprep accordingly (I would set it to half a gig lower than the docker -m flag, so something like -m 64 and —mem 63500).

Best,
Steven

Hi @HanyuShao,

I have also replicated this error, and it doesn’t appear to be a memory issue. We will look into this. It appears to be an old version of the mri_synthseg script from FreeSurfer that is not compatible with the higher version of numpy we use.

Best,
Steven

1 Like

Hi Steven,

Thank you for your prompt feedback regarding the mri_synthseg script compatibility issue with the higher version of numpy.

I would like to ask whether this bug is something we should try to resolve on our end, or if your team plans to release an updated version that addresses this issue? If it is recommended that we fix it ourselves, could you please provide some guidance or suggestions on how to proceed?

Looking forward to your advice. Thank you very much for your support!

Best regards,
Hanyu

Hi @HanyuShao,

It should be fixed in next release. In the meantime if you have a local version of FreeSurfer 7.4+ you can patch it in with -B /path/to/your/FreeSurfer:/opt/freesurfer in the singularity preamble, and that will allow the code to run.

You can see discussion on this GitHub issue: Errors related to `SynthSegOutputSpec` · Issue #977 · PennLINC/qsiprep · GitHub

Best,
Steven

1 Like

Thank you very much for your suggestion. I followed your instructions and patched in my local FreeSurfer 7.4+ installation . The pipeline now runs successfully without the previous error.

I appreciate your help and prompt support!