QSIPrep: SynthSeg killed

Software Support
1

Summary of what happened:

I am running QSIPrep but encountered this crash. Can anyone please guide what file is missing or where exactly i need to do correction? I attached part of the terminal blow where it started giving the error

Command used (and if a helper script was used, a link to the helper script or the command generated):

apptainer run qsiprep-0.19.1.sif \ 
/home/user1nrx/software_research/tractography/bids_dir \
/home/user1nrx/software_research/tractography/output_dir \ 
participant \
  --output-resolution 1.2 \                                                       
  --fs-license-file /home/user1nrx/software_research/tractography/freesurfer_license/license.txt

Version:

0.19.1

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity / Apptainer

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

Terminal:

[Node] Error on "qsiprep_wf.single_subject_01_wf.anat_preproc_wf.synthseg_anat_wf.synthseg" (/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg)
240220-11:40:11,250 nipype.workflow ERROR:
	 Node synthseg failed to run on host srv-imgrm-04.
240220-11:40:11,256 nipype.workflow ERROR:
	 Saving crash info to /home/user1nrx/software_research/tractography/output_dir/qsiprep/sub-01/log/20240220-112610_9288b6ab-7469-469c-94ce-c77d971964e1/crash-20240220-114011-user1nrx-synthseg-f1934210-52bf-406c-8ce9-2617ea97fc8d.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.

Cmdline:
	mri_synthseg --i /home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-01_T1w_lps_trans.nii --threads 1 --post sub-01_T1w_lps_trans_post.nii.gz --qc sub-01_T1w_lps_trans_qc.csv --o sub-01_T1w_lps_trans_aseg.nii.gz
Stdout:

Stderr:
	Killed
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_seg' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' for output 'out_seg' of a SynthSeg interface


QSIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.

Cmdline:
	mri_synthseg --i /home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-01_T1w_lps_trans.nii --threads 1 --post sub-01_T1w_lps_trans_post.nii.gz --qc sub-01_T1w_lps_trans_qc.csv --o sub-01_T1w_lps_trans_aseg.nii.gz
Stdout:

Stderr:
	Killed
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_seg' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' for output 'out_seg' of a SynthSeg interface


Traceback (most recent call last):
  File "/usr/local/miniconda/bin/qsiprep", line 8, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/run.py", line 677, in main
    qsiprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node synthseg.

Cmdline:
	mri_synthseg --i /home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/pad_anat_reference_wf/resample_skulled_to_reference/sub-01_T1w_lps_trans.nii --threads 1 --post sub-01_T1w_lps_trans_post.nii.gz --qc sub-01_T1w_lps_trans_qc.csv --o sub-01_T1w_lps_trans_aseg.nii.gz
Stdout:

Stderr:
	Killed
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_seg' trait of a _SynthSegOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/synthseg_anat_wf/synthseg/sub-01_T1w_lps_trans_aseg.nii.gz' for output 'out_seg' of a SynthSeg interface


240220-11:45:32,271 nipype.workflow INFO:
	 [Node] Finished "synthseg", elapsed time 496.794309s.
240220-11:45:32,276 nipype.workflow WARNING:
	 Storing result file without outputs
240220-11:45:32,280 nipype.workflow WARNING:
	 [Node] Error on "qsiprep_wf.single_subject_02_wf.anat_preproc_wf.synthseg_anat_wf.synthseg" (/home/user1nrx/software_research/tractography/work/qsiprep_wf/single_subject_02_wf/anat_preproc_wf/synthseg_anat_wf/synthseg)
240220-11:49:30,312 nipype.workflow INFO:
	 [Node] Finished "synthseg", elapsed time 694.807802s.
240220-11:51:15,554 nipype.workflow INFO:
	 [Node] Finished "synthseg", elapsed time 697.937542s.```

### Screenshots / relevant information:
<!-- Below this line, provide any other information that might be meaningful. This could be screenshots, troubleshooting steps you have tried, or information about your operating system, etc. -->

_____
2

Hi @FARAH_NAZ,

In the future, please do not delete the post template for the Software Support category, as it prompts you for important information that can help us address your issue.

This suggests that you are out of memory. How much memory and CPUs are you giving the task? What version of QSIPrep are you using? In 0.19.1, synthseg is forced to single CPU which tends to lower memory usage.

Best,
Steven

3

HI,
Thanks for the response. The version i am using is 0.19.1.

terminal:

(base) user1nrx@srv-imgrm-04:~/software_research/tractography$ free
               total        used        free      shared  buff/cache   available
Mem:        32556448     3547440    26925088      589036     2083920    27951676
Swap:        2097148     1537304      559844

base) user1nrx@srv-imgrm-04:~/software_research/tractography$ cat /proc/meminfo
MemTotal:       32556448 kB
MemFree:        26858276 kB
MemAvailable:   27890708 kB
Buffers:           71524 kB
Cached:          1808628 kB
SwapCached:        86104 kB
Active:          2325212 kB
Inactive:        2225516 kB
Active(anon):    1644344 kB
Inactive(anon):  1673320 kB
Active(file):     680868 kB
Inactive(file):   552196 kB
Unevictable:      180144 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:         566436 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:               280 kB
Writeback:            20 kB
AnonPages:       2764864 kB
Mapped:           354600 kB
Shmem:            647088 kB
KReclaimable:     267664 kB
Slab:             599692 kB
SReclaimable:     267664 kB
SUnreclaim:       332028 kB
KernelStack:       38784 kB
PageTables:        63992 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    18375372 kB
Committed_AS:   14549284 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      155840 kB
VmallocChunk:          0 kB
Percpu:            35328 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:   464896 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      362292 kB
DirectMap2M:     8773632 kB
DirectMap1G:    24117248 kB
4

Hi @FARAH_NAZ,

Thanks, but there is still more information that we would need to best answer this. How are you running QSIPrep? Are you doing it in a way that you are sure all of those resources are available, or are you running it in a way that you can explicitly define the resources? Also, for readability, it would help if you format the code by surrounding it in tick marks (`) so it looks nicely formatted

like this

I made the changes to your above post, as an example.

Best,
Steven

5

I am running QSIPrep on data in BIDS format, and it was running smoothly, producing results for several subjects before encountering an error for one subject and terminating the program. I haven’t explicitly defined the resources, but I believe the resources required should be available.

The only command i used to run the qsiprep was like this:

(base) user1nrx@srv-imgrm-04:~/software_research/tractography$  apptainer run qsiprep-0.19.1.sif \                                                                /home/user1nrx/software_research/tractography/bids_dir /home/user1nrx/software_research/tractography/output_dir participant \
  --output-resolution 1.2 \                                                       
  --fs-license-file /home/user1nrx/software_research/tractography/freesurfer_license/license.txt

ps. Sorry I am a bit new on this website, so is taking a little time to familiarize with working here.

6

Hi @FARAH_NAZ,

Are you submitting this to a job scheduler? E.g., SLURM or SGE? If so, there are ways to define the resources (e.g., for SLURM you can use the SBATCH header or define resources with srun arguments). Also, for apptainer you should include the -e --containall flags to make sure nothing from your local environment is conflicting with software dependencies in the container. I also recommend adding a working directory with the -w flag. If memory is the problem you could also just run on subsets of subjects with the --participant-label argument.

Best,
Steven

1 Like
7

Thanks… The subsets of subjects using --participant-label worked for me.

1 Like