Summary of what happened:
I have a BIDS-like dataset that looks like this:
test_gunzip/
└── sub-INDI1W934
└── ses-01
└── func
├── sub-INDI1W934_ses-01_task-faces_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii.gz
├── sub-INDI1W934_ses-01_task-mid_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii.gz
└── sub-INDI1W934_ses-01_task-nback_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii.gz
I already know that in the future, there will be more sessions for each subject and that for some subjects the second session could be missing (due to drop outs). I want to implement a nipype script that runs a first-level analysis (using SPM) for each subject, each task and each session. It makes sense to write my code in a fashion that I iterate over each combination of subjects, task, and sessions knowing that this will lead to some branches failing (in this case, for all ses-02-files because right now, these files don’t exist, and I also cannot guarantee that each subject will participate in each session). The problem is that the gunzip-node (I need to unzip the .nii.gz files for SPM) never exits. I would expect that all gunzip nodes for session 02 should fail because I provide an invalid None value to them.
Command used (and if a helper script was used, a link to the helper script or the command generated):
from bids import BIDSLayout
from nipype import Workflow,Node
from nipype.interfaces.utility import Function
from nipype.algorithms.misc import Gunzip
# set task specific data directory
data_directory = '/home/johannes.wiesner/work/testing/test_gunzip/'
# get functional images as dataframe
data_df = BIDSLayout(data_directory,validate=False).to_df()
# define a node that gets data
# (if you cannot find a file for this subject, task, session, return None)
def get_files(data_df,subject,task,session):
# query dataframe for a specific subject, specific task and specific session
subject_df = data_df.query("subject==@subject & task==@task & session==@session")
subject_df = subject_df.set_index('suffix')
if not subject_df.empty:
func = subject_df.at['bold','path']
else:
func = None
return func
file_getter = Node(Function(input_names=['data_df','subject','task','session'],
output_names='func',
function=get_files),
name='file_getter')
file_getter.inputs.data_df = data_df
file_getter.iterables = [('subject',data_df['subject'].unique()),
('task',['nback','faces','mid']),
('session',['01','02'])]
# unzip the images
gunzipper = Node(Gunzip(),name='gunzipper')
# run the workflow
wf = Workflow(name='debug_gunzip')
wf.base_dir = './'
wf.config['execution']['crashfile_format'] = 'txt'
wf.connect(file_getter,'func',gunzipper,'in_file')
wf_results = wf.run('MultiProc',plugin_args={'n_procs':3})
Version:
nipype version: 1.8.6
Environment (Docker, Singularity, custom installation):
Nipype in a conda environment on a Ubuntu machine (also tested inside a docker container)
Relevant log outputs (up to 20 lines):
Nipype seems to have struggle setting up the gunzipper directory for all ses-02 folders. For the ses-01 folders everything works (see the directory tree of my workflow directory below).
FileNotFoundError: /home/johannes.wiesner/work/projects/project_indicate/code/debug_gunzip/_session_02_subject_INDI1W934_task_mid/gunzipper/result_gunzipper.pklz
230628-16:56:47,637 nipype.workflow INFO:
[MultiProc] Running 3 tasks, and 12 jobs ready. Free memory (GB): 55.68/56.28, Free processors: 0/3.
Currently running:
* debug_gunzip.gunzipper
* debug_gunzip.gunzipper
* debug_gunzip.gunzipper
Screenshots / relevant information:
Output from the workflow directory:
debug_gunzip
├── d3.js
├── graph1.json
├── graph.json
├── index.html
├── _session_01_subject_INDI1W934_task_faces
│ ├── file_getter
│ │ ├── _0xb0d52d4c470d86121cceb72f3ee346bc.json
│ │ ├── _inputs.pklz
│ │ ├── _node.pklz
│ │ ├── _report
│ │ │ └── report.rst
│ │ └── result_file_getter.pklz
│ └── gunzipper
│ ├── _0xd14c9e25a031bbfb9fb2033aa0890176.json
│ ├── _inputs.pklz
│ ├── _node.pklz
│ ├── _report
│ │ └── report.rst
│ ├── result_gunzipper.pklz
│ └── sub-INDI1W934_ses-01_task-faces_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii
├── _session_01_subject_INDI1W934_task_mid
│ ├── file_getter
│ │ ├── _0x98b1fd56fee60b072781e9833e6b7443.json
│ │ ├── _inputs.pklz
│ │ ├── _node.pklz
│ │ ├── _report
│ │ │ └── report.rst
│ │ └── result_file_getter.pklz
│ └── gunzipper
│ ├── _0x6ed69c476e0a6186ff1f554797d72ff6.json
│ ├── _inputs.pklz
│ ├── _node.pklz
│ ├── _report
│ │ └── report.rst
│ ├── result_gunzipper.pklz
│ └── sub-INDI1W934_ses-01_task-mid_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii
├── _session_01_subject_INDI1W934_task_nback
│ ├── file_getter
│ │ ├── _0x5ad9a982e728af9d5edd169f53c065e8.json
│ │ ├── _inputs.pklz
│ │ ├── _node.pklz
│ │ ├── _report
│ │ │ └── report.rst
│ │ └── result_file_getter.pklz
│ └── gunzipper
│ ├── _0x48c76f3c4131b759a51b226c0f25a661.json
│ ├── _inputs.pklz
│ ├── _node.pklz
│ ├── _report
│ │ └── report.rst
│ ├── result_gunzipper.pklz
│ └── sub-INDI1W934_ses-01_task-nback_space-MNI152NLin2009cAsym_res-2_desc-preproc_bold.nii
├── _session_02_subject_INDI1W934_task_faces
│ └── file_getter
│ ├── _0xf30317cbe64f29788e0aaa0734d9f96c.json
│ ├── _inputs.pklz
│ ├── _node.pklz
│ ├── _report
│ │ └── report.rst
│ └── result_file_getter.pklz
├── _session_02_subject_INDI1W934_task_mid
│ └── file_getter
│ ├── _0xeee937a981c3415ad4039d1ec13e9ab5.json
│ ├── _inputs.pklz
│ ├── _node.pklz
│ ├── _report
│ │ └── report.rst
│ └── result_file_getter.pklz
└── _session_02_subject_INDI1W934_task_nback
└── file_getter
├── _0x6657d86580f0594179158cccad03220f.json
├── _inputs.pklz
├── _node.pklz
├── _report
│ └── report.rst
└── result_file_getter.pklz