[NiPype] How can I iterate over subjects and then dynamically over sessions?

, , ,
1

My NiPype pipeline aims to iterate across a known subject list (e.g. ['sub-001', 'sub-002']), repeating the entire workflow for each subject. I have achieved this using an IdentityInterface with the iterables field set to the subject list.

The difficult part is that I also want to repeat the entire workflow again for each session. However, the session names are unknown until runtime, so they have to be set dynamically (unlike the subjects). I want something like this, but the n_selectSessions node is invalid - you can’t set a dynamic iterable like I’ve attempted to do:

# iterate across subject list
n_selectSubjects = Node(
    interface=IdentityInterface(
        fields=['subject_id'],
    ),
    name="select_subjects"
    # input: 'subject_id'
    # output: 'subject_id'
)
n_selectSubjects.iterables = ('subject_id', subject_list)

# select all session folders
n_selectSessions = Node(
    interface=sf.SelectFiles(
        templates={'session_dir' : '{subject_id_p}/ses*'},
        base_directory=bids_dir
    ),
    name='select_sessions'
)
wf.connect([
    (n_selectSubjects, n_selectSessions, [('subject_id', 'subject_id_p')])
])
n_selectSessions.iterables = ('session_dir') # INVALID

# select all files
n_selectFiles = Node(
    interface=sf.SelectFiles(
        templates=bids_templates
    ),
    iterfield='session_dir_p',
    name='select_files'
)
wf.connect([
    (n_selectSessions, n_selectFiles, [('session_dir', 'session_dir_p')])
])

Is there a way of achieving dynamic iterables or otherwise repeating the entire workflow both for subjects and sessions determined at runtime?

Thanks!

1 Like
2

This is one of the limitations that we’re seeking to address with Nipype 2. The new engine, Pydra, allows you to “split” over any input, whether statically set at build time or dynamically determined at run-time.

We recently had a hackathon, and I think the main limitation to converting workflows to Nipype 2 at this point is that the interfaces aren’t all recreated. We’re going to start having weekly hack sessions on Thursdays, if you would like to join. More information on the mattermost.

If you want to do this in Nipype 1, one approach would be:

from nipype.pipeline import engine as pe
def init_group_workflow(dataset_data):
    wf = pe.Workflow("top")
    wf.add([init_subject_workflow(dataset_data[subject])
            for subject in dataset_data])
    return wf

def init_subject_workflow(subject_data):
    wf = pe.Workflow(f"subject_{subject_data[id]}")
    # construct workflow for subject with iterable over session
    return wf

Assuming some kind of dataset_data representation of the dataset so you can know the sessions at build time.

3

In addition to what @effigies said, you could mix Pydra and a Nipype1 workflow if you want to. In pydra you could iterate over your participants and extract their variable sessions, and call the Nipype workflow for each combination of participant_id + session_id. also in Nipype1, for your specific use case, you could pregenerate this list of paired ids and run that through iterables as well.