Create subdirectories with subject id

[mri]

Hi,

I’m trying to find a way to create subdirectories with the subject id within a base directory. For example

subject_list = [“01”, “02”, “03”]

infosource = Node(IdentityInterface(fields=[“subject_id”]), name=“infosource”)
infosource.iterables = [(“subject_id”, subject_list)]

func_file = {“func”: os.path.join(“sub-{subject_id}”, “*.nii.gz”)}
selectfiles = Node(SelectFiles(func_file, base_directory=’/data’), name=‘selectfiles’)
merge = Node(fsl.Merge(merged_file=‘data.nii.gz’),name=‘merge’)

wf = Workflow(name=“wf”)
wf.connect(infosource, “subject_id”, selectfiles, “subject_id”)
wf.connect(selectfiles, “func”, merge, “in_files”)

How can I modify the script to save the merged files in the subdirectory {subject_id} (ie 01, 02 and 03) within the directory /data?

/data/wf/01/merged_file_sub-01
/data/wf/02/merged_file_sub-02
/data/wf/03/merged_file_sub-03

Thanks

[mgxd]

@mri

Have you tried using DataSink? Perhaps something like:

...
sinker = Node(DataSink(), name='sinker')
sinker.inputs.base_directory = '/your/base/dir'
wf.connect(infosource, 'subject_id', sinker, 'container')
wf.connect(merge, 'out', sinker, 'merged_files')

[mri]

Thank you @mgxd

I tried the following based on your suggestion

subject_list = [“01”, “02”, “03”]

infosource = Node(IdentityInterface(fields=[“subject_id”]), name=“infosource”)
infosource.iterables = [(“subject_id”, subject_list)]

func_file = {“func”: os.path.join(“sub-{subject_id}”, “*.nii.gz”)}
selectfiles = Node(SelectFiles(func_file, base_directory=’/data’), name=‘selectfiles’)
merge = Node(fsl.Merge(merged_file=‘data.nii.gz’),name=‘merge’)

wf = Workflow(name=“wf”)
wf.connect(infosource, “subject_id”, selectfiles, “subject_id”)
wf.connect(selectfiles, “func”, merge, “in_files”)

sinker = Node(DataSink(), name=‘sinker’)
sinker.inputs.base_directory = ‘/data/wf’
wf.connect(infosource, ‘subject_id’, sinker, ‘container’)
wf.connect(merge, ‘merged_file’, sinker, ‘merged_files’)

wf.run()

However this creates two subfolders within wf/:

/data/wf/01/merged_files/_subject_id_01/data.nii.gz
/data/wf/02/merged_files/_subject_id_02/data.nii.gz
/data/wf/03/merged_files/_subject_id_03/data.nii.gz

whereas I’d like to store the files in:
/data/wf/01/data.nii.gz
/data/wf/02/data.nii.gz
/data/wf/03/data.nii.gz

I thought this might work:
wf.connect(infosource, ‘subject_id’, sinker, ‘container’)
wf.connect(merge, ‘merged_file’, sinker, ‘container’)

but it throws an Exception…

[mgxd]

you could use the substitutions argument

subs = [('merged_files/_subject_id_', '')]
sinker.inputs.substitutions = subs

[mri]

Thanks @mgxd, that makes sense.

I was also wondering how to create the following folder structure

/data/wf/sinker/01/merged_file/data.nii.gz
/data/wf/sinker/02/merged_file/data.nii.gz
/data/wf/sinker/03/merged_file/data.nii.gz

I tried:

sinker = Node(DataSink(), name=“sinker”)
sinker.inputs.base_directory = “/data/wf/sinker”
subs = [(“subject_id*”, “”)]
sinker.inputs.substitutions = subs

wf.connect(infosource, “subject_id”, sinker, “container”)
wf.connect(merge, “merged_file”, sinker, “merged_file”)

But I got
/data/wf/sinker/01/merged_file/_subject_id_01/data.nii.gz
/data/wf/sinker/02/merged_file/_subject_id_02/data.nii.gz
/data/wf/sinker/03/merged_file/_subject_id_03/data.nii.gz

Won’t wildcards such as * work inside substitutions? Is there a way to achieve my desired folder structure?

[nikadon]

It is doable using datasink.inputs.regexp_substitutions.

See: python regex (re)

Following is a short snippet utilising re.

import re
source_text = '/anatomical/sub-sb256/'
target_text = re.subn(r'/(.*)/(sub-sb(\d){3})/', r'/\2/\1/', source_text)[0]
print(source_text)
print(target_text)

From here it should be super-easy to tailor it to your needs…

[mri]

Thanks @nikadon, that’s exactly what I was looking for.