Nipype Workflow not using multiple inputs

1

Hey everyone, I have setup a simple preprocessing pipeline in Nipype. It works fine when using a single file as input. Now I have switched to using a DataGrabber and DataSink and testwise two input images but only 1 file is being processed.

Datagrabber

datasource = Node(interface=DataGrabber(infields=[‘subject_id’],
outfields=[‘struct’]),
name=“DataGrabber”)
datasource.inputs.base_directory = os.path.abspath(root_dir)
datasource.inputs.template = ‘%s.nii.gz’
datasource.inputs.template_args[‘struct’] = [[‘subject_id’]]
datasource.inputs.subject_id = subjects
datasource.inputs.sort_filelist = False

Datasink

datasink = Node(interface=DataSink(), name=“datasink”)
datasink.inputs.substitutions = [(‘subject_id’, ‘’)]
datasink.inputs.base_directory = os.path.abspath(’/home/fabiane/experiments’)

preprocessing = Workflow(name=“preprocessing”)
preprocessing.connect(datasource, “struct”, reg, “moving_image”)
preprocessing.connect(reg, “warped_image”, bet, “in_file”)
preprocessing.connect(bet, “out_file”, datasink, “processed”)

When looking at the output of ANTS registration I can see that both files are identified (one ending on 58, one ending on 61) but it does not seem to be the correct usage. It starts only one job for each processing step. My goal is to parallelize the processing for multiple images.

Running node “Registration” (“nipype.interfaces.ants.registration.Registration”), a CommandLine Interface with command:
antsRegistration --collapse-output-transforms 0 --dimensionality 3 --initial-moving-transform [ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0058.nii.gz, 1 ] --initialize-transforms-per-stage 0 --interpolation Linear --output [ output_, INTERNAL_WARPED.nii.gz ] --transform Translation[ 0.1 ] --metric Mattes[ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0058.nii.gz, 1, 32, Regular, 0.3 ] --convergence [ 10000x111110x11110, 1e-08, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 6x4x2 --use-estimate-learning-rate-once 1 --use-histogram-matching 0 --transform Rigid[ 0.1 ] --metric Mattes[ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0058.nii.gz, 1, 32, Regular, 0.3 ] --convergence [ 10000x111110x11110, 1e-08, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 3x2x1 --use-estimate-learning-rate-once 1 --use-histogram-matching 0 --transform Affine[ 0.1 ] --metric Mattes[ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0058.nii.gz, 1, 32, Regular, 0.3 ] --convergence [ 10000x111110x11110, 1e-08, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 3x2x1 --use-estimate-learning-rate-once 1 --use-histogram-matching 0 --transform SyN[ 0.2, 3.0, 0.0 ] --metric Mattes[ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0058.nii.gz, 0.5, 32 ] --metric CC[ /analysis/fabiane/data/Templates/T1.nii, /analysis/fabiane/data/ADNI_subset/ADNI_023_S_0061.nii.gz, 0.5, 4 ] --convergence [ 100x50x30, -0.01, 5 ] --smoothing-sigmas 1.0x0.5x0.0vox --shrink-factors 4x2x1 --use-estimate-learning-rate-once 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.0, 1.0 ] --write-composite-transform 1.

Also how can I include the subject_id in the filename in the end?

Thanks!
Fabi

2

You might want to check Nipype iterables.
Examples of preprocessing for multiple images are also in this tutorial notebook. Hopefully this will help.

1 Like
3

I am now using the IdentityInterface and it works great, thanks! :slight_smile:

Nevertheless, I am running into a different problem now when trying to distribute the tasks over multiple machines using the Ipython (ipyparallel) plugin. It works fine if the controller and engines are on the same machine, but whenever I try to run the engines in a distributed manner on different machines if fails. Apparently it does not find the temporary files which the nodes output… They seem to be in a /tmp directory i.e. locally on one of the machines. I don’t understand why I cannot change the directories (I tried the base_dir) nor why the different steps are not done on the same machine. I.e. first_step[file1]->second_step[file1]->last_step[file1] should all be on the same machine, right? But then they should have access to the same /tmp directory.

FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmpsv83oyqc/preprocessing/_subject_id_ADNI_136_S_0426/DataGrabber/result_DataGrabber.pklz’

Any help will be appreciated!

4

I’m glad IdentityInterface works for you. I don’t have experience with the ipyparallel plugin, but other nipype user can probably help you. You might get help faster if you open a new issue with a new more accurate title.

5

Okay thanks, I was thinking about it but didn’t want to spam right away with a new thread.