Connect node with multiple output files to another node

,
1

Hi,
I’m trying to connect a node (files_node) to another node (convert_all_files) but using different output files from files_node. I tried

files_node = Node(...) #outputs: many_files (a list of files) and single_file (a single file)
convert_all_files = MapNode(..., iterfield=['in_file'])
wf.connect(files_node, convert_all_files, [(('many_files','single_file'), 'in_file')])

but it throws an exception “IndexError: tuple index out of range”. Is there a way to fix this?

Thanks!

2

Can you share all the code? I fear that the important details went away in your pseudo-code.

3

From what I understand you’re trying to connect two outputs from files_node to one input from convert_all_files. The connect doesn’t provide syntax for this, you would have to create a list from these two outputs first. But @oesteban is right, a code might be useful to understand what you’re trying to do.

4

Hi @oesteban and @djarecka,

Sorry for not having been specific, unfortunately I don’t have the code with me at the moment. But I can think of a similar problem. Let’s say I’d like to run FAST and then convert some of the output files from FAST into nifti (I know I can just use output_type="NIFTI", it’s just so that I can illustrate my problem).
So I would do something like this:

fast = Node(FAST(in_files="structural.nii"), node="fast")
convert = MapNode(Gunzip(),name="convert",iterfield="in_file")
wf.connect(fast, convert, [(("probability_maps","partial_volume_map"),"in_file")])

So basically I would like to gunzip all probability_map files (which are a list of files) and partial_volume_map (which is a single file), and would like to use convert for that. However the above code would throw a “IndexError: tuple index out of range” exception.

Doing:

wf.connect(fast, convert, [("probability_maps","in_file")])
wf.connect(fast, convert, [("partial_volume_map","in_file")])

won’t work either because convert is already connected.

I hope that’s clearer.

5

Hi @mri - so as you discovered, connect unfortunately doesn’t support it. I think you can either create an extra node to concatenate your outputs from fast (i.e. write a simple python function and use Function_Interface) or have two convert nodes.

6

Hi @djarecka, thanks for your help. I tried to write a function to do this but am really stuck. I simply don’t get how to connect the fast node (which should output probability maps and partial_volume_map) to the function node. It’s my original problem all over again… Could you help me please?

fast = Node(FAST(in_files="structural.nii"), node="fast")
convert = MapNode(Gunzip(),name="convert",iterfield="in_file")

# fun creates a single list of files
def fun(in_files):    
    try: 
        iter(in_files)
        return [y for x in in_files for y in fun(x)]
    except:
        return [in_files]

combinefun = Node(Function(input_files=["in_files"], output_files=["out_files"], function=fun), name="combinefun")
wf.connect(fast, combine, [(("probability_maps","partial_volume_map"),"in_files")])
wf.connect(combine, convert, [("out_files","in_file")])
7

Hi @mri, I was thinking about function that can concatenate two outputs from fast. Taking your example and assuming that probability_maps is a list of files and partial_volume_map is a single file, the function can look similar to this:

def concatenate_files(files_list, file):
    files_list.append(file)
    return files_list

concatenate = Node(Function(input_names=["files_list", "file"], output_names=["all_files"], function=concatenate_files), name="concat_files")

# connecting two outputs from fast to two inputs from concatenate and one output from concatenate to convert
wf.connect([(fast, concatenate, [("probability_maps", "file_list"), ("partial_volume_map", "file")]),
            (concatenate, convert, [("all_files", "in_file")])
            ])

Let me know if this works for you.

1 Like
8

Hi @djarecka, thank you very much for your help, it works perfectly now! I didn’t know we could specify more than one parameter inside connect, it’s a very useful feature. Can I ask you just one more question please?
Using the same example, let’s say probability_maps returns ["f1","f2","f3","f4","f5"] and partial_volume_map returns'f6'. What if later I would like to retrieve'f3' to do some additional analysis (eg masking). I thought of creating a general helper function to extract the file number I want, the problem is that I do not know how to specify n when I connect the nodes.

def concatenate_files(files_list, file):
    files_list.append(file)
    return files_list

def helper(file,n):
    return file[n]
    
concatenate = Node(Function(input_names=["files_list", "file"], output_names=["all_files"], function=concatenate_files), name="concat_files")

masker = Node(ApplyMask(mask_file="mask.nii"), name=masker)

wf.connect([(fast, concatenate, [("probability_maps", "file_list"), ("partial_volume_map", "file")]),
            (concatenate, convert, [("all_files", "in_file")]),
            **(convert, masker, [( ("all_files", helper), "in_file")] #how can I add the argument n in the helper function?**
 	     ])
9

I’m afraid that you can’t connect straight to function, you have to create an interface using Function and put into Node (in a similar way as for concatenate)