Niworkflows: appropriate for non-fmriprep developers/pipelines?

1

Hey,

I guess this is for @effigies and @ChrisGorgolewski

I am very curious about the ‘niworkflows’-project. This seems to have some very nice features on top of nipype. For example, you can make ‘hooks’ that draw nice HTML reports for a given node. This is something that would be very useful in my own project, where I am setting up a workflow for laminar-resolved fMRI at UHF (fmriprep doesn’t cut it there for me).

However, I also notice niworkflows contains a new branch of the Nipype engine that it is not backwards-compatible. For example, as I understand it is not possible to ‘manually’ set up nodes inputs during runtime with this version of the engine. Furthermore, some classic nodes are rewritten?

What is the plan for this package? Is niworkflows supposed to be something that end-users/people that build their own pipelines can use at some point? Or is it really just the engine for the fmriprep/Poldrack-people? Are there any plans to port some of the features (especially the report feature) to Nipype at some point?

Cheers!
Gilles

2

tl;dr: I think I’ve addressed all of your questions below, though perhaps not satisfactorily or in order. Basically, you’re welcome to use niworkflows (caveat emptor), but what would probably be most valuable for everybody is figuring out how to upstream the useful bits.

Original post

So niworkflows is a couple things, including a staging ground for interfaces and small workflows that we might want to merge into nipype one day. But most importantly, it’s what we need to make fmriprep and mriqc work, which includes updates to nipype on a much shorter time scale than nipype’s release cycle. We certainly intend everything in it to be open and reusable, but it should be understood to depend on features that are not in a released version of nipype, and we’ve intentionally sacrificed usability of niworkflows as a library for the smooth installation of fmriprep and mriqc.

As you’ve noticed, it is not interoperable with a separate installation of nipype. (It shouldn’t interfere – this was another reason for the choicese we made – but nipype and niworkflows.nipype are just not going to work together.) This could be eventually resolved by moving to a duck-typing approach to nipype; instead of checking whether objects are an instance of a class, we could just require that they implement an interface. But I digress, since that’s not going to help you right now.

The easiest way to use niworkflows with existing pipelines is to simply replace nipype with niworkflows.nipype everywhere you see it in your existing code. There’s the possibility of breakage, since the most recent niworkflows release includes nipype master as of 16 days ago. And of course, now your code is dependent on niworkflows.nipype, etc, etc.

What would probably be the most useful way forward is to consider how we can integrate the features you want back upstream into niworkflows proper. Our visualization interfaces could plausibly become part of nipype, but we’d need input from the wider community to make them less specific to our use cases. And any other interfaces that you particularly find valuable, we can prioritize moving those over (and any help you have time to give beyond simply identifying those interfaces would be invaluable).

The nodes you see “rewritten” are (for the most part) drop-in replacements for the originals that include reporting automatically. e.g. BBRegisterRPT is BBRegister, but it also generates an SVG showing the quality of registration, using the inputs and outputs of the interface.

And because I like to maintain links to an ongoing discussion so that if I miss something people can read the whole history:

(To be clear, I’m not saying you should have dug this up. I just want to make it easier for me to dig it all up next time it comes up, because this has bitten people a number of times.)

3

Cool. This is pretty clear.

I think especially the registration visualization can be very useful in a very wide range of applications. I would also be very happy to help integrate the features into Nipype, but I’m afraid I’m missing some software engineering skills here (:duck:-typing?! :stuck_out_tongue: ).

4

Duck-typing is “If it quacks like a duck, it’s a duck” as a type-system philosophy. Instead of checking whether an interface is a list, just try to get an element from it. If you can, it’s enough of a list for our purposes; if not, then somebody gave you the wrong object. Nipype currently checks whether an object is an nipype.interfaces.BaseInterface object, instead of checking whether we can treat it like an interface, which is one big reason we can’t use recent niworkflows interfaces in a vanilla nipype pipeline.

Even if we fixed that, it wouldn’t be until there’s a new release of nipype we can depend on that we could be meaningfully interoperable, so in the short term getting niworkflows interfaces into nipype is going to be the much more fruitful approach.

@satra or @mgxd have you had a look at our visualization tools, and have a sense of how much would need to be changed to merge into nipype? If not, we can just start with a PR and see what people think needs changing.

5

Ah! So you mean taking those "SimpleInterface"s from niworkflows and put them in the vanilla BaseInterface suit?

I actually already did a little bit of that with some nodes I found very useful.

For the visualization tools, this would amount to making niworkflows-visualization “Nodes” (something like nipype.interfaces.niworkflows.visualization.check_registration) that are conceptually the same as a fsl.MCFlirt or a ants.Registration-Node?

That I could do.

6

Ah, no. I mean that

>>> isinstance(nipype.interfaces.base.BaseInterface(),
               nipype.interfaces.base.BaseInterface)
True
>>> isinstance(niworkflows.nipype.interfaces.base.BaseInterface(),
               nipype.interfaces.base.BaseInterface)
False

There are a number of places where nipype uses isinstance checks that are going to fail across nipype and niworkflows.nipype (as a representative but incomplete list):

That said, I have submitted a PR to get SimpleInterface or something like it into nipype, which would make bringing SimpleInterface-derived interfaces easier to upstream. But feel free to go ahead and submit CopyHeader. I think there are some more features/checks we’ll want to add before it goes in (including more complete docs), but that seems like a good place to start.