Pass function definition into node

Gregory_Ciccarelli · December 4, 2016, 4:12pm

def fve(stuff):
    return value

def foo(params, fve_func):
    #fve_func(stuff)
    return value2

def node1(params, learning_alg, fve_func):
    learning_alg(params, fve_func)

node1.inputs.fve_func = fve
node1.inputs.learning_alg = foo

Runs fine locally, but crashes when using a slurm plugin with error:
AttributeError: ‘module’ object has no attribute ‘fve’

Originally, I had the input and variable name also as fve, but saw a SO post saying to name differently. Didn’t help. Also, I don’t even call the fve_func in foo (I commented out that line to debug as in my above example), so somehow it’s the passing of the function that is causing problems.

satra · December 5, 2016, 1:56am

the function definitions should be inside the function or imported into the function:

Option 1:

define the functions internally and send functions by name:

def outer_func(**outer_params):
    def inner_func1(params):
        pass
    def inner_func2(params):
        pass
    if outer_param['func'] == 'func1':
        inner_func1()
    else:
        inner_func2()

node1.inputs.func = 'func1'

Option 2:

define the functions in a file:

def outer_func(**outer_params):
    import sys
    sys.path.append('/path/to/func_file')
    if outer_param['func'] == 'func1':
        from func_file import inner_func1
        inner_func1()
    else:
        from func_file import inner_func2
        inner_func2()

Option 3:

convert function to string and then unconvert it inside the function:

github.com

nipy/nipype/blob/master/nipype/utils/misc.py#L69


# Determine minimum indentation (first line doesn't count):
indent = sys.maxsize
for line in lines[1:]:
    stripped = line.lstrip()
    if stripped:
        indent = min(indent, len(line) - len(stripped))
# Remove indentation (first line is special):
trimmed = [lines[0].strip()]
if indent < sys.maxsize:
    for line in lines[1:]:
        # replace existing REST marker with doc level marker
        stripped = line.lstrip().strip().rstrip()
        if marker is not None and stripped and \
           all([s == stripped[0] for s in stripped]) and \
           stripped[0] not in [':']:
            line = line.replace(stripped[0], marker)
        trimmed.append(line[indent:].rstrip())
# Strip off trailing and leading blank lines:
while trimmed and not trimmed[-1]:
    trimmed.pop()
while trimmed and not trimmed[0]:

github.com

nipy/nipype/blob/master/nipype/utils/misc.py#L75


# Remove indentation (first line is special):
trimmed = [lines[0].strip()]
if indent < sys.maxsize:
    for line in lines[1:]:
        # replace existing REST marker with doc level marker
        stripped = line.lstrip().strip().rstrip()
        if marker is not None and stripped and \
           all([s == stripped[0] for s in stripped]) and \
           stripped[0] not in [':']:
            line = line.replace(stripped[0], marker)
        trimmed.append(line[indent:].rstrip())
# Strip off trailing and leading blank lines:
while trimmed and not trimmed[-1]:
    trimmed.pop()
while trimmed and not trimmed[0]:
    trimmed.pop(0)
# Return a single string:
return '\n'.join(trimmed)




def find_indices(condition):

Gregory_Ciccarelli · December 6, 2016, 2:16pm

Option 3 spelled out:

from nipype.utils.misc import getsource as f2s

def fve(stuff):
return value

Make a string form of the function
node1.inputs.fve = f2s(fve)

Inside node1, or its called functions, turn string back into function
from nipype.utils.misc import create_function_from_source as s2f
fve = s2f(fve)

Beware of python’s pass by reference default, which means that using the same name for the string form and function form of “fve” could lead to s2f(fve) or f2s(fve) operating on the incorrect form.

Also note: f2s() does not work on class definitions (so it seems). Class definitions can be defined in a separate file, added to path as in option 2, and then imported inside the function.

Option 4:
If all your function and class defs are in a jupyter notebook, and the notebook is being executed, then nothing needs to be turned into a string or imported because the defs exist within the scope of the whole notebook. This is true even if slurm is being used to launch jobs from within the notebook provided the notebook itself is being run headerless. If the notebook is being executed interactively, and some jobs are being sent to slurm, then all the scope problems occur that prompted this question.

BEWARE: A breaking change in the latest version of nipype (pip install --upgrade https://github.com/nipy/nipype/archive/master.zip ### 2252579cf1828f5b9422f2cdc6563c5ee0550e07)

Print statements must be written in functional form within functions that are turned into strings aka print statements must follow python 3.x rules e.g. print('hello world') NOT python 2.7 print 'hello world'