Can't pickle instancemethod when using MapNode in Nipype

nipype
#1

I cannot get MapNode commands within a Workflow to execute properly (e.g. with BET nodes of MCFLIRT nodes). It always returns TypeError: can't pickle instancemethod objects. To diagnosis the problem, I’ve tried restructuring my workflow to avoid using MapNode and the code always returns that error at the step where I kept MapNode.

Is there a workaround or some incompatibility that is leading to this?

My system (running on macOS mojave) info:

{'commit_hash': '08d70920c',
 'commit_source': 'installation',
 'networkx_version': '2.2',
 'nibabel_version': '2.4.0',
 'nipype_version': '1.1.9',
 'numpy_version': '1.16.2',
 'pkg_path': '/anaconda2/lib/python2.7/site-packages/nipype',
 'scipy_version': '1.2.1',
 'sys_executable': '/anaconda2/bin/python',
 'sys_platform': 'darwin',
 'sys_version': '2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 13:10:39) \n[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]',
 'traits_version': '5.0.0'}
0 Likes

#2

@Amyunimus

Can you try the following minimal example and post the full trackback if you are still running into an issue?

from nipype import MapNode, Workflow
from nipype.interfaces.fsl import BET

better = MapNode(BET(), iterfield=['in_file'], name='better')
better.inputs.in_file = ['/path/to/some/image'] * 2
wf = Workflow('testwf')
wf.add_nodes([better])
wf.run()
0 Likes

#3

Yes, same error. Here’s traceback:

190409-10:22:06,554 nipype.workflow INFO:
	 Workflow testwf settings: ['check', 'execution', 'logging', 'monitoring']
190409-10:22:06,574 nipype.workflow INFO:
	 Running serially.
190409-10:22:06,578 nipype.workflow INFO:
	 [Node] Setting-up "testwf.better" in "/private/var/folders/9k/3m1f_r5s21gfhb3nh5lv5bh00000gn/T/tmpBZq_0o/testwf/better".
190409-10:22:06,639 nipype.workflow ERROR:
	 Node better failed to run on host Emilys-MacBook-Pro-2.local.
190409-10:22:06,647 nipype.workflow ERROR:
	 Saving crash info to /Users/ejward/Documents/fMRI/testing/crash-20190409-102206-ejward-better-75789d63-87d0-463e-810f-b558964b05c4.pklz
Traceback (most recent call last):
  File "/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 48, in run
    node.run(updatehash=updatehash)
  File "/anaconda2/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 469, in run
    savepkl(op.join(outdir, '_node.pklz'), self)
  File "/anaconda2/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 731, in savepkl
    pickle.dump(record, pkl_file)
  File "/anaconda2/lib/python2.7/pickle.py", line 1376, in dump
    Pickler(file, protocol).dump(obj)
  File "/anaconda2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 568, in save_tuple
    save(element)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 568, in save_tuple
    save(element)
  File "/anaconda2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/anaconda2/lib/python2.7/pickle.py", line 425, in save_reduce
    save(state)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/anaconda2/lib/python2.7/pickle.py", line 669, in _batch_setitems
    save(v)
  File "/anaconda2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/anaconda2/lib/python2.7/pickle.py", line 606, in save_list
    self._batch_appends(iter(obj))
  File "/anaconda2/lib/python2.7/pickle.py", line 621, in _batch_appends
    save(x)
  File "/anaconda2/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/anaconda2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-133-fe5eb1f80a1f> in <module>()
      6 wf = Workflow('testwf')
      7 wf.add_nodes([better])
----> 8 wf.run()

/anaconda2/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.pyc in run(self, plugin, plugin_args, updatehash)
    597         if str2bool(self.config['execution']['create_report']):
    598             self._write_report_info(self.base_dir, self.name, execgraph)
--> 599         runner.run(execgraph, updatehash=updatehash, config=self.config)
    600         datestr = datetime.utcnow().strftime('%Y%m%dT%H%M%S')
    601         if str2bool(self.config['execution']['write_provenance']):

/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/linear.pyc in run(self, graph, config, updatehash)
     51                 # bare except, but i really don't know where a
     52                 # node might fail
---> 53                 crashfile = report_crash(node)
     54                 if str2bool(config['execution']['stop_on_first_crash']):
     55                     raise

/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/tools.pyc in report_crash(node, traceback, hostname)
     60     else:
     61         savepkl(crashfile, dict(node=node, traceback=traceback),
---> 62                 versioning=True)
     63     return crashfile
     64 

/anaconda2/lib/python2.7/site-packages/nipype/utils/filemanip.pyc in savepkl(filename, record, versioning)
    729         pkl_file.write('\n'.encode('utf-8'))
    730 
--> 731     pickle.dump(record, pkl_file)
    732     pkl_file.close()
    733 

/anaconda2/lib/python2.7/pickle.pyc in dump(obj, file, protocol)
   1374 
   1375 def dump(obj, file, protocol=None):
-> 1376     Pickler(file, protocol).dump(obj)
   1377 
   1378 def dumps(obj, protocol=None):

/anaconda2/lib/python2.7/pickle.pyc in dump(self, obj)
    222         if self.proto >= 2:
    223             self.write(PROTO + chr(self.proto))
--> 224         self.save(obj)
    225         self.write(STOP)
    226 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_tuple(self, obj)
    566         write(MARK)
    567         for element in obj:
--> 568             save(element)
    569 
    570         if id(obj) in memo:

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_tuple(self, obj)
    566         write(MARK)
    567         for element in obj:
--> 568             save(element)
    569 
    570         if id(obj) in memo:

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    329 
    330         # Save the reduce() output and finally memoize the object
--> 331         self.save_reduce(obj=obj, *rv)
    332 
    333     def persistent_id(self, obj):

/anaconda2/lib/python2.7/pickle.pyc in save_reduce(self, func, args, state, listitems, dictitems, obj)
    423 
    424         if state is not None:
--> 425             save(state)
    426             write(BUILD)
    427 

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_dict(self, obj)
    653 
    654         self.memoize(obj)
--> 655         self._batch_setitems(obj.iteritems())
    656 
    657     dispatch[DictionaryType] = save_dict

/anaconda2/lib/python2.7/pickle.pyc in _batch_setitems(self, items)
    667             for k, v in items:
    668                 save(k)
--> 669                 save(v)
    670                 write(SETITEM)
    671             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    284         f = self.dispatch.get(t)
    285         if f:
--> 286             f(self, obj) # Call unbound method with explicit self
    287             return
    288 

/anaconda2/lib/python2.7/pickle.pyc in save_list(self, obj)
    604 
    605         self.memoize(obj)
--> 606         self._batch_appends(iter(obj))
    607 
    608     dispatch[ListType] = save_list

/anaconda2/lib/python2.7/pickle.pyc in _batch_appends(self, items)
    619         if not self.bin:
    620             for x in items:
--> 621                 save(x)
    622                 write(APPEND)
    623             return

/anaconda2/lib/python2.7/pickle.pyc in save(self, obj)
    304             reduce = getattr(obj, "__reduce_ex__", None)
    305             if reduce:
--> 306                 rv = reduce(self.proto)
    307             else:
    308                 reduce = getattr(obj, "__reduce__", None)

/anaconda2/lib/python2.7/copy_reg.pyc in _reduce_ex(self, proto)
     68     else:
     69         if base is self.__class__:
---> 70             raise TypeError, "can't pickle %s objects" % base.__name__
     71         state = base(self)
     72     args = (self.__class__, base, state)

TypeError: can't pickle instancemethod objects
0 Likes

#4

@Amyunimus thanks for the report - I was also able to reproduce this. It looks like there is some incompatibility currently with FSL interfaces (and possibly others) with py2 and traits 5.0

While we address this in nipype, you should be able to get around this by downgrading traits to 4.6.0

pip install -U traits==4.6

related issue: https://github.com/nipy/nipype/issues/2913

0 Likes

#5

@mgxd - let’s open an issue for this. i’m surprised our tests have not caught this. but an old bug/PR was merged into the new release: https://github.com/enthought/traits/pull/373

it’s possible all we need to do is get rid of this: https://github.com/nipy/nipype/blob/master/nipype/interfaces/base/specs.py#L336

0 Likes

#6

@Amyunimus and @mgxd - traits reverted that pull request and made a new release. can you upgrade to 5.1 and try? See Discussion here

0 Likes

#7

yes, can confirm traits 5.1 fixes the issue.

0 Likes