Is there any way to tell nipype to try the job on particular node more than one time? I work on a cluster and sometimes things fail for no apparent reason and run OK on a second try. I know that it is possible to set max_tries for PBS plugin but it only tells to resubmit the job when submitting failed. I want to do the same but in a more broad way, like whatever the reason for failing, try again N times.
currently this functionality is not there, but wouldn’t be too hard to add. essentially a modification of clean_queue. one would need to figure out a bit about the underlying cluster system (slurm, pbs, etc.,.) and this option may not work for graph plugins.
this would be a good feature to add. could you please submit an issue on nipype? a potential PR would be much welcome as well. we will try ti get this in before the end of june release.
OK, I’ve submitted an issue. One more question: is it possible to make node.plugin_args dependent on some of the iterables? I.e., I do decoding for different numbers of voxels and it’d be great if I can ask for cluster resources depending on that number.