Heudiconv completes but also times out

, ,
1

Hello!

I ran heudiconv on a set of dicoms that I have to convert to BIDS. In my output directory, I seem to get all the outputs and files from a cursory visual inspection, which is nice!

I ran heudiconv using SLURM on an HPC, and use a job array to spin off jobs for each participantXsession. Some but not all of my jobs timed out (I only requested a compute node for 12 hours). However, when I check the output folder it seems like everything is there.

I can maybe copy my .out and .err files (not sure if I can, need to check) but I am curious if:

  1. Anyone has seen this before? Should heudiconv take 12 hours to run on one session from one participant (there are multiple imaging files/tasks but still).
  2. How do I determine if it ran to completion on the nodes that hung since I see all the outputs I think I would expect (need to confirm)

Happy to provide more details (singularity command, etc.) if needed.

Thanks!

Rahul

2

I would first make sure that you are using recent release of heudiconv (which one?)

  1. search for hanging in heudiconv github points to NiPype convert node hanging · Issue #584 · nipy/heudiconv · GitHub . please check details to see if matches your case
  2. that it exits with 0 exit code.

in case of debugging – check .out .err for details indeed to see where it gets stuck. Might want to use py-spy to see the traceback of “hanging” python process – might reveal what is it waiting on if not clear from logging .

3

Thanks everyone. The solution for me was moving my data off a directory with slow read/writes on my HPC to a scratch directory that was a lot faster.