Heudiconv_filelock_timeout

Hello,
I am running heudiconv (0.11.6) with singularity on a HPC cluster and I am getting the following warning:

WARNING: Existing lockfile found in /base/nifti/heudiconv.lock - waiting for the lock to be released. To set a timeout limit, set the HEUDICONV_FILELOCK_TIMEOUT environmental variable to a value in seconds. If this process hangs, it may require a manual deletion of the /base/nifti/heudiconv.lock.

It looks like after deleting /base/nifti/heudiconv.lock., it keeps generating a new one.

What would be a ‘good’ value for HEUDICONV_FILELOCK_TIMEOUT ?

Thank you,
Fab

Do you run multiple heudiconv in parallel? I would assume “yes” and that is why you get this warning. It is ok to not set it to any value – it would just wait without a limit. Do you see any heudiconv process getting suck/hanging?

Yes, I am running multiple heudiconv in parallel. And yes, sever heudiconv processes are hanging. Any idea about what could cause it?
And, there is no need to delete /base/nifti/heudiconv.lock. ?

Thank you!
Fab

the fact that you’re running multiple heudiconv in parallel.

I would assume no – processes should wait until the other process frees access to it. But if there was interference from the user/batch system (lock file removal, process killing) - may be the locking somehow lead to stalling. Did you remove it manually? (I guess that might have confused it somehow).

I thought that may be there could be some issue with the filelock and NFS (I assume you use) – Reliable over NFS? · Issue #73 · tox-dev/py-filelock · GitHub suggests that it should be all ok if you are on some modern system. but it could be not working on correctly with linux < 2.6.5 but you don’t have it that old, do you?

FWIW, in the future you could also run it with --bids notop in parallel and then eventually have a single run with just --bids to produce/update those top level files. This way there would be no competition etc.

Even if I am running a job for each participant? (The participant jobs are parallelized)

Yes I tried to remove it, and it looks like with no effects.

It’s a CentOS Linux 7 (Core).
Is there documentation about how to use the --bids notop ?

Thank you!

yes… after each participant conversion heudiconv adjusts top level files, and that is where collision happens.

not really beyond “just add it to your invocation” :wink: and

❯ heudiconv --help | grep -3 notop
                        None.
  -b [BIDSOPTION1 [BIDSOPTION2 ...]], --bids [BIDSOPTION1 [BIDSOPTION2 ...]]
                        Flag for output into BIDS structure. Can also take
                        BIDS-specific options, e.g., --bids notop. The only
                        currently supported options is "notop", which skips
                        creation of top-level BIDS files. This is useful when
                        running in batch mode to prevent possible race
                        conditions.

well it is not saying it directly… uname -a could be more informative. Or even better, if you have datalad installed, then output of datalad wtf -S system

Hello,
I don’t have datalad installed. With uname I’m getting the following:
Linux login03 4.18.0-240.el8.x86_64 #1 SMP Fri Sep 25 19:48:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I am not really sure how to get the participants.tsv file after using the --bids notop option. Do you have an example by chance?
Thanks!

good, so you have an old but not ancient kernel :wink:

re participants.tsv – dang, it seems it is a “documented shortcoming”. From Usage — heudiconv 0.11.6.post3+g5db2ede documentation :

If using bids, the notop bids option suppresses creation of top-level files in the bids directory (e.g., dataset_description.json ) to avoid possible race conditions. These files may be generated later with populate_templates.sh below (except for participants.tsv , which must be created manually).

So, I am afraid, you are “doomed” to create that file manually now. Since after the conversion we have no guaranteed access to original dicoms, and participant age/sex metadata was not extracted – we can’t really populate it reliably. And populating basic one is as easy as smth like

for s in sub-*; do grep -q -e "^$s\s" participants.tsv || echo -e "$s\tn/a\tn/a" >> participants.tsv; done