MRIqc does not skip existing data

1

Hi @oesteban and others,

I am running MRIqc on a dataset with 440 subjects, 1900 sessions. For the run I am using a singularity container in datalad (mriqc ver 0.15.1), and the input is the entire raw bids dataset.

The problem is that after running for 2-3 days (and producing mostly bold statistics), the RAM consumption grew slowly but steadily, until it reached the limits on my server, about 250Gb. It then crashed, although the error in the terminal was not exaclty memory related. I decided to restart it. hoping it will skip existing outputs. I also read one of your replies that, if the work directory is the same and untouched, it should skipped past processes. And I checked the work directory, which is of ~87Gb in size. But looks like the same QCs are being calculated again (I can see the timestamps of the files) and console output shows again the same subject being processed on bold images. For some strange reason, the process goes in reverse alfanumeric order, the last subject in BIDS is processed first.

Question therefore is how to make mriqc skip existing outputs and focus on what is missing. If you know anything about why memory consumption increases, please let me know. It went up again from few hours ago, from 6Gb to 36Gb, and at this rate can continue for a couple of days until it hits the limit.

This is my datalad call, if it helps:

datalad containers-run  --explicit \
        -n containers/bids-mriqc \
        --input 'bids/' \
        --output './' -- \
        '{inputs}' '{outputs}' \
        --no-sub \
        --nprocs 20 \
        --mem_gb 250 \
        --ants-nthreads 15 \
        --verbose-reports \
        participant \
        | tee ./MRIQC2.log

For my first run I had 30 nprocs, no explicit memory limit, and no ants nthread definition. Changed those to hopefully be more effective.

2

Just to give an update, the current RAM consumption is 122Gb for the python process. Looking forward if someone has any tip on how to skip existing QCs and what might be the issue with the increased memory consumption.

3

I ran into a similar issue on my dataset (1400 subjects, 2500 sessions), and the way I solved it was to run MRIqc on each subject individually, then run the group analysis after, using the subject outputs. With these outputs, the group analysis was extremely fast.

Unfortunately I’m using docker and don’t have a lot of familiarity with datalad, so I’m not 100% sure if that solution would work for you.

4

Thanks @cbin-cnl ! It helps to know I am not the only one experiencing this. My second run has now crashed too, and I was thinking to go for individual runs. But looks like mriqc containers have some bugs worth noting: (1) with memory handling and (2) with not recognizing outputs to be skipped.