It makes all sense that it wouldn’t run at all before 1.0.0-rc12 with 16MB of RAM. We made great progress in that rc understanding how memory was being used. We are dealing with the issue and it is one of the lines in the roadmap with the highest priority.
With 4 CPUs, 16GB of RAM I would run fmriprep with the following options:
--nthreads 2 --omp-nthreads 4 --mem-mb 16000.
To talk about performance, we first need to clarify that executing FreeSurfer’s
recon-all will add 6-12 h. to your processing. In a machine like the one you describe (4CPUS) probably more. So that would account for most of your 20 hours, unless you were using the
That said, with FreeSurfer pre-run, I can tell you (based on my experience) that a dataset like the one you describe would take:
- About 2h 30min in an 8CPU 32GB RAM desktop.
- About 1h ~ 1h 30min in a 16 CPU, 64 GB RAM compute node of our cluster.
We just released FMRIPREP 1.0, and we are making a lot of progress about memory consumption. For example, before buying more memory, make sure that you can overcommit a lot of memory. This means that your
/proc/sys/vm/overcommit_memory is set to
0 and/or that you have a large value in
/proc/sys/vm/overcommit_ratio. If your system runs with cgroups, again make sure that you have large limits of virtual memory. Right now, the main bottleneck for FMRIPREP is the virtual memory. In terms of physical memory, I haven’t seen many datasets over 12 GB RSS, and those are datasets with a lot more and a lot lengthier runs compared to your dataset.
We also enabled a new
--resource-monitor flag for FMRIPREP. You could use it to profile memory consumption in your settings. If you wanted to try that, we’ll be very happy to assist you and make good use of your feedback.
Thanks very much and happy fmriprep’ing,