Very long run time on Mac (fmriprep-docker)

[tiborst]

Summary of what happened:

Running fmriprep-docker on a Mac Studio (M1 Max: 10 Core CPU and 32GB RAM; macOS Ventura 13.1) for a single subject takes a very long time.

E.g.:
Subject with 1 T1 weighted structural image, 1 EPI (>820 volumes), 1 phasedifference fieldmap (acquired with a Siemens scanner) takes ~ 16h without freesurfer surface reconstruction and ~ 3-4 days with freesurfer reconstruction.
Subject with 1 T1 weighted strucutral image, 8 EPIs (>190 volumes each), 1 phasedifference fieldmap (acquired with a Siemens scanner) takes ~ 26h without freesurfer surface reconstruction.

This is nowhere close to the expected run time mentioned in the docs (FAQ, Tips, and Tricks — fmriprep version documentation).
We have > 150 Subjects to be preprocessed and at this rate it would take several months to complete (currently we do not have access to a HPC). Therefore, I am wondering whether there are any options that I am missing that might help accelerate fmriprep?

Thank you very much for any help in advance!

Best,
Tibor

Command used (and if a helper script was used, a link to the helper script or the command generated):

    participant_label=participant01
    max_mem_mb=24000
    max_cores_cpu=8
    freesurfer_file=~/freesurfer_license.txt

    sourcepath=/path_to/sourcedata
    preprocpath=/path_to/derivatives/preproc
    workdirpath=/path_to/derivatives/workdir

    fmriprep-docker ${sourcepath} ${preprocpath} \
        participant \
        --participant-label ${participant_label} \
        --fs-license-file ${freesurfer_file} \
        --nthreads ${max_cores_cpu} \
        --stop-on-first-crash \
        --mem-mb ${max_mem_mb} \
        -v \
        -w ${workdirpath} \
        --dummy-scans 3 \
        --fs-no-reconall

Version:

Tried on both v20.2.7 and v23.1.4

Environment (Docker, Singularity, custom installation):

Docker v4.21.1 with 8 CPU cores and 27 GB RAM assigned.

Data formatted according to a validatable standard? Please provide the output of the validator:

Data is Bids valid.

[Steven]

Hi @tiborst,

You can try Brainlife.io to parallelize processing over the cloud.

Best,
Steven

[effigies]

@tiborst

M1

Running fmriprep on M1 chip Mac Mini (using Docker Wrapper)

This just isn’t going to work without someone contributing a Docker image and build process for the M1 architecture. You may have some success installing fMRIPrep in a conda environment instead of Docker:

$ git clone https://github.com/nipreps/fmriprep.git
$ cd fmriprep
$ git checkout $VERSION
$ conda env create -f env.yml
$ conda activate fmriprep
$ pip install -r requirements.txt .

If you inspect the Dockerfile, you should be able to see all of the other tools that need installing, such as AFNI, FreeSurfer, Connectome Workbench, etc. Note that any of these tools might also not be compiled for the M1/M2 architecture, so I can’t guarantee any performance.

We also can only give limited support for “bare-metal” installations, so taking this route is accepting some responsibility for figuring things out. I definitely cannot help with reproducing bugs on a Mac.

[stebo85]

Dear @tiborst,

I am currently working on brining our www.Neurodesk.org project to Apple Silicon and one thing I found with respect to performance is that activating Rosetta for x86 images brings a 5 fold speedup in my tests and I haven’t seen a downside yet:

You find this under “Features in development”:

I hope that helps
Kind regards
Steffen

[ilaria]

@stebo85

thanks for posting this solution.
I’m running into the following warning and, my first attempt at running fMRIprep on my new M2 Mac mini with dockers, did not complete the preprocessing after over 30h.

The requested image’s platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I’ve flagged the Rosetta option as you suggested. When I run the code, I still get the warning. Is this your observation as well when implementing your solution? Just to understand if this is going to solve the 30h+ of processing of 1 participant.

Thanks,
Ilaria

[stebo85]

Dear @ilaria,

Yes, the warning will still appear, because you are still running an amd64 image on arm64, but now the processing will be faster because Rosetta 2 is more performant than the default qemu-based emulation in docker.

Kind regards
Steffen

[ilaria]

Hi,

Thanks for confirming! It is running faster if it weren’t for the errors that still pop up…
Ilaria

[stebo85]

@ilaria - what other errors are popping up? To avoid the requested image’s platform error there are two options:

• develop an arm64 image of fmriprep (which currently doesn’t exist, but there is progress on getting freesurfer to work natively in arm64 and the rest in that image could also be ported)
• run the current container on an x86 machine and not on apple silicon

Kind regards
Steffen

[ilaria]

Hello @stebo85:

I’ve opened another thread for this.

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending

Summary of what happened: Hello: I am having issues with running fMRIprep on my new Mac M2 pro, 16GB, ~2TB available. I have also added the Use Rosetta for x86/amd64 emulation on Apple Silicon option in Dockers as kindly recommended in another post of mine. Further, I learnt that where the data is written is important so the current set up has the raw data on the HD; while derivatives and working folder are on the actual computer (reading and writing to hard drives apparently slows down the p…

If you have any insight, please conisder adding them to that thread.
Thanks

Ilaria