Really struggling to get xcp_d running

Software Support
1

Summary of what happened:

Can anyone help me to get xcp_d running with docker? Specifically I just don’t get what each of these “-v” arguments is supposed to mean:

docker run --rm -it \
   -v /dset/derivatives/fmriprep:/fmriprep:ro \
   -v /tmp/wkdir:/work:rw \
   -v /dset/derivatives/xcp_d:/out:rw \
   -v /dset/derivatives/freesurfer:/freesurfer:ro \  # Necessary for fMRIPrep versions <22.0.2
   pennlinc/xcp_d:latest \
   /fmriprep /out participant \
   --file-format cifti --despike --head_radius 40 -w /work --smoothing 6

For example I don’t have any directory called “dset/derivatives”. My data is in BIDs format and was already preprocessed in fMRIPrep, and the results of fMRIprep is under “raw/output” but docker run is telling me that that folder doesn’t exist. Any help you can give would be greatly appreciated!

Command used (and if a helper script was used, a link to the helper script or the command generated):

PASTE CODE HERE

Version:

Environment (Docker, Singularity / Apptainer, custom installation):

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

PASTE LOG OUTPUT HERE

Screenshots / relevant information:

2

-v mounts a folder or file on your local filesystem into the Docker image. So /dset/derivatives/fmriprep would be the path to the fmriprep derivatives on your computer, while /fmriprep is what it’s going to be called within the Docker image. ro just means that the folder is mounted in read-only mode, while rw means that Docker can read from and write to the folder. Note that /fmriprep is the argument for the BIDS dataset (specifically the BIDS derivatives) that XCP-D will process. You can use whatever path you want as long as it doesn’t conflict with any of the paths within the Docker image.

If your fMRIPrep derivatives are in “raw/output”, then put the full path to that folder instead of /dset/derivatives/fmriprep. Don’t change the /fmriprep it’s mounting as.

By the way, --file-format cifti isn’t available in the latest release of XCP-D (it’ll be in the next release), so you’ll need to replace that with --cifti. I’d recommend checking the documentation for the latest XCP-D release (confusingly named “stable” on readthedocs): XCP-D : A Robust Postprocessing Pipeline of fMRI data — xcp_d 04.0 documentation

3

Hey, thanks for the response! This is really helpful! So in the line "-v /dset/derivatives/fmriprep:/fmriprep:ro " is it the case that the first half before the colon is where my fmriPrep output is and so I should replace “/dset/derivatives/fmriprep” with “/raw/output”, and then the part after the colon "/fmriprep:ro " is what it’s going to be called in the docker image. So this line should be “-v /raw/output:/fmriprep:ro”? Likewise with the line “-v /dset/derivatives/xcp_d:/out:rw”, should it be “-v /raw/output:/out:rw”?

4

Hi @Steve_Sizzou,

For both of those lines, I would do the full path to the outputs folders, so /raw/output/fmriprep (or whatever it’s called on your machine) and /raw/output/xcp_d.

Best,
Steven

1 Like
5

Thanks. just to make sure that i understood, I don’t have any folder called “/raw/output/fmriprep”, The results of fMRIprep are sitting in"/raw/output". Do i need to create that folder or does vcp_d create it? I’ve tried various combination of both with no luck. For example:

% docker run --rm -it  -v /raw/output/fmriprep:/fmriprep:ro  -v /tmp/wkdir:/work:rw  -v  /raw/output/xcp_d:/out:rw  -v /freesurfer:/freesurfer:ro \  pennlinc/xcp_d:latest  /fmriprep /out participant  –cifti --despike --head_radius 40 -w /work --smoothing 6

docker: invalid reference format.
See 'docker run --help'.

And also:

% docker run --rm -it   -v /raw/output:/fmriprep:ro  \
-v /tmp/wkdir:/work:rw  -v  /raw/output:/out:rw \
-v /freesurfer:/freesurfer:ro \
pennlinc/xcp_d:latest  /raw/output /out participant  –cifti --despike --head_radius 40 -w /work --smoothing 6


docker: invalid reference format.
See 'docker run --help'.
zsh: no such file or directory: pennlinc/xcp_d:latest

Do you have any idea how I can format this? As you can tell I’m still a little confused :sob: :joy:

6

Ok I tried to run a reduced version of the command after having learned a little more about the docker arguments. Now xcp is telling me that the paths in my directory where the data is and on docker are not shared. Here is what I ran:

(base) me@mes-MacBook-Air fMRIprep % docker run -it -v /ml_projects/fMRIprep/raw/output:/fmriprep:ro pennlinc/xcp_d:latest  ~/Users/me/ml_projects/fMRIprep/raw/output  ~/Users/me/ml_projects/fMRIprep/raw/out  participant

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
docker: Error response from daemon: Mounts denied:
The path /ml_projects/fMRIprep/raw/output is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/desktop/mac for more info.
(base) me@mes-MacBook-Air fMRIprep %

Any ideas?

7

Hi @Steve_Sizzou,

Since we don’t have access to your file system and know your file organization, our ability to help precisely is limited. But it looks like the folder you’re trying to mount doesn’t exist.

What do you get when you run ls /ml_projects/fMRIprep/raw/output ?

Whatever folder contains the fmriprep output subject folder and htmls is what is input to Xcp_d.

No, xcp presumes you have already run fmriprep and therefore will not create that.

I’ll also note that you might consider organizing your data more like other BIDS projects, with pipeline outputs going in the derivatives folder within the BIDS root directory.

Best,
Steven

8

So when I run it I get:

(base) me@mes-MacBook-Air ~ % ls ml_projects/fMRIprep/raw/output
dataset_description.json	desc-aseg_dseg.tsv		sourcedata			sub-01.html
desc-aparcaseg_dseg.tsv		logs				sub-01

The folder “raw” is where the raw data is which was also BIDs formatted. The output of fMRIprep is the “output” folder (so the raw and fmriprep preprocessed data aren’t separated). Maybe it’s the “sub-01” folder which is what contains the anat and func preprocessed data that I need to send to xcp? “sub-01” contains:

(base) me@mes-MacBook-Air ~ % ls ml_projects/fMRIprep/raw/output/sub-01
anat	figures	func	log

I was also trying to run a stripped down version of the command in order to debug it one line at a time and I’m finding that it is also not finding the freesurfer license.txt which is weird as it is at the specified location:

(base) me@mes-MacBook-Air ~ % docker run --rm -it -v ~/ml_projects/fmriprep/freesurfer:/freesurfer:ro pennlinc/xcp_d:latest . /out participant --fs-license-file license.txt

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
usage: xcp_d [-h] [--version] [--participant-label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-t TASK_ID] [--bids-filter-file FILE]
             [--bids-database-dir PATH] [-s] [--nprocs NPROCS] [--omp-nthreads OMP_NTHREADS] [--mem-gb MEMORY_GB] [--low-mem]
             [--use-plugin USE_PLUGIN] [-v] [--input-type {fmriprep,dcan,hcp,nibabies,ukb}] [--dummy-scans {{auto,INT}}] [--despike]
             [-p {27P,36P,24P,acompcor,aroma,acompcor_gsr,aroma_gsr,custom,none,gsr_only}] [-c CUSTOM_CONFOUNDS]
             [--smoothing SMOOTHING] [-m] [--motion-filter-type {lp,notch}] [--band-stop-min BPM] [--band-stop-max BPM]
             [--motion-filter-order MOTION_FILTER_ORDER] [-r HEAD_RADIUS] [-f FD_THRESH] [--min-time MIN_TIME]
             [--disable-bandpass-filter] [--lower-bpf HIGH_PASS] [--upper-bpf LOW_PASS] [--bpf-order BPF_ORDER]
             [--atlases {4S1056Parcels,4S156Parcels,4S256Parcels,4S356Parcels,4S456Parcels,4S556Parcels,4S656Parcels,4S756Parcels,4S856Parcels,4S956Parcels,Glasser,Gordon,HCP,Tian} [{4S1056Parcels,4S156Parcels,4S256Parcels,4S356Parcels,4S456Parcels,4S556Parcels,4S656Parcels,4S756Parcels,4S856Parcels,4S956Parcels,Glasser,Gordon,HCP,Tian} ...]
             | --skip-parcellation] [--min-coverage MIN_COVERAGE] [--exact-time EXACT_TIME [EXACT_TIME ...]]
             [--aggregate-session-reports AGGR_SES_REPORTS] [--random-seed _RANDOM_SEED] [-w WORK_DIR] [--clean-workdir]
             [--resource-monitor] [--config-file FILE] [--write-graph] [--stop-on-first-crash] [--notrack]
             [--debug {pdb,all} [{pdb,all} ...]] [--fs-license-file FILE] [--md-only-boilerplate] [--boilerplate-only] [--reports-only]
             [--warp-surfaces-native2std] [--skip-dcan-qc]
             fmri_dir output_dir {participant}
xcp_d: error: Path does not exist: <license.txt>.

Thanks in advance for your help!

9

Just to give more context in case that is helpful. to run frmiprep I did:

fmriprep-docker /Users/me/ml_projects/fmriprep/raw /Users/me/ml_projects/fmriprep/raw/output participant --skip-bids-validation --fs-license-file  /Users/me/ml_projects/fmriprep/license.txt --mem_mb 30000

And this was successful. The folder “raw” had the bids validated raw resting state data. and frmri prep created the “raw/output” folder. I just went back and ran this again as a sanity check and it still runs smoothly! :slight_smile:

10

Hi @Steve_Sizzou,

I mentioned in my last response, that the fmriprep output folder that contains the subject folders and html is what is passed into XCP input. For you that appears to be ml_projects/fMRIprep/raw/output.

You will need to mount the license with another -v statement.
in Docker preamble:

-v /Users/me/ml_projects/fmriprep/license.txt:/license.txt

and in xcp:

--fs-license-file /license.txt

Best,
Steven

11

Thanks! It still doesn’t seem to find the license, I tried:

(base) me@mes-MacBook-Air ~ % docker run --rm -it -v /Users/me/ml_projects/fmriprep/license.txt:/license.txt pennlinc/xcp_d:latest . /out participant --fs-license-file license.txt

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
usage: xcp_d [-h] [--version] [--participant-label PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]] [-t TASK_ID] [--bids-filter-file FILE]
             [--bids-database-dir PATH] [-s] [--nprocs NPROCS] [--omp-nthreads OMP_NTHREADS] [--mem-gb MEMORY_GB] [--low-mem] [--use-plugin USE_PLUGIN]
             [-v] [--input-type {fmriprep,dcan,hcp,nibabies,ukb}] [--dummy-scans {{auto,INT}}] [--despike]
             [-p {27P,36P,24P,acompcor,aroma,acompcor_gsr,aroma_gsr,custom,none,gsr_only}] [-c CUSTOM_CONFOUNDS] [--smoothing SMOOTHING] [-m]
             [--motion-filter-type {lp,notch}] [--band-stop-min BPM] [--band-stop-max BPM] [--motion-filter-order MOTION_FILTER_ORDER] [-r HEAD_RADIUS]
             [-f FD_THRESH] [--min-time MIN_TIME] [--disable-bandpass-filter] [--lower-bpf HIGH_PASS] [--upper-bpf LOW_PASS] [--bpf-order BPF_ORDER]
             [--atlases {4S1056Parcels,4S156Parcels,4S256Parcels,4S356Parcels,4S456Parcels,4S556Parcels,4S656Parcels,4S756Parcels,4S856Parcels,4S956Parcels,Glasser,Gordon,HCP,Tian} [{4S1056Parcels,4S156Parcels,4S256Parcels,4S356Parcels,4S456Parcels,4S556Parcels,4S656Parcels,4S756Parcels,4S856Parcels,4S956Parcels,Glasser,Gordon,HCP,Tian} ...]
             | --skip-parcellation] [--min-coverage MIN_COVERAGE] [--exact-time EXACT_TIME [EXACT_TIME ...]]
             [--aggregate-session-reports AGGR_SES_REPORTS] [--random-seed _RANDOM_SEED] [-w WORK_DIR] [--clean-workdir] [--resource-monitor]
             [--config-file FILE] [--write-graph] [--stop-on-first-crash] [--notrack] [--debug {pdb,all} [{pdb,all} ...]] [--fs-license-file FILE]
             [--md-only-boilerplate] [--boilerplate-only] [--reports-only] [--warp-surfaces-native2std] [--skip-dcan-qc]
             fmri_dir output_dir {participant}
xcp_d: error: Path does not exist: <license.txt>.
(base) me@mes-MacBook-Air ~ %

Honesty, I think my brain is not cut out to successfully do this :confused: I don’t suppose you could tell me what you think the whole syntax should be? Thanks in advance!

12

Hi @Steve_Sizzou; you forgot the leading slash before /license.txt 

mkdir -p /Users/me/ml_projects/fmriprep/raw/output/xcp_d/
docker run --rm -it \
-v /Users/me/ml_projects/fmriprep/license.txt:/license.txt \
-v /Users/me/ml_projects/fmriprep/raw/output:/fmriprep_input \
-v /Users/me/ml_projects/fmriprep/raw/output/xcp_d:/xcp_d_output \
pennlinc/xcp_d:0.7.5 \
/fmriprep_input \
/xcp_d_output \
participant \
--fs-license-file /license.txt

Best,
Steven

14

WOW!!! Thank you, it’s finally running! I actually thought that those last arguments (fmri_dir output_dir {participant}) are supposed to refer to folders in my local machine, not folders that get created within the xcp container. Will “fmriprep_input” and “xcp_d_output” get. written locally once it’s finished? Thanks again!

15

Hi @Steve_Sizzou,

Those folders are on your local machine already, but are renamed in the container with the -v statements.

Glad it’s running!

Best,
Steven

16

I absolutely owe you a beer!

1 Like
17

Hey, that was pretty quick too :slight_smile: