fMRIPrep on Docker Desktop on Windows 10 with NTFS network share

andrewcrow · October 11, 2021, 2:50pm

Hello,

We’ve been using fMRIPrep within Docker Desktop on Windows 10 for sometime now, and we have a NTFS network share to store our data. While other programs seem to work over the network share (e.g., FreeSurfer, SPM), fMRIPrep fails to work due to what I believe is a BIDS validation error based on the output I get on my machine.

Based fMRIPrep output, there seems to be a period that has been prepended to what I believe is the input data directory. Based on the output of tree, nothing to me seems amiss with respect to whether this test dataset is BIDS valid or not. Output of ‘docker inspect’ might have more clues. Conceptually, this issue seems similar to issue #1944 posted on GitHub.

I have output in the form of text files but can’t upload them due to the fact I’m a new user.

To get around this, we locally push all the data we intend to process with fMRIPrep where it works seamlessly, so it be wonderful if we could get help with this problem. We’ve also consulted local IT who’ve tried to tweak things to no avail (nor do they support Linux or Docker). Please let me know if there are questions or other information needed on my end. Looking forward to a solution!

Thanks,
Andrew

oesteban · October 11, 2021, 10:02pm

Hi Andrew, thanks much for moving the conversation over here.

We are going to need first to confirm the fMRIPrep version you are running. You can check it by executing the docker container with the --version argument.

Then, the output logs from the console will also give us some clues. As you correctly point out, this resembles issue #1944 so maybe trying a later version of fMRIPrep will do.

andrewcrow · October 12, 2021, 2:45pm

Hi Oscar,

I’ve tried this with versions 20.2.3 and 20.2.4, which I understand are currently supported. (Admittedly, I’ve tried with much older versions, as well.) Interestingly, I get slightly different errors with each.

I’m still unable to upload the logs I’ve obtained. How should I go about submitting those?

Thanks,
Andrew

oesteban · October 21, 2021, 8:47am

Hi Andrew, sorry for the slow response.

Normally, this platform should allow you to attach text files. There are two kinds of logs you will want to share with us:

The output of fMRIPrep (which you can typically gather by “piping” the output of your command into a text file). A copy of this should be also found somewhere under the work directory (I believe).
Crashfiles - these are small text files that fMRIPrep generates when a particular step of the workflow goes wrong. They are stored under the log/ folder of each of the subjects you are processing, at the output folder.

Cheers,
Oscar

andrewcrow · October 22, 2021, 2:45pm

I have the output of fMRIPrep itself, but not the crash logs since fMRIPrep fails almost instantaneously and doesn’t/can’t write anything to the specified output directory. Still can’t upload files, so pardon the length reply. I also have output of docker inspect if that would be helpful.

Output of tree (from WSL Ubuntu since I can’t do this in PowerShell for some reason?). Including this to demonstrate that this directory should be BIDS compliant, as it’s worked locally after all.

/y/tmp/docker_test/
├── README
├── code
│   └── fmriprep_batch.ps1
├── dataset_description.json
└── sub-R0230
    ├── anat
    │   ├── sub-R0230_T1w.json
    │   └── sub-R0230_T1w.nii
    └── func
        ├── sub-R0230_task-VerbGeneration_bold.json
        └── sub-R0230_task-VerbGeneration_bold.nii

4 directories, 7 files

Terminal output of fMRIPrep v20.2.3:

You are using fMRIPrep-20.2.3, and a newer version of fMRIPrep is available: 20.2.5.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
bids-validator@1.4.0

	e[31m1: [ERR] Quick validation failed - the general folder structure does not resemble a BIDS dataset. Have you chosen the right folder (with "sub-*/" subfolders)? Check for structural/naming issues and presence of at least one subject. (code: 61 - QUICK_VALIDATION_FAILED)e[39m
		.data

e[36m	Please visit https://neurostars.org/search?q=QUICK_VALIDATION_FAILED for existing conversations about this issue.e[39m



Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 17, in main
    parse_args()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/parser.py", line 724, in parse_args
    config.environment.exec_env, opts.bids_dir, opts.participant_label
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/utils/bids.py", line 155, in validate_input_dir
    subprocess.check_call(['bids-validator', bids_dir, '-c', temp.name])
  File "/usr/local/miniconda/lib/python3.7/subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bids-validator', PosixPath('/data'), '-c', '/tmp/tmp47b0vj5p']' returned non-zero exit status 1.
RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=20.10.8 -it -v C:\SOFTWARE\freesurfer\license.txt:/opt/freesurfer/license.txt:ro -v Y:\tmp\docker_test:/data:ro -v Y:\tmp\docker_test\sub-R0230:/out nipreps/fmriprep:20.2.3 /data /out participant --participant-label sub-R0230 --use-syn-sdc --stop-on-first-crash --fs-no-reconall --ignore slicetiming --output-spaces MNI152NLin6Asym
fMRIPrep: Please report errors to https://github.com/nipreps/fmriprep/issues

Terminal output of fMRIPrep v20.2.4:

You are using fMRIPrep-20.2.4, and a newer version of fMRIPrep is available: 20.2.5.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 17, in main
    parse_args()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/parser.py", line 635, in parse_args
    config.from_dict(vars(opts))
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 648, in from_dict
    execution.load(settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 218, in load
    cls.init()
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 469, in init
    indexer=_indexer,
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 137, in __init__
    root, description = validate_root(root, validate)
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py", line 77, in validate_root
    json.dumps(EXAMPLE_BIDS_DESCRIPTION)
bids.exceptions.BIDSValidationError: 'dataset_description.json' is missing from project root. Every valid BIDS dataset must have this file.
Example contents of 'dataset_description.json': 
{"Name": "Example dataset", "BIDSVersion": "1.0.2"}
RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=20.10.8 -it -v C:\SOFTWARE\freesurfer\license.txt:/opt/freesurfer/license.txt:ro -v Y:\tmp\docker_test:/data:ro -v Y:\tmp\docker_test\sub-R0230:/out nipreps/fmriprep:20.2.4 /data /out participant --participant-label sub-R0230 --use-syn-sdc --stop-on-first-crash --fs-no-reconall --ignore slicetiming --output-spaces MNI152NLin6Asym
fMRIPrep: Please report errors to https://github.com/nipreps/fmriprep/issues

oesteban · October 22, 2021, 3:52pm

fMRIPrep runs the BIDS-Validator (unless the argument --skip-bids-validation is provided) before it starts. In this case, the BIDS-Validator is complaining that the dataset is not correctly organized. In particular, it found a non-conformant folder .data under the structure of at least one of the subjects.

andrewcrow · November 10, 2021, 2:31pm

Pardon the late reply.

That’s understood. However, there is no folder labeled .data under any subject directory as in the tree output above. Is there any reason to believe this has something to do with fMRIPrep?

oesteban · November 10, 2021, 6:02pm

Okay, actually there are two errors, both derived from nonvalid BIDS structures. fMRIPrep 20.2.3 seems to contain an older version of the validator and is complaining about that .data folder - or the folder just got deleted prior the execution of 20.2.4.

Then 20.2.4 is trying to find a top-level dataset_description.json file, which is mandated by the standard, but is missing in your dataset.

andrewcrow · December 1, 2021, 3:26pm

Hi Oscar,

Thanks for continuing to follow-up with this.

Regarding the error with version 20.2.3, files aren’t deleted when fMRIPrep via Docker is executed. With respect to version 20.2.4, the dataset_description.json is similarly never missing from the project folder.

Being that I can (or could) run both versions of fMRIPrep via Docker locally when the data is pushed to my local hard drive, as opposed to over the network share, I’m starting to think things have less to do with the fMRIPrep Docker implementation. That leaves with their Docker or the network share being the culprit, unless I’m misunderstanding something. Would you agree with that?

Like I’ve said, I have output from docker inspect which I think could be useful. Would you be willing to take a look at that?

Thanks again,
Andrew

oesteban · December 1, 2021, 5:37pm

Yes, to be clear, this doesn’t seem like an fMRIPrep problem. It could be a Docker issue (the folder is not correctly accessed from the container) or a data issue (data are indeed invalid).