fMRIPrep Python StopIteration Problem

Software Support
1

Summary of what happened:

Hi all,
I’m getting a StopIteration when trying to start fMRIPrep. We have a pipeline for preprocessing that we scripted, and it calls fMRIPrep, and it had been working fine until about a week ago. Nothing has changed - same fmriprep container, this data had been run successfully before, etc…

Command used (and if a helper script was used, a link to the helper script or the command generated):

export OMP_NUM_THREADS=64
export APPTAINERENV_TEMPLATEFLOW_HOME=/opt/templateflow
unset PYTHONPATH; singularity run -e \
        --bind </path/to/inputDir>/ \
        --bind </path/to/opt/templateflow:/opt/templateflow> \
        --bind </path/to/workflowDir> \
        </path/to/container/${fmriprepversion}.sif> \
        </path/to/inputDir> <path/to/outputDir> participant \
        --participant-label sub-${participantid} \
        -w </path/to/workflowDir> \
        --fs-license-file </path/to/license.txt> \
        -v \
        --skip_bids_validation \
        --bids-filter-file </path/to/filterfiles.json> \
        --stop-on-first-crash \
        --mem-mb 190000 --nthreads ${OMP_NUM_THREADS} \
        --output-spaces MNI152NLin2009cAsym:res-2 MNI152NLin2009cAsym

Version:

fmriprepv23.2.0

Environment (Docker, Singularity / Apptainer, custom installation):

Using Apptainer

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/bin/fmriprep", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/run.py", line 40, in main
    parse_args()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/parser.py", line 802, in parse_args
    config.from_dict({})
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 691, in from_dict
    execution.load(settings, init=initialize('execution'), ignore=ignore)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 235, in load
    cls.init()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 485, in init
    cls._layout = BIDSLayout(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/layout.py", line 177, in __init__
    _indexer(self)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 147, in __call__
    all_bfs, all_tag_dicts = self._index_dir(self._layout._root, self._config)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 232, in _index_dir
    dir_bfs, dir_tag_dicts = self._index_dir(d, config, force=force)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 232, in _index_dir
    dir_bfs, dir_tag_dicts = self._index_dir(d, config, force=force)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 206, in _index_dir
    _, dirnames, filenames = next(os.walk(path))
StopIteration

Thanks for any help!
Wade

2

Hi @wade,

Please return the BIDS validation report. Also see Got StopIteration: ('/data', [], []) error · Issue #3396 · nipreps/fmriprep · GitHub.

Best,
Steven

3

We added the --skip_bids_validation, so I don’t think we have a report. We temporarily have some folders in the “BIDS” area that are not BIDS-compliant, but those will go away. Do you think that could be the issue?

4

Could be an issue. May you please produce a report?

5

I can work on that, but it may take a little time. Our data is on an HPC, so I think I’d have to download our data (at least a sample) to run it on a validator, or run without the --skip_bids_validation (which is fine, but since it’s a busy system we may end up waiting in the queue until tomorrow for it to run. I’ll work on getting that going, but in the meantime, the thing that is making me suspicious is that we are rerunning some data. For instance, subject b199. I took the old “sub-b199” folder in the BIDS area and renamed it “sub-b199-rerunning.” Was this just a completely dumb idea and creating the issue?

6

Hi @wade,

You can run the validator in terminal without submitting to job scheduler. GitHub - bids-standard/bids-validator: Validator for the Brain Imaging Data Structure for most recent version. Or singularity exec -e -B $PWD /path/to/image.sif bids-validator $PWD (run in the BIDS root) to use that bundled with fmriprep version.

You should keep your BIDS rootdir as valid as possible. If you need to add folders/files to .bidsignore you can but that should not be considered a sustainable solution. Also I am not sure I understand why you would be renaming the original data folder for rerunning purposes.

Best,
Steven

7

I’ve been trying to keep the BIDS root as clean as I can. I ran the bids-validator and it popped an “Unhandled rejection” error saying there is a folder in our derivatives area that is denying permissions. (It’s a folder one of our co-PIs made for some of his analyses, so it has a different group number). Maybe this is it? I deleted all the “-rerun” folders from the BIDS root, and still got the same error when trying to run fMRIPrep. I’m going to create a new project directory and see if a “clean,” new BIDS environment will work.

8

I would suggest not putting your derivatives inside your dataset. Consider something along the lines of Nipoppy.

Any of a number of layouts is fine. If you’re going to nest at all, nest sources inside derivatives, not the other way around, or else you change your input dataset every time you process it. Trusting software to ignore certain parts of the dataset is bound to fail, and you’re lucky if the failures only occur for read operations and not write.

9

I agree. I didn’t set up the initial file structure, and your example is what I am “overhauling” the pipeline to. Right now, I just have to get it to work. We have a deadline.

I did get a work-around for now until I can clean everything up, so it’s running.

Question for all, though: Our HPC has strict file count restrictions, so we need to .tar just about everything. Do you know, if we tar the session folders (e.g. we have a sub-### folder, and under that we have ses-01.tar, ses-02.tar, and ses-03.tar), will this freak the bids-validator out and crash fmriprep?

Thanks for all the help everyone.

10

Are you talking about raw or derived?

If raw, perhaps put these in a separate directory and unzip and reorganize files at runtime. If these .tar files do end up in the raw root dir (which they shouldn’t) put them in .bidsignore.

Best,
Steven

11

Both, actually. Raw and derivatives.

So when fMRIPrep checks for validation, does it check the whole BIDS area, or just the subject’s structure?

When our fMRIPrep session is called, the session isn’t tarred at that point. That subject’s other sessions may be tar files (in the raw area), but there would not be a ses-03 and a ses-03.tar at the same time.

Thanks, again. I’m not too bad with the BIDS file/folder structuring and naming, but it’s the details like this that I’m fuzzy on. And how to balance BIDS’ requirements and our HPC. :slight_smile:

12

Yes. Some things like derivatives, code, sourcedata, and things you put in .bidsignore are ignored but can still cause crashes if their file permissions are restrictive.

13

OK, that makes sense.
I’m pretty sure this is just a problem with out BIDS structure, then.

Thanks!