Fmriprep v.23.1.3 fails to decode temporary JSON file

1

Hello,
We have been re-analyzing some data that had been preprocessed successfully with fmriprep v.20.0.6. Upgrading to 23.1.3, we are getting errors related to the encoding of the json files, even though the dataset seems to be bids compliant (no errors thrown when using older fmriprep versions and checked with the online bids validator). This thread seems related: Error running fmriprep 23.x using Docker
and the solution there was to upgrade Docker version. We are using singularity on a shared HPC cluster where we are not in control of the singularity containers (Singularity version: apptainer version 1.1.9-1.el7). I was wondering if there are other ideas for troubleshooting or suggested stable fmriprep versions.

This is the log file with the error:

RUNNING FMRIPREP
INFO:    Environment variable SINGULARITYENV_FS_LICENSE is set, but APPTAINERENV_FS_LICENSE is preferred
INFO:    underlay of /etc/localtime required more than 50 (100) bind mounts
You are using fMRIPrep-23.1.3, and a newer version of fMRIPrep is available: 23.1.4.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 295, in load_json
    return json.load(handle)
  File "/opt/conda/envs/fmriprep/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/opt/conda/envs/fmriprep/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/bin/fmriprep", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/run.py", line 43, in main
    parse_args()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/parser.py", line 786, in parse_args
    config.from_dict({})
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 678, in from_dict
    execution.load(settings, init=initialize('execution'), ignore=ignore)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 232, in load
    cls.init()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/config.py", line 476, in init
    cls._layout = BIDSLayout(
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/layout.py", line 176, in __init__
    indexer(self)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 150, in __call__
    self._index_metadata()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 403, in _index_metadata
    file_md.update(pl())
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/bids/layout/index.py", line 297, in load_json
    raise OSError(
OSError: Error occurred while trying to decode JSON from file <MYPATH/input/sub-xxx/ses-1/anat/._sub-xxx_ses-1_T1w.json

I’m getting the same error with the same file even though I’m running this job on a different subject, and using the --skip-bids-validation flag.
Anyone ran into similar issues or has suggestions of bypassing this issue? recommended fmriprep version that’s working well using singularity? Thank you for any leads,
Maya

2

Hi @mayay,

It looks like you have some hidden files in your directory (based on it beginning with a “.”). Some text editors create these temporary files when you’re in the editor, and they aren’t always deleted. File browsers hide files that begin with a “.” by default, so there’s sometimes no way to know that they’re there.

If you feel comfortable running a rm command, you can try something like rm sub*/*/._* from the root of your BIDS directory (but please test that on a copy directory first, I don’t want to be responsible for deleting your data :sweat_smile:). Otherwise, you can create a .bidsignore file in your BIDS root directory, and have a line that says sub*/*/._*

Best,
Steven

3

I deleted these hidden files and that solved the issue, thanks so much! I would never have thought about something like that. Bids-validator had no issues with these files and neither have previous versions of fmriprep, so it’s interesting that the new version (23.1.3) somehow does see them.