FMRIPREP from Singularity: UnboundLocalError

oesteban · January 19, 2019, 6:34am

Thanks for reporting, I’ve started a solution here https://github.com/poldracklab/niworkflows/pull/285

Will let you know when a new release fixing this comes out (should be pretty fast)

oesteban · January 19, 2019, 6:52am

BTW, I would advise against running 41 subjects in parallel, you’ll probably hit memory limits. I would recommend using the flag --participant-label <label> and parallelize by two or three tops.

PhilK · January 19, 2019, 10:10am

Great, thank you very much!

PhilK · January 19, 2019, 11:15am

By the way, I’m working with multiecho (2 echos) data, in case this may matter here.

oesteban · January 19, 2019, 7:22pm

I think it is. Could you post the BIDS filesystem tree for the one subject failing here?

EDIT: I noticed only now that you were using --participant-label 01

PhilK · January 19, 2019, 10:58pm

Sure, here you go:

sourcedata/
   sub-01/
      anat/      
         sub-01_T1w.json
         sub-01_T1w.nii
      fmap/      
         sub-01_run-1_magnitude1.nii
         sub-01_run-1_magnitude2.nii
         sub-01_run-1_phasediff.json
         sub-01_run-1_phasediff.nii
         sub-01_run-2_magnitude1.nii
         sub-01_run-2_magnitude2.nii
         sub-01_run-2_phasediff.json
         sub-01_run-2_phasediff.nii
      func/      
         sub-01_task-AudLoc_echo-1_bold.json
         sub-01_task-AudLoc_echo-1_bold.nii
         sub-01_task-AudLoc_echo-2_bold.json
         sub-01_task-AudLoc_echo-2_bold.nii
         sub-01_task-AudLoc_events.tsv
         sub-01_task-FJT_echo-1_bold.json
         sub-01_task-FJT_echo-1_bold.nii
         sub-01_task-FJT_echo-2_bold.json
         sub-01_task-FJT_echo-2_bold.nii
         sub-01_task-FJT_events.tsv
         sub-01_task-LDT_echo-1_bold.json
         sub-01_task-LDT_echo-1_bold.nii
         sub-01_task-LDT_echo-2_bold.json
         sub-01_task-LDT_echo-2_bold.nii
         sub-01_task-LDT_events.tsv
         sub-01_task-MotLoc_echo-1_bold.json
         sub-01_task-MotLoc_echo-1_bold.nii
         sub-01_task-MotLoc_echo-2_bold.json
         sub-01_task-MotLoc_echo-2_bold.nii
         sub-01_task-MotLoc_events.tsv

Most other subjects have the same exact structure, with the exception of 3 subjects with an additional fieldmap (run-3) as they had to be taken out of and back in to the scanner within a session.

EDIT: fmriprep fails with the same error message posted above on every subject.

oesteban · January 25, 2019, 7:07pm

Hi @PhilK,

We just rolled out a new patch that should address your issue. Could you build a new singularity image from poldracklab/fmriprep:unstable and let us know if that fixed your issue?

Thanks!

Gilles_de_Hollander · January 27, 2019, 1:37pm

Hey,

I’ve been struggling with exactly this error for a week now. If I run the same version in Docker everything works fine. When I use singularity on a clean HPC account, I get this:
singularity run fmriprep-1.2.6.simg ~/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt
UnboundLocalError: local variable 'ses_uids' referenced before assignment

When I use the unstable branch, I get this error:
singularity run fmriprep-1.2.6_fix.simg ~/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt
UnboundLocalError: local variable 'ses_uids' referenced before assignment

Subject 01 looks like this:

sub-01
|-- anat
|   `-- sub-01_T1w.nii
|-- fmap
|   |-- sub-01_magnitude1.nii
|   |-- sub-01_magnitude2.nii
|   |-- sub-01_phasediff.json
|   `-- sub-01_phasediff.nii
`-- func
    |-- sub-01_task-randomdotmotion_run-01_bold.nii
    |-- sub-01_task-randomdotmotion_run-02_bold.nii
    `-- sub-01_task-randomdotmotion_run-03_bold.nii

and runs fine with the docker image at 1.2.6

oesteban · January 28, 2019, 6:06am

Can you copy the full traceback with fmriprep-1.2.6_fix.simg? I’m curious whether it is the same line.

Gilles_de_Hollander · January 28, 2019, 3:08pm

Sure

gist.github.com

https://gist.github.com/Gilles86/068b92c8a71818b4a7e693f7a3005b5b

fmriprep-1.2.6

[gillesd@tcn662 ~]$ singularity run fmriprep-1.2.6.simg ~/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 02 --fs-license-file ~/license.txt --output-space T1w
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
	1: [WARN] You should define 'EffectiveEchoSpacing' for this file. If you don't provide this information field map correction will not be possible. (code: 8 - EFFECTIVE_ECHO_SPACING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-03_bold.nii

	2: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii

This file has been truncated. show original

fmriprep-1.2.6_fix ('poldracklab-fmriprep:unstable`)

[gillesd@tcn662 ~]$ singularity run fmriprep-1.2.6_fix.simg ~/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 02 --fs-license-file ~/license.txt --output-space T1w
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
	1: [WARN] You should define 'EffectiveEchoSpacing' for this file. If you don't provide this information field map correction will not be possible. (code: 8 - EFFECTIVE_ECHO_SPACING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-03_bold.nii

	2: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii

This file has been truncated. show original

oesteban · January 28, 2019, 4:01pm

Thanks Gilles, this is super-helpful. Let me write a couple of tests with your structure and confirm there is still a problem.

PhilK · January 28, 2019, 4:04pm

Dear @oesteban,

I get exactly the same pattern of problems as @Gilles_de_Hollander. With the new version, I now get the error ‘Exception: No BOLD images found for participant 01 and task <all>. All workflows require BOLD images.’ This also occurs when explicitly asking for a specific task (e.g. using the tag “–task-id LDT” results in the error message ‘Exception: No BOLD images found for participant 01 and task LDT. All workflows require BOLD images.’)

Here’s the full report:

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
This dataset appears to be BIDS compatible.
        Summary:                    Available Tasks:        Available Modalities: 
        1239 Files, 167.27GB        AudLoc                  T1w                   
        41 - Subjects               FJT                     bold                  
        1 - Session                 LDT                     fieldmap              
                                    MotLoc                                        

If you have any questions please post on https://neurostars.org/tags/bids

/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
  imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
190128-15:55:35,135 nipype.workflow IMPORTANT:
	 
    Running fMRIPREP version 1.2.6-1:
      * BIDS dataset path: /data/pt_01902/Data/BIDS/sourcedata.
      * Participant list: ['01'].
      * Run identifier: 20190128-155535_4455ad01-1a89-4217-ab60-161e62a53d49.
    
/usr/local/miniconda/lib/python3.7/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping, Set, Iterable
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:121: ResourceWarning: unclosed file <_io.TextIOWrapper name='/data/pt_01902/Data/BIDS/sourcedata/dataset_description.json' mode='r' encoding='UTF-8'>
  self.description = json.load(open(target, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/grabbit/core.py:436: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/config/bids.json' mode='r' encoding='UTF-8'>
  domain = json.load(open(domain, 'r'))
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 749, in build_workflow
    ignore_aroma_err=opts.ignore_aroma_denoising_errors,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 217, in init_fmriprep_wf
    ignore_aroma_err=ignore_aroma_err,
  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/workflows/base.py", line 389, in init_single_subject_wf
    subject_id, task_id if task_id else '<all>'))
Exception: No BOLD images found for participant 01 and task <all>. All workflows require BOLD images.

oesteban · January 28, 2019, 4:06pm

I wonder whether the nii extension (instead of nii.gz) is causing this. Could you check?

I’m thinking we might have to revise the use of pybids to query the dataset. Maybe some filter has changed.

Gilles_de_Hollander · January 28, 2019, 4:30pm

I wonder whether the nii extension (instead of nii.gz ) is causing this. Could you check?

For me this didn’t change anything

gist.github.com

https://gist.github.com/Gilles86/3174e5cc219d9acf6c0f14f75c57e698

gistfile1.txt

[gillesd@tcn987 ~]$ singularity run fmriprep-1.2.6_fix.simg ~/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 02 --fs-license-file ~/license.txt
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
	1: [WARN] You should define 'EffectiveEchoSpacing' for this file. If you don't provide this information field map correction will not be possible. (code: 8 - EFFECTIVE_ECHO_SPACING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii.gz
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii.gz
		./sub-02/func/sub-02_task-randomdotmotion_run-03_bold.nii.gz

	2: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)
		./sub-02/func/sub-02_task-randomdotmotion_run-01_bold.nii.gz
		./sub-02/func/sub-02_task-randomdotmotion_run-02_bold.nii.gz

This file has been truncated. show original

PhilK · January 28, 2019, 4:32pm

@oesteban Same here, still the same issue.

oesteban · January 28, 2019, 4:32pm

Thanks for the prompt answer

oesteban · January 28, 2019, 5:04pm

Okay, this seems like a pretty serious bug: https://github.com/poldracklab/niworkflows/issues/293. I’ll try to get this fixed ASAP. Thanks a lot for reporting.

oesteban · January 28, 2019, 5:22pm

Both of you are using singularity when running into issues. Is there a python installation with a different version of pybids in it? I’m guessing that your python environment might be sneaking into the container.

@PhilK, in your latest trial, did you unset PYTHONPATH? (I checked you did in the beginning).

PhilK · January 28, 2019, 7:21pm

Yes, the problem also occurs when I run it with ‘unset PYTHONPATH’.

oesteban · January 28, 2019, 10:46pm

Can I ask you to download and use this singularity image? - https://www.dropbox.com/s/3vhx6soodrtjei0/poldracklab_fmriprep_unstable-2019-01-28-5960e58ff833.simg?dl=0