FMRIPREP from Singularity: UnboundLocalError

fmriprep
bids

#21

Can I ask you to download and use this singularity image? - https://www.dropbox.com/s/3vhx6soodrtjei0/poldracklab_fmriprep_unstable-2019-01-28-5960e58ff833.simg?dl=0


#22

Both of you are using singularity when running into issues. Is there a python installation with a different version of pybids in it? I’m guessing that your python environment might be sneaking into the container.

I had a similar hunch, since I had that problem before. I thus tried:

  1. Setting the PYTHONPATH to ‘’
  2. Using the --cleanenv argument
  3. Run the image on a completely different HPC environment, on a clean account where I did not touch anything yet.

All to no avail :confused:

Can I ask you to download and use this singularity image? - https://www.dropbox.com/s/3vhx6soodrtjei0/poldracklab_fmriprep_unstable-2019-01-28-5960e58ff833.simg?dl=0

Thanks. I keep having the not found error though :confused: :

Important thing to mention is that older fmriprep versions now also do not work for me anymore. Specifically fmriprep 1.1.2, 1.1.4, 1.1.8. I played around with images

  1. Built on a different server
  2. Built using singularity build
  3. Built using singularity pull

To no avail…


#23

OK. Some more info: on yet another cluster system (“spinoza”) I can get fmriprep to work on the exact same data. That system uses Singularity 2.6.1-dist. The HPC where it does not work is singularity version 3.0.1-1.el7 (“LISA”) and 2.6.1-HEAD.9103f015 (“Cartesius”).

Yet another dataset with only anatomical data, that worked fine before, also does not work on Cartesius.


#24

It seems that in our HPC they have recently updated Singularity and since then some folders that used to be bound are not anymore. Could this be the case?


#25

I can’t replicate with singularity 2.6.1-2~nd18.04+1


#26

I also get the same issue of ‘Exception: No BOLD images found’ for the singularity image on Dropbox.


#27

Hi @PhilK

Can you check with other subjects (--participant-label)?

Thanks!


#28

It seems that in our HPC they have recently updated Singularity and since then some folders that used to be bound are not anymore. Could this be the case?

Maybe, but when I do

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gillesd/sourcedata_fmriprep')
>>> layout.get(subject='01', type='bold')

I do get what I expect:

[File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii', subject='01', task='randomdotmotion', run=1, type='bold', modality='func'), 
File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii', subject='01', task='randomdotmotion', run=2, type='bold', modality='func'), 
File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-03_bold.nii', subject='01', task='randomdotmotion', run=3, type='bold', modality='func')]

#29

I’m sorry for having to disappoint you again, but the problem occurs with other subjects as well (I tested a random sample of ~10 other subjects).


#30

Hey @effigies any thoughts on this one? I might be missing out something.


#31

Possibly let’s make sure we’re importing the correct things. I don’t see how they can possibly be hitting this ses_uids thing in the latest release:

singularity shell fmriprep-1.2.6-1.simg
> python
>>> import bids, niworkflows, fmriprep
>>> print(fmriprep.__version__)
>>> print(niworkflows.__version__)
>>> print(bids.__version__)

#32

Some additional commands that may help diagnose this:

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gillesd/sourcedata_fmriprep')
>>> layout.get(subject='01', type='bold', modality='func', extensions=['nii', 'nii.gz'])

and

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from niworkflows.utils.bids import collect_data
>>> collect_data('/home/gillesd/sourcedata_fmriprep', '01')

#33

Possibly let’s make sure we’re importing the correct things. I don’t see how they can possibly be hitting this ses_uids thing in the latest release:

>>> import bids, niworkflows, fmriprep
>>> print(fmriprep.__version__)>>> print(fmriprep.__version__)
1.2.6-1
>>> print(niworkflows.__version__)
0.5.2.post7
>>> print(bids.__version__)
0.6.5
>>>

#34

OK

> singularity shell fmriprep-latest.simg # (This is 1.2.6-1)
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gholland/data/bias_task/fmriprep_sourcedata')
>>> layout.get(subject='01', type='bold', modality='func', extensions=['nii', 'nii.gz'])
[File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii', subject='01', task='randomdotmotion', run=1, type='bold', modality='func'), File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii', subject='01', task='randomdotmotion', run=2, type='bold', modality='func'), File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-03_bold.nii', subject='01', task='randomdotmotion', run=3, type='bold', modality='func')]

But:

>>> from niworkflows.utils.bids import collect_data
>>> collect_data('/home/gholland/data/bias_task/fmriprep_sourcedata')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/bids.py", line 181, in collect_data
    subj_data.update({"bold": ses_uids})
UnboundLocalError: local variable 'ses_uids' referenced before assignment

#35

Further update: a colleague of mine gets the “no T1w images found”-error with fmriprep 1.1.8, even though this exact image, code, and data used to run fine on this cluster (note that this dataset only contains anatomical data). I will drop an email to the people that take care of the cluster as well.


#36

@Gilles_de_Hollander and @PhilK, can I trouble you to try out the recently release fMRIPrep 1.3.0? We didn’t add any specific fixes for this issue, but I cannot understand how you were getting those errors on 1.2.6-1, unless there somehow was a container-related issue where remnants of 1.2.6 were still around to cause trouble. It’s my hope that a brand new version will have cleared out the debris.


#37

A new error :smiley:


#38

Ah, thanks. We really need to start directly testing Singularity images; this is a permissions issue related to the fact that files generated by Docker are not generally writeable. In a few hours we should have an :unstable image you can try.


#39

Can you try with poldracklab/fmriprep:unstable?


#40

I’ve just triggered the deployment of a new post release (version 1.3.0.post1) which should not have this problem (I’ve tested it myself and could not replicate). It should be ready in a few hours (when all these steps should are green - https://circleci.com/workflow-run/ed055daa-988f-4ac1-8bea-7c610009f54a).