FMRIPREP from Singularity: UnboundLocalError

oesteban · January 28, 2019, 10:46pm

Can I ask you to download and use this singularity image? - https://www.dropbox.com/s/3vhx6soodrtjei0/poldracklab_fmriprep_unstable-2019-01-28-5960e58ff833.simg?dl=0

Gilles_de_Hollander · January 29, 2019, 10:15am

Both of you are using singularity when running into issues. Is there a python installation with a different version of pybids in it? I’m guessing that your python environment might be sneaking into the container.

I had a similar hunch, since I had that problem before. I thus tried:

Setting the PYTHONPATH to ‘’
Using the --cleanenv argument
Run the image on a completely different HPC environment, on a clean account where I did not touch anything yet.

All to no avail

Can I ask you to download and use this singularity image? - https://www.dropbox.com/s/3vhx6soodrtjei0/poldracklab_fmriprep_unstable-2019-01-28-5960e58ff833.simg?dl=0

Thanks. I keep having the not found error though :

gist.github.com

https://gist.github.com/Gilles86/908aeed19d382aaf1aab75c232f74bc6

gistfile1.txt

[gillesd@tcn600 ~]$ singularity run --cleanenv fmriprep-fix.simg /home/gillesd/fmriprep_sourcedata/ ~/derivatives/ participant --participant-label 01 --fs-license-file ~/license.txt --no-submm-recon
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
	1: [WARN] You should define 'EffectiveEchoSpacing' for this file. If you don't provide this information field map correction will not be possible. (code: 8 - EFFECTIVE_ECHO_SPACING_NOT_DEFINED)
		./sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii
		./sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii
		./sub-01/func/sub-01_task-randomdotmotion_run-03_bold.nii

	2: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. (code: 13 - SLICE_TIMING_NOT_DEFINED)
		./sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii
		./sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii

This file has been truncated. show original

Important thing to mention is that older fmriprep versions now also do not work for me anymore. Specifically fmriprep 1.1.2, 1.1.4, 1.1.8. I played around with images

Built on a different server
Built using singularity build
Built using singularity pull

To no avail…

Gilles_de_Hollander · January 29, 2019, 5:40pm

OK. Some more info: on yet another cluster system (“spinoza”) I can get fmriprep to work on the exact same data. That system uses Singularity 2.6.1-dist. The HPC where it does not work is singularity version 3.0.1-1.el7 (“LISA”) and 2.6.1-HEAD.9103f015 (“Cartesius”).

Yet another dataset with only anatomical data, that worked fine before, also does not work on Cartesius.

oesteban · January 29, 2019, 7:15pm

It seems that in our HPC they have recently updated Singularity and since then some folders that used to be bound are not anymore. Could this be the case?

oesteban · January 29, 2019, 7:28pm

I can’t replicate with singularity 2.6.1-2~nd18.04+1

PhilK · January 29, 2019, 8:29pm

I also get the same issue of ‘Exception: No BOLD images found’ for the singularity image on Dropbox.

oesteban · January 29, 2019, 8:56pm

Hi @PhilK

Can you check with other subjects (--participant-label)?

Thanks!

Gilles_de_Hollander · January 30, 2019, 9:47am

It seems that in our HPC they have recently updated Singularity and since then some folders that used to be bound are not anymore. Could this be the case?

Maybe, but when I do

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gillesd/sourcedata_fmriprep')
>>> layout.get(subject='01', type='bold')

I do get what I expect:

[File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii', subject='01', task='randomdotmotion', run=1, type='bold', modality='func'), 
File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii', subject='01', task='randomdotmotion', run=2, type='bold', modality='func'), 
File(filename='/home/gillesd/fmriprep_data/sub-01/func/sub-01_task-randomdotmotion_run-03_bold.nii', subject='01', task='randomdotmotion', run=3, type='bold', modality='func')]

PhilK · January 30, 2019, 7:25pm

I’m sorry for having to disappoint you again, but the problem occurs with other subjects as well (I tested a random sample of ~10 other subjects).

oesteban · January 30, 2019, 9:01pm

Hey @effigies any thoughts on this one? I might be missing out something.

effigies · January 30, 2019, 9:18pm

Possibly let’s make sure we’re importing the correct things. I don’t see how they can possibly be hitting this ses_uids thing in the latest release:

singularity shell fmriprep-1.2.6-1.simg
> python
>>> import bids, niworkflows, fmriprep
>>> print(fmriprep.__version__)
>>> print(niworkflows.__version__)
>>> print(bids.__version__)

oesteban · January 30, 2019, 11:53pm

Some additional commands that may help diagnose this:

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gillesd/sourcedata_fmriprep')
>>> layout.get(subject='01', type='bold', modality='func', extensions=['nii', 'nii.gz'])

and

> singularity shell fmriprep-1.2.6.simg
>> python
>>> from niworkflows.utils.bids import collect_data
>>> collect_data('/home/gillesd/sourcedata_fmriprep', '01')

Gilles_de_Hollander · January 31, 2019, 10:45am

Possibly let’s make sure we’re importing the correct things. I don’t see how they can possibly be hitting this ses_uids thing in the latest release:

>>> import bids, niworkflows, fmriprep
>>> print(fmriprep.__version__)>>> print(fmriprep.__version__)
1.2.6-1
>>> print(niworkflows.__version__)
0.5.2.post7
>>> print(bids.__version__)
0.6.5
>>>

Gilles_de_Hollander · January 31, 2019, 11:13am

OK

> singularity shell fmriprep-latest.simg # (This is 1.2.6-1)
>>> from bids import BIDSLayout
>>> layout = BIDSLayout('/home/gholland/data/bias_task/fmriprep_sourcedata')
>>> layout.get(subject='01', type='bold', modality='func', extensions=['nii', 'nii.gz'])
[File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-01_bold.nii', subject='01', task='randomdotmotion', run=1, type='bold', modality='func'), File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-02_bold.nii', subject='01', task='randomdotmotion', run=2, type='bold', modality='func'), File(filename='/home/gholland/data/bias_task/fmriprep_sourcedata/sub-01/func/sub-01_task-randomdotmotion_run-03_bold.nii', subject='01', task='randomdotmotion', run=3, type='bold', modality='func')]

But:

>>> from niworkflows.utils.bids import collect_data
>>> collect_data('/home/gholland/data/bias_task/fmriprep_sourcedata')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/miniconda/lib/python3.7/site-packages/niworkflows/utils/bids.py", line 181, in collect_data
    subj_data.update({"bold": ses_uids})
UnboundLocalError: local variable 'ses_uids' referenced before assignment

Gilles_de_Hollander · January 31, 2019, 11:14am

Further update: a colleague of mine gets the “no T1w images found”-error with fmriprep 1.1.8, even though this exact image, code, and data used to run fine on this cluster (note that this dataset only contains anatomical data). I will drop an email to the people that take care of the cluster as well.

effigies · February 8, 2019, 2:05pm

@Gilles_de_Hollander and @PhilK, can I trouble you to try out the recently release fMRIPrep 1.3.0? We didn’t add any specific fixes for this issue, but I cannot understand how you were getting those errors on 1.2.6-1, unless there somehow was a container-related issue where remnants of 1.2.6 were still around to cause trouble. It’s my hope that a brand new version will have cleared out the debris.

oesteban · March 5, 2019, 10:04pm

16 posts were split to a new topic: fMRIPrep 1.3.x on Singularity - git-annex permission errors

effigies · February 14, 2019, 3:01pm

Is the UnboundLocalError resolved as well?

Gilles_de_Hollander · February 14, 2019, 3:15pm

Yes. Fmriprep now starts to run properly on my data…

oesteban · February 14, 2019, 4:49pm

This has been a puzzling one. Thank you all for your patience and collaborative spirit!

@PhilK, @Benjamin_Zimmerman can you confirm the good news on your end?