Post-fMRIPrep Processing - Container Error

, ,
1

Hello,

I used fMRIPrep to pre-process task-based data, and am now attempting to do a 1st level analysis based on the container / code found here: https://github.com/poldracklab/ds003-post-fMRIPrep-analysis

I create the image by building a docker image locally (on Windows) then directly pulling it with singularity.

I execute the following commands (which use the container):

export TEMPLATEFLOW_HOME=/home/username/.cache/templateflow

singularity run \
--cleanenv \
-B ${tpflow}:/opt/templateflow \
-B ${bids_root}:/bids_dir \
-B ${output}:/out \
${images}/<container>.sif \
${derivatives} ${out} \
participant \
--participant-label $SUBJECT \
--bids-dir /bids_dir \
--ncpus 12 \
-w ${intermediate}

And get an error:

Failed to import duecredit due to No module named 'duecredit'
Traceback (most recent call last):
  File "/src/run.py", line 6, in <module>
    from templateflow.api import get as tpl_get, templates as get_tpl_list
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/__init__.py", line 19, in <module>
    from . import api
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 7, in <module>
    from .conf import TF_LAYOUT, TF_S3_ROOT, TF_USE_DATALAD
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/conf/__init__.py", line 92, in <module>
    "scripts",
  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 188, in __init__
    **kwargs)
TypeError: __init__() got an unexpected keyword argument 'ignore'

Is this a versioning issue? Based on the error cited below, it seems like the code/container might have more versioning issues? Any guidance would be much appreciated!!

note - this is similar to a previous (unresolved) comment found in this post: https://neurostars.org/t/nipype-second-level-analysis-with-fsl-interface/4659

As an aside, upon attempting to remove the templateflow import statement in run.py, I also got this error:

/usr/local/miniconda/lib/python3.7/site-packages/nipype/workflows/__init__.py:28: UserWarning: Nipype 1 workflows have been moved to the niflow-nipype1-workflows package. pip install niflow-nipype1-workflows to continue using them.
  warnings.warn(" ".join(_msg))
Traceback (most recent call last):
  File "/src/run.py", line 192, in <module>
    sys.exit(main())
  File "/src/run.py", line 135, in main
    from workflows import first_level_wf
  File "/src/workflows.py", line 8, in <module>
    from nipype.workflows.fmri.fsl.preprocess import create_susan_smooth
ModuleNotFoundError: No module named 'nipype.workflows.fmri'

but was able to resolve it by adding niflow-nipype1-workflows to the requirements.txt file. The first cited error pops up later on regardless.

Thank you,
Eli

2

Turns out it was a versioning issue - I was able to solve this by specifying the templateflow version in the requirements file (0.1.9).

I will likely make a post in the future with the changes I had to make to the example workflow. Hopefully it will help any others that want to use it as a starting point!

2 Likes
3

If you want to use the environment that was used when that dataset was analyzed, you could use their prebuilt docker image instead of recreating it locally:

singularity pull ds003-example-v0.0.3.sif docker://poldracklab/ds003-example:0.0.3

And if you wanted to make modifications you could likely make the changes to the code locally and patch them into the container, like so:

singularity run \
--cleanenv \
-B /dir/to/ds003-post-fMRIPrep-analysis:/src \
-B ${tpflow}:/opt/templateflow \
-B ${bids_root}:/bids_dir \
-B ${output}:/out \
${images}/<container>.sif \
${derivatives} ${out} \
participant \
--participant-label $SUBJECT \
--bids-dir /bids_dir \
--ncpus 12 \
-w ${intermediate}

with this line:

-B /dir/to/ds003-post-fMRIPrep-analysis:/src \

patching in your local copy of the repository into the container.

Just as another option in case you do not wish to go through the pain of dependency management.

Best,
James

4

@ebulger Thanks for mentioning this - ran into the same problem and manually downgraded templateflow (and then had to install niflow-nipype1-workflows). Is there a specific reason you used templateflow 0.1.9?

[edit] I should note that I ran into this problem even though I used their dockerfile.

5

@hechlera I went through the templateflow past release dates and matched those up with the date the dockerfile was last committed. See https://github.com/templateflow/python-client/releases?after=0.4.0. Looking again, it seems that version 0.1.8 is more likely what was used back then.

@jdkent Thank you for the tip! That would definitely save some time, but I couldn’t get it working. I got a permissions error:

/.singularity.d/runscript: 39: exec: /src/run.py: Permission denied

Any idea how to solve this?

I am currently running into another issue where the feat_fit node hangs once film_gls is called. The latest log file (stored in the intermediates directory) ends with:

/usr/share/fsl/5.0/bin/film_gls --in=filtered_func_data --rn=stats --pd=design.mat --thr=0.000068 --sa --ms=5 --con=design.con  
Log directory is: stats
paradigm.getDesignMatrix().Nrows()=1000
paradigm.getDesignMatrix().Ncols()=60
sizeTS=1000
numTS=949496
Calculating residuals...

I haven’t done much debugging yet, so will update once I solve this. In the meantime, any ideas are welcome!

Thanks!
Eli

6

if you are running commands in a bash terminal, the following may work:
chmod -R +r ds003-post-fMRIPrep-analysis/ (gives read permissions to your username for all files within the ds003-post-fMRIPrep-analysis directory). I’m still not great at debugging permission issues, sometimes files don’t belong to who you expect resulting in permission denied errors.

If you want all the versions of the python software used in the container, you can type:

 singularity exec --cleanenv ds003-example-0.0.3.simg pip freeze

which outputs this list:

asn1crypto==0.24.0
bids-validator==1.2.2
certifi==2019.3.9
cffi==1.11.5
chardet==3.0.4
Click==7.0
conda==4.6.11
cryptography==2.6.1
cycler==0.10.0
decorator==4.4.0
docopt==0.6.2
funcsigs==1.0.2
future==0.17.1
grabbit==0.2.6
idna==2.7
imageio==2.5.0
isodate==0.6.0
Jinja2==2.10.1
kiwisolver==1.0.1
lxml==4.3.3
MarkupSafe==1.1.1
matplotlib==2.2.2
mkl-fft==1.0.6
mkl-random==1.0.1
networkx==2.2
neurdflib==5.0.0.post1
nibabel==2.4.0
nilearn==0.5.0
nipype==1.1.9
niworkflows==0.8.2
num2words==0.5.9
numpy==1.15.4
packaging==19.0
pandas==0.23.4
patsy==0.5.1
Pillow==6.0.0
prov==1.5.3
pybids==0.7.1
pycosat==0.6.3
pycparser==2.18
pydot==1.4.1
pydotplus==2.0.2
pyOpenSSL==18.0.0
pyparsing==2.3.1
PySocks==1.6.8
python-dateutil==2.8.0
pytz==2018.9
PyWavelets==1.0.3
rdflib==4.2.2
requests==2.19.1
ruamel-yaml==0.15.46
scikit-image==0.15.0
scikit-learn==0.19.1
scipy==1.1.0
seaborn==0.9.0
simplejson==3.16.0
six==1.11.0
svgutils==0.3.1
templateflow==0.1.7
tornado==6.0.2
tqdm==4.31.1
traits==4.6.0
urllib3==1.23

but again if you’re trying to recreate the environment and not change the code, I would recommend using the container since that is the container’s purpose, and in case you need to change the code, we can figure out the permission’s issue you are experiencing.

7

Hi James,

Unfortunately the chmod -R +r ds003-post-fMRIPrep-analysis/ method didn’t solve the problem. It would be helpful to use the pre-built container to ensure that the issue I am currently facing isn’t related to container inconsistencies, so if you have any other ideas please let me know! I have looked around but haven’t come up with anything.

With regards to the current problem I am facing - the container I built runs up until film_gls is at the ‘calculating residuals’ step. I’m not sure why it hangs there. Possibly because the numTS=949496 variable is so large, and if this is the executed code (found here), it could be making the loop shown below run a very long time:

if(!globalopts.noest.value()) {
	cout << "Calculating residuals..." << endl; 
	for(int i = 1; i <= numTS; i++)
	  {						    
            glimGls.setData(datam.Column(i), paradigm.getDesignMatrix(i,mask,labels), i, tContrasts, fContrasts);
	    residuals.Column(i)=glimGls.getResiduals();
	  }
	cout << "Completed" << endl;

Do you (or anyone else) know what numTS refers to here? There must be some error earlier in the container / code that’s propagating to cause this.

Thanks again for your help. I am very new to fmri data processing (and neuroscience in general), so any bits of information are much appreciated!!
Eli