fMRIPrep: Could not generate CITATION.html file

fmriprep

#1

I think I might be hitting an issue related to the one described above. I was using a singularity image of fmriprep 1.2.6-1 to process some data, which was working fine. I then upgraded to a container of version 1.3.1 and ran on some of the same data, which initially worked fine. However, I then started getting the string of errors pasted below. I never cleared out the work directory when making the version switch - not sure if that contributed. I now get the errors even though I erased the 1.3.1 singularity image, rebuilt one for 1.2.6-1, and tried running on clean data with an empty work directory. Any idea what I might be doing wrong or how I can debug? Thanks for any suggestions.

Here is the command I used:

unset PYTHONPATH
fmriprep --participant_label 01 --nthreads 8 --verbose --output-space template T1w fsnative --use-aroma --fs-license-file 
/gpfs/group/sjw42/default/sjw42_collab/sw/freesurfer-6.0.1/license.txt --work /gpfs/group/sjw42/default/ASH/TEST/work /gpf
s/group/sjw42/default/ASH/TEST/bids /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives participant

And here is the section of errors:

### References

/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, Set, Iterable
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:121: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/dataset_description.json' mode='r' encoding='UTF-8'>
self.description = json.load(open(target, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/grabbit/core.py:436: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/config/bids.json' mode='r' encoding='UTF-8'>
domain = json.load(open(domain, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/task-cardguess_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-01_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/phasediff.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/fmap/sub-01_phasediff.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/workflows/dmri/mrtrix/group_connectivity.py:16: UserWarning: cmp not installed
warnings.warn('cmp not installed')
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-02_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-03_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-04_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-05_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-06_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/task-rest_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file <_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-rest_bold.json' mode='r' encoding='utf-8'>
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (<class 'DeprecationWarning'>): Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
Captured warning (<class 'ResourceWarning'>): unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
Captured warning (<class 'ImportWarning'>): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'>
imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)

FMRIPREP from Singularity: UnboundLocalError
#2

Hi @Steve_Wilson, your snippet only contains warnings. They are noisy and scary, but I couldn’t find any errors on them. Could you post the full output of your command line?


#3

Thanks, @oesteban . I may have been overinterperting the warnings (I don’t recall getting the before and they are a bit scary, as you mentioned :slight_smile:). I was having an issue with the freesurfer steps not working, but I have figured out that this was related to an issue with the environment. I mistakenly thought it was related to the warning messages. Now that I’ve gotten the environment setup straightened out, it looks like I am only getting the error pasted below. And I only got it for one of two subjects that I tested. I retried the problematic subject separately and wrote to a clean directory to make sure it wasn’t an overwrite issue, but I still got the error. I’m guessing it is not a big problem – so far the results seem fine – but I thought I would pass along the information in case it is useful. Thanks again for your help.

 Could not generate CITATION.html file:

pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle="fMRIPrep citation boilerplate" /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.md -o /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html

#4

Thanks a lot @Steve_Wilson. Could you post the full output log and exact command line for me to trace this problem back?


#5

Sure thing, @oesteban - sorry for the delay. I’ve pasted the exact command and output below. Let me know if either is not what you were looking for or if you need more information.

Here is the set of commands, which were submitted together as a batch job:

set DISPLAY variable

tmpdis1=echo $DISPLAY | awk -F ":" '{print $1}'
tmpdis2=echo $DISPLAY | awk -F ":" '{print $2}'| awk -F '.' '{print $1}'
export DISPLAY=echo $tmpdis1":"$tmpdis2

freesurfer setup

export FREESURFER_HOME=/gpfs/group/sjw42/default/sjw42_collab/sw/freesurfer-6.0.1
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export SUBJECTS_DIR="/gpfs/group/sjw42/default/freesurfer_subjects"
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export FS_LICENSE=$FREESURFER_HOME/license.txt

Go to the correct location

cd $PBS_O_WORKDIR

Run job

unset PYTHONPATH
/usr/bin/singularity run /gpfs/group/sjw42/default/sjw42_collab/sw/singularity/fmriprep/sjw42-fmriprep_icsaci-master-rec.simg --participant_label 05 --nthreads 8 --verbose --work /gpfs/group/sjw42/default/ASH/TEST/work /gpfs/group/sjw42/default/ASH/TEST/bids /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives participant

Here is a link to the output file: https://www.dropbox.com/s/viv47b0a6vf848l/sub-05_fmriprep.pbs.o11460664?dl=0


#6

Thanks very much! That is a WARNING too, it seems it worked well. The problem of that particular WARNING is that your reports will not have an html boilerplate under the “Methods” section.

Can I ask you to run the following commands?

unset PYTHONPATH
/usr/bin/singularity exec /gpfs/group/sjw42/default/sjw42_collab/sw/singularity/fmriprep/sjw42-fmriprep_icsaci-master-rec.simg \
    pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle="fMRIPrep citation boilerplate" /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.md -o /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html

In principle, that should give you an error. I would really appreciate to have the output.

When successful, it should write an HTML file (/gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html).


#7

Okay, so that doesn’t actually give any errors. It produces the html file (or rewrites if it already exists). I took a closer look, and the html file is being created/rewritten as designed for most of my subjects. So far, I am only getting the error when using fmriprep to process data for one specific subject, so there seems to be something idiosyncratic going awry? I’ve double-checked the data, command, etc., and can’t find anything that looks problematic. I’m happy to run any additional testing that would be useful on your end.


#8

It might be a one-off filesystem synchronization problem. Please ping us back if suddenly it becomes a problem.