fMRIPrep: Could not generate CITATION.html file

Steve_Wilson · March 12, 2019, 6:08pm

I think I might be hitting an issue related to the one described above. I was using a singularity image of fmriprep 1.2.6-1 to process some data, which was working fine. I then upgraded to a container of version 1.3.1 and ran on some of the same data, which initially worked fine. However, I then started getting the string of errors pasted below. I never cleared out the work directory when making the version switch - not sure if that contributed. I now get the errors even though I erased the 1.3.1 singularity image, rebuilt one for 1.2.6-1, and tried running on clean data with an empty work directory. Any idea what I might be doing wrong or how I can debug? Thanks for any suggestions.

Here is the command I used:

unset PYTHONPATH
fmriprep --participant_label 01 --nthreads 8 --verbose --output-space template T1w fsnative --use-aroma --fs-license-file 
/gpfs/group/sjw42/default/sjw42_collab/sw/freesurfer-6.0.1/license.txt --work /gpfs/group/sjw42/default/ASH/TEST/work /gpf
s/group/sjw42/default/ASH/TEST/bids /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives participant

And here is the section of errors:

### References

/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;
imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Mapping, Set, Iterable
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:121: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/dataset_description.json' mode='r' encoding='UTF-8'&gt;
self.description = json.load(open(target, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/grabbit/core.py:436: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/config/bids.json' mode='r' encoding='UTF-8'&gt;
domain = json.load(open(domain, 'r'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/task-cardguess_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-01_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/phasediff.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/fmap/sub-01_phasediff.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/workflows/dmri/mrtrix/group_connectivity.py:16: UserWarning: cmp not installed
warnings.warn('cmp not installed')
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-02_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-03_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-04_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-05_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-cardguess_run-06_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/task-rest_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/bids_layout.py:208: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/gpfs/group/sjw42/default/ASH/TEST/bids/sub-01/func/sub-01_task-rest_bold.json' mode='r' encoding='utf-8'&gt;
encoding='utf-8'))
/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/fsl/preprocess.py:1887: UserWarning: This has not been fully tested. Please report any failures.
warn('This has not been fully tested. Please report any failures.')
Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
Captured warning (&lt;class 'DeprecationWarning'&gt;): Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
Captured warning (&lt;class 'ResourceWarning'&gt;): unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;
Captured warning (&lt;class 'ImportWarning'&gt;): can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
/usr/local/miniconda/lib/python3.7/site-packages/nilearn/datasets/neurovault.py:16: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Container
/usr/local/miniconda/lib/python3.7/site-packages/skimage/__init__.py:80: ResourceWarning: unclosed file &lt;_io.TextIOWrapper name='/usr/local/miniconda/lib/python3.7/site-packages/pytest.py' mode='r' encoding='utf-8'&gt;
imp.find_module('pytest')
/usr/local/miniconda/lib/python3.7/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)

oesteban · March 12, 2019, 8:50pm

Hi @Steve_Wilson, your snippet only contains warnings. They are noisy and scary, but I couldn’t find any errors on them. Could you post the full output of your command line?

Steve_Wilson · March 13, 2019, 11:36pm

Thanks, @oesteban . I may have been overinterperting the warnings (I don’t recall getting the before and they are a bit scary, as you mentioned ). I was having an issue with the freesurfer steps not working, but I have figured out that this was related to an issue with the environment. I mistakenly thought it was related to the warning messages. Now that I’ve gotten the environment setup straightened out, it looks like I am only getting the error pasted below. And I only got it for one of two subjects that I tested. I retried the problematic subject separately and wrote to a clean directory to make sure it wasn’t an overwrite issue, but I still got the error. I’m guessing it is not a big problem – so far the results seem fine – but I thought I would pass along the information in case it is useful. Thanks again for your help.

 Could not generate CITATION.html file:

pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle=&quot;fMRIPrep citation boilerplate&quot; /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.md -o /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html

oesteban · March 13, 2019, 11:59pm

Thanks a lot @Steve_Wilson. Could you post the full output log and exact command line for me to trace this problem back?

Steve_Wilson · March 14, 2019, 7:25pm

Sure thing, @oesteban - sorry for the delay. I’ve pasted the exact command and output below. Let me know if either is not what you were looking for or if you need more information.

Here is the set of commands, which were submitted together as a batch job:

set DISPLAY variable

tmpdis1=echo $DISPLAY | awk -F ":" '{print $1}'
tmpdis2=echo $DISPLAY | awk -F ":" '{print $2}'| awk -F '.' '{print $1}'
export DISPLAY=echo $tmpdis1":"$tmpdis2

freesurfer setup

export FREESURFER_HOME=/gpfs/group/sjw42/default/sjw42_collab/sw/freesurfer-6.0.1
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export SUBJECTS_DIR="/gpfs/group/sjw42/default/freesurfer_subjects"
source $FREESURFER_HOME/SetUpFreeSurfer.sh
export FS_LICENSE=$FREESURFER_HOME/license.txt

Go to the correct location

cd $PBS_O_WORKDIR

Run job

unset PYTHONPATH
/usr/bin/singularity run /gpfs/group/sjw42/default/sjw42_collab/sw/singularity/fmriprep/sjw42-fmriprep_icsaci-master-rec.simg --participant_label 05 --nthreads 8 --verbose --work /gpfs/group/sjw42/default/ASH/TEST/work /gpfs/group/sjw42/default/ASH/TEST/bids /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives participant

Here is a link to the output file: https://www.dropbox.com/s/viv47b0a6vf848l/sub-05_fmriprep.pbs.o11460664?dl=0

oesteban · March 14, 2019, 9:27pm

Thanks very much! That is a WARNING too, it seems it worked well. The problem of that particular WARNING is that your reports will not have an html boilerplate under the “Methods” section.

Can I ask you to run the following commands?

unset PYTHONPATH
/usr/bin/singularity exec /gpfs/group/sjw42/default/sjw42_collab/sw/singularity/fmriprep/sjw42-fmriprep_icsaci-master-rec.simg \
    pandoc -s --bibliography /usr/local/miniconda/lib/python3.7/site-packages/fmriprep/data/boilerplate.bib --filter pandoc-citeproc --metadata pagetitle="fMRIPrep citation boilerplate" /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.md -o /gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html

In principle, that should give you an error. I would really appreciate to have the output.

When successful, it should write an HTML file (/gpfs/group/sjw42/default/ASH/TEST/bids/derivatives/fmriprep/logs/CITATION.html).

Steve_Wilson · March 15, 2019, 8:32pm

Okay, so that doesn’t actually give any errors. It produces the html file (or rewrites if it already exists). I took a closer look, and the html file is being created/rewritten as designed for most of my subjects. So far, I am only getting the error when using fmriprep to process data for one specific subject, so there seems to be something idiosyncratic going awry? I’ve double-checked the data, command, etc., and can’t find anything that looks problematic. I’m happy to run any additional testing that would be useful on your end.

oesteban · March 15, 2019, 8:41pm

It might be a one-off filesystem synchronization problem. Please ping us back if suddenly it becomes a problem.

Rubina · July 6, 2021, 1:39pm

Hi,

I’m having a similar issue. I run this code in Anaconda Prompt:

C:\Users\rubin> fmriprep-docker C:\Users\rubin\BIDS C:\Users\rubin\derivatives --longitudinal --participant_label sub-A0123061 sub-A1130051 --fs-license-file C:\Users\rubin\license.txt

The issue, however, is that it doesn’t generate a citation.html file. I only get this (please see below without the brain images):

I’ve been reading some posts related to this. I don’t know if I have to edit the citation (.tex file and what to include) or if there’s another way to resolve this?

Thank you in advance for your help.

Best,

Rubina

Steven · July 6, 2021, 6:05pm

You should not have to edit anything. The image you send looks like it comes from the subject-specific HTML file. The citation.html file would be in OUTPUT_DIRECTORY/logs/. Also at the bottom of subject specific html file (where you got that image), you should be able to see the boilerplate written out as it would be in the citation.html file.

Rubina · July 6, 2021, 7:24pm

Hi Steven,

I do have the fMRIPrep boilerplate (Citation html file) and the summary file (the previous image I shared with the html file but without the boilerplate) so when I scroll down I don’t see brain images.

Also, after I run fMRIPrep, I don’t see a message saying that fMRIPrep was completed successfully so I know I am missing something from my end but I don’t know what it is.