XCP-D crashes, missing font

Software Support
1

Summary of what happened:

When running XCP-D v. 0.10.5 (then reproduced with v. 0.11.1), various nodes (plot_design_matrix and censor report) fail with the following message:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 45, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 525, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 643, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 769, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node plot_design_matrix.

Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 401, in run
	    runtime = self._run_interface(runtime)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/interfaces/utility/wrappers.py", line 139, in _run_interface
	    out = function_handle(**args)
	  File "<string>", line 43, in plot_design_matrix
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nilearn/_utils/helpers.py", line 108, in wrapper
	    return func(*args, **kwargs)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nilearn/plotting/matrix/matrix_plotting.py", line 400, in plot_design_matrix
	    return save_figure_if_needed(axes, output_file)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/nilearn/plotting/_utils.py", line 50, in save_figure_if_needed
	    fig.savefig(output_file)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/figure.py", line 3490, in savefig
	    self.canvas.print_figure(fname, **kwargs)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/backend_bases.py", line 2155, in print_figure
	    self.figure.draw(renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/artist.py", line 94, in draw_wrapper
	    result = draw(artist, renderer, *args, **kwargs)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/artist.py", line 71, in draw_wrapper
	    return draw(artist, renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/figure.py", line 3257, in draw
	    mimage._draw_list_compositing_images(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/image.py", line 134, in _draw_list_compositing_images
	    a.draw(renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/artist.py", line 71, in draw_wrapper
	    return draw(artist, renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/axes/_base.py", line 3181, in draw
	    mimage._draw_list_compositing_images(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/image.py", line 159, in _draw_list_compositing_images
	    a.draw(renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/artist.py", line 71, in draw_wrapper
	    return draw(artist, renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/axis.py", line 1416, in draw
	    tlb1, tlb2 = self._get_ticklabel_bboxes(ticks_to_draw, renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/axis.py", line 1345, in _get_ticklabel_bboxes
	    [tick.label2.get_window_extent(renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/axis.py", line 1345, in <listcomp>
	    [tick.label2.get_window_extent(renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/text.py", line 969, in get_window_extent
	    bbox, info, descent = self._get_layout(self._renderer)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/text.py", line 373, in _get_layout
	    _, lp_h, lp_d = _get_text_metrics_with_cache(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/text.py", line 69, in _get_text_metrics_with_cache
	    return _get_text_metrics_with_cache_impl(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/text.py", line 77, in _get_text_metrics_with_cache_impl
	    return renderer_ref().get_text_width_height_descent(text, fontprop, ismath)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/backends/backend_svg.py", line 1299, in get_text_width_height_descent
	    return self._text2path.get_text_width_height_descent(s, prop, ismath)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/textpath.py", line 63, in get_text_width_height_descent
	    font = self._get_font(prop)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/textpath.py", line 34, in _get_font
	    filenames = _fontManager._find_fonts_by_props(prop)
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/font_manager.py", line 1441, in _find_fonts_by_props
	    self.findfont(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/font_manager.py", line 1352, in findfont
	    ret = self._findfont_cached(
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/font_manager.py", line 1503, in _findfont_cached
	    return self.findfont(default_prop, fontext, directory,
	  File "/usr/local/miniconda/lib/python3.10/site-packages/matplotlib/font_manager.py", line 1356, in findfont
	    raise ret.to_exception()
	ValueError: Failed to find font DejaVu Sans:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0, and fallback to the default font was disabled

Command used (and if a helper script was used, a link to the helper script or the command generated):

apptainer run \
    -B ${fmri_dir}:/fmri_dir/:ro                    \
    -B ${output_dir}:/output/                       \
    -B "${wkdir}":/wkdir/                           \
    -B /tmp:/scrth                                  \
    ${xcpd}                                         \
        /fmri_dir/ /output/ participant             \
        --work-dir              "/wkdir/"       \
        --clean-workdir                         \
        --participant-label     "${sub}"        \
        --session-id            01              \
        --task-id               rest            \
        --mode                  none            \
        --file-format           nifti           \
        --input-type            fmriprep        \
        --nuisance-regressors   '27P'           \
        --motion-filter-type    lp              \
            --band-stop-min     6               \
        --despike               y               \
        --fd-thresh             0.2             \
        --min-coverage          0.5             \
        --min-time              60              \
        --combine-runs          y               \
        --output-type           censored        \
        --warp-surfaces-native2std n            \
        --atlases               Glasser  Gordon \
        --smoothing             6               \
        --abcc-qc n --linc-qc n                 \
        --nprocs                27              \
        --debug all -vvv --write-graph          | \
        tee -a "${logfile}"

Version:

0.10.5 and 0.11.1

Environment (Docker, Singularity / Apptainer, custom installation):

Apptainer

Data formatted according to a validatable standard? Please provide the output of the validator:

Input is fMRIPREP v. 25.0.0 output.

Relevant log outputs (up to 20 lines):

250820-14:14:24,174 nipype.workflow DEBUG:
	 Resolving paths in outputs loaded from results file.
250820-14:14:24,174 nipype.workflow DEBUG:
	 output: denoised_interpolated_bold
250820-14:14:24,174 nipype.workflow DEBUG:
	 [Node] censor_interpolated_data - setting input in_file = /wkdir/xcp_d_0_0_wf/sub_CLB00005_wf/postprocess_0_wf/denoise_bold_wf/regress_and_filter_bold/filtered_denoised.nii.gz
250820-14:14:24,174 nipype.utils DEBUG:
	 Loading pkl: /wkdir/xcp_d_0_0_wf/sub_CLB00005_wf/postprocess_0_wf/prepare_confounds_wf/process_motion/result_process_motion.pklz
250820-14:14:24,195 nipype.workflow DEBUG:
	 Resolving paths in outputs loaded from results file.
250820-14:14:24,196 nipype.workflow DEBUG:
	 output: temporal_mask
250820-14:14:24,196 nipype.workflow DEBUG:
	 [Node] censor_interpolated_data - setting input temporal_mask = /wkdir/xcp_d_0_0_wf/sub_CLB00005_wf/postprocess_0_wf/prepare_confounds_wf/process_motion/desc-fd_outliers.tsv
250820-14:14:24,197 nipype.utils DEBUG:
	 Removing contents of /wkdir/xcp_d_0_0_wf/sub_CLB00005_wf/postprocess_0_wf/denoise_bold_wf/censor_interpolated_data
250820-14:14:24,200 nipype.workflow DEBUG:
	 [Node] Writing pre-exec report to "/wkdir/xcp_d_0_0_wf/sub_CLB00005_wf/postprocess_0_wf/denoise_bold_wf/censor_interpolated_data/_report/report.rst"
250820-14:14:24,202 nipype.workflow INFO:
	 [Node] Executing "censor_interpolated_data" <xcp_d.interfaces.censoring.Censor>

I have verified that DejaVu Sans does seem to be installed on the computer (Ubuntu 22.04.5 LTS).
I have not tried binding host font directories to the container.

2

Hi @Trevor_Day,

Unrelated but please use -e or --containall when running containerized pipelines to make sure you are not bringing in unwanted local variables.

This would probably be the next move, and also defining export APPTAINERENV_FONTCONFIG_PATH= and export APPTAINERENV_XDG_DATA_DIRS= to your font library (usually something like /usr/share/). Keep in mind if you rename the font directory when you mount it, that should be reflected in the export commands as well.

Best,
Steven

1 Like
3

Thanks Steven.

Just to clarify, add the export commands to the script invoking XCP-D, using one of these:

$ whereis fonts
fonts: /etc/fonts /usr/share/fonts

And also -B /usr/share/fonts:/usr/share/fonts?

4

Yah, and when you are mounting a drive and not renaming it, you don’t need the : and stuff that come after it.

5

Yeah, but I usually write it out for consistency/legibility.

Hmm, so if I add -C / --containall, XCP-D has trouble accessing the templates in ~/.cache/, but binding /usr/share/fonts doesn’t work without -C. Even with perms set to o=rwx.

Using /etc/fonts/ doesn’t help. Am I missing something about using the templates with -C?

6

Hi @Trevor_Day,

Yes, containall means even your home directory isn’t mounted. If you want to mount something from home (e.g., templateflow) you will have to explicitly mount it.

Best,
Steven

7

Thanks @Steven. I see what the help for -C means now, I was understanding it backward, I think. I bound my home directory (which I’m not sure the need to do that is mentioned in the XCP-D docs?), and the font directory, and additionally passed the variables you suggested with --env, and still no dice.

fonts="/usr/share/fonts/"

export APPTAINERENV_FONTCONFIG_PATH=${fonts}
export APPTAINERENV_XDG_DATA_DIRS=${fonts}

apptainer run -C \
    -B ${fmri_dir}:/fmri_dir/:ro                    \
    -B ${output_dir}:/output/                       \
    -B "${wkdir}":/wkdir/                           \
    -B ${fonts}:${fonts}                            \
    -B /home/tkmday/:/home/tkmday/                  \
    --env APPTAINERENV_FONTCONFIG_PATH=${fonts}     \
    --env APPTAINERENV_XDG_DATA_DIRS=${fonts}       \
    ${xcpd} ... (same as before)

I’m a little extra baffled because it worked fine with one participant last week, then started throwing this error when I moved on to participant #2 this week.

8

Hi @Trevor_Day,

I think the XDG_DATA_DIR argument should just be /usr/share, without the /fonts directory. You can confirm by seeing what it is on your cluster.

Also, you can try appending it to the path rather than fully replacing it. For example:

export APPTAINERENV_FONTCONFIG_PATH="${fonts}:$FONTCONFIG_PATH"
export APPTAINERENV_XDG_DATA_DIRS="${fonts}:$XDG_DATA_DIRS"

It is not necessary. That’s just where templateflow might be for people. In either case, templateflow can be separately mounted.

This was done incorrectly. When you are using the --env flag you do not need the APPTAINERENV_ prefix.

Best,
Steven

9

I bound my home directory to address the templateflow thing - not to address the font issues, but just documenting all the changes.

I’m still not getting it to work. It seemed like export APPTAINERENV_* and --env was redundant, but maybe I misunderstood?

I’m still struggling to resolve the error, most recently:

export APPTAINERENV_FONTCONFIG_PATH=/usr/share/fonts/
export APPTAINERENV_XDG_DATA_DIRS=/usr/share/:${XDG_DATA_DIRS}

apptainer run -C \
    -B ${fmri_dir}:/fmri_dir/:ro                    \
    -B ${output_dir}:/output/                       \
    -B "${wkdir}":/wkdir/                           \
    -B /usr/share/                                  \
    -B /home/tkmday/:/home/tkmday/                  \

Could you maybe show me how you’d edit these lines?

10

If you apptainer shell -e into the container, what are those environment variables listed as by default, and what are contained in them?

11 

Apptainer> printenv
SHELL=/bin/bash
PYTHONNOUSERSITE=1
SINGULARITY_NAME=xcpd-ver0.11.1.simg
AFNI_IMSAVE_WARNINGS=NO
OS=Linux
SINGULARITY_ENVIRONMENT=/.singularity.d/env/91-environment.sh
PWD=/home/tkmday/pathfinders/brainlat
APPTAINER_ENVIRONMENT=/.singularity.d/env/91-environment.sh
APPTAINER_APPNAME=
HOME=/home/tkmday
LANG=C.UTF-8
APPTAINER_COMMAND=shell
FSLLOCKDIR=
SINGULARITY_CONTAINER=/home/tkmday/trevor/xcpd-ver0.11.1.simg
FSLDIR=/opt/fsl
C3DPATH=/opt/c3d/bin
APPTAINER_CONTAINER=/home/tkmday/trevor/xcpd-ver0.11.1.simg
TERM=xterm-256color
FIX_VERTEX_AREA=
SHLVL=1
ANTSPATH=/usr/lib/ants
APPTAINER_NAME=xcpd-ver0.11.1.simg
SINGULARITY_BIND=
FSLMULTIFILEQUIT=TRUE
APPTAINER_BIND=
LD_LIBRARY_PATH=/opt/fsl/lib::/.singularity.d/libs
FSLREMOTECALL=
PS1=Apptainer> 
OMP_NUM_THREADS=1
LC_ALL=C.UTF-8
FSLMACHINELIST=
FSLGECUDAQ=cuda.q
PATH=/usr/lib/ants/ants-2.5.3/bin:/opt/fsl/lib:/opt/fsl/bin:/opt/c3d/bin:/usr/local/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/afni-latest
FSL_DEPS=libquadmath0;libnewimage.so;libmiscmaths.so
MKL_NUM_THREADS=1
CPATH=/usr/local/miniconda/include:
FSLOUTPUTTYPE=NIFTI_GZ
AFNI_INSTALLDIR=/opt/afni-latest
_=/usr/bin/printenv
12

Do you see anything in paths where fonts might be found? Is there a similar whereis fonts output?

13

The two variables we’ve been manipluating were unset, and

whereis fonts
fonts: /etc/fonts /usr/share/fonts
14

Do you see dejavu font in either of those directories in the container?

15

Yes

Apptainer> ls /etc/fonts/conf.avail/
10-antialias.conf	    10-sub-pixel-rgb.conf		       20-unhint-small-dejavu-lgc-serif.conf  45-latin.conf		    58-dejavu-lgc-sans.conf   70-no-bitmaps.conf
10-autohint.conf	    10-sub-pixel-vbgr.conf		       20-unhint-small-dejavu-sans-mono.conf  49-sansserif.conf		    58-dejavu-lgc-serif.conf  70-yes-bitmaps.conf
10-hinting-full.conf	    10-sub-pixel-vrgb.conf		       20-unhint-small-dejavu-sans.conf       50-user.conf		    60-generic.conf	      80-delicious.conf
10-hinting-medium.conf	    10-unhinted.conf			       20-unhint-small-dejavu-serif.conf      51-local.conf		    60-latin.conf	      90-synthetic.conf
10-hinting-none.conf	    11-lcdfilter-default.conf		       20-unhint-small-vera.conf	      53-monospace-lcd-filter.conf  65-fonts-persian.conf
10-hinting-slight.conf	    11-lcdfilter-legacy.conf		       25-unhint-nonlatin.conf		      57-dejavu-sans-mono.conf	    65-khmer.conf
10-no-sub-pixel.conf	    11-lcdfilter-light.conf		       30-metric-aliases.conf		      57-dejavu-sans.conf	    65-nonlatin.conf
10-scale-bitmap-fonts.conf  20-unhint-small-dejavu-lgc-sans-mono.conf  40-nonlatin.conf			      57-dejavu-serif.conf	    69-unifont.conf
10-sub-pixel-bgr.conf	    20-unhint-small-dejavu-lgc-sans.conf       45-generic.conf			      58-dejavu-lgc-sans-mono.conf  70-force-bitmaps.conf
Apptainer> ls /usr/share/fonts/
.uuid     X11/      truetype/ 
Apptainer> ls /usr/share/fonts/truetype/
dejavu
Apptainer>
16

To clarify, you are getting the exact same error across all of these testing iterations?

17

Yep, all end up with:

ValueError: Failed to find font DejaVu Sans:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0, and fallback to the default font was disabled
18

Huh strange, I am unable to replicate this error. Is it only with this container? What if you try rebuilding or rolling back / forward versions as a test?

What if you try

export APPTAINERENV_FONTCONFIG_PATH=/etc/fonts # FIRST CONFIRM THAT fonts.conf EXISTS THERE
export APPTAINERENV_XDG_DATA_DIRS=/usr/share/:${XDG_DATA_DIRS}

apptainer run -C \
    -B ${fmri_dir}:/fmri_dir/:ro                    \
    -B ${output_dir}:/output/                       \
    -B "${wkdir}":/wkdir/                           \
    -B /etc/fonts:/etc/fonts:ro \
    -B /usr/share/fonts:/usr/share/fonts:ro \
    -B /home/tkmday/:/home/tkmday/                  \
19

I first noticed with ver. 0.10.5 so I rebuilt the current stable version, 0.11.1.

No dice with those lines. And fonts.conf exists in /etc/fonts.

I hadn’t previously encountered this issue with the container, but I’m not really sure what would have changed?

20

Huh, sorry I have no idea how to further troubleshoot. @tsalo thoughts?