Summary of what happened:
I have been using an XCP-D pipeline for processing HCD subjects for sometime without any issues. Lately, I switched to a different line of subjects (ECDMT) to process these using the same XCP-D pipeline I used earlier successfully. Unfortunately, despite following multiple troubleshooting steps, I keep getting a same error message where it says it cannot locate the participant summary report; that is, the xcpd/sub-${sub_id}.html file is missing.
The output log does say that XCP-D finished successfully!. I am also seeing most of the output files have been generated in the respective XCP-D folder (though I have not checked these thoroughly). However, due to the missing participant summary report, I keep getting the error message. I wanted to find out if there is a way to troubleshoot this error, such that the missing *.html file in question can be generated by XCP-D, or if I can ignore this summary report and use the output files that were created, despite the error message, though I believe this may not be a good approach.
Command used (and if a helper script was used, a link to the helper script or the command generated):
# set dirs
wk_dir=$1 # location of data & outputs
sub=$2 # format of HCDXXXXXX_VX_MR
log_dir=/scratch/sinha.s/ECDMT_xcpd_processing/logs/
sub_log=${wk_dir}/${sub}/proc_xcpd_job_log
code_dir=$3 # location of scripts & container
subdirs=( A B ) # two separate acquisition directions
xcpd_container=${code_dir}/xcp_d-0.12.0.sif
############################################
################ START PROC ################
############################################
echo "Processing $sub "
#if [ ! -d ${wk_dir}/${sub} ]; then
# mkdir -p ${wk_dir}/${sub}
#fi
time=$(date "+%Y-%m-%d %H:%M:%S")
echo "## Proc Job Starts for $sub @ ${time}" >> ${sub_log}
# sub_num=`echo "$i" | sed 's/[^0-9]//g'`
#unproc_dir=${ceph_dir}/${sub}/unprocessed
# BIDS structure
sub_id=$(echo "${sub}" | awk -F"_" '{print $1}')
#sub_dir=${wk_dir}/${sub}/BIDS/sub-${sub_id}/ses-01/
#############################################
################### XCP-D ###################
#############################################
# create a dummy file to indicate completion of job
xcpd_job=${wk_dir}/${sub}/xcpd_finished
## Only execute if fMRIPrep succeeded!!
#if [ -f ${fprep_job} ]; then
#echo ".. XCPD Proc" >> ${sub_log}
# set dirs
sub_dir=$wk_dir/${sub}
opt_dir=${sub_dir}/xcpd
if [ ! -d ${opt_dir} ]; then
mkdir ${opt_dir}
fi
apptainer run --cleanenv \
-e -B ${sub_dir} \
-B ${sub_dir}/derivatives \
-B ${opt_dir} \
${xcpd_container} \
${sub_dir}/derivatives \
${opt_dir} \
participant \
--work-dir ${opt_dir} \
--smoothing 2 \
--dummy-scans 6 \
--random-seed 0 \
--bpf-order 2 \
--despike \
--lower-bpf 0.01 \
--upper-bpf 0.08 \
--min-time 1 \
-p 36P \
--motion-filter-type notch \
--band-stop-min 15 \
--band-stop-max 25 \
--motion-filter-order 4 \
--head-radius auto \
--file-format cifti \
--warp-surfaces-native2std \
--mode abcd \
--abcc-qc y \
--create-matrices all \
--skip-parcellation \
--fd-thresh 0.3 \
--nthreads 16 \
--omp-nthreads 8 \
--mem-mb 30000
#search output html file for the code block that indicates no errors were found
if grep -Pzoq '<div[^>]*id="errors"[^>]*>[\s\S]*?<p[^>]*>No errors to report!<\/p>[\s\S]*?<\/div>' \
"${sub_dir}/xcpd/sub-${sub_id}.html" ; then
echo "XCPD: NO error found!"
echo ".. .. completed!" >> ${sub_log}
echo "" > ${xcpd_job}
else
echo "XCPD: error found!"
echo ".. .. failed!" >> ${sub_log}
echo ${sub_id} >> ${log_dir}/xcpd_failed_subs
fi
Version:
xcp_d-0.12.0
Environment (Docker, Singularity / Apptainer, custom installation):
Apptainer (file format used for the image: *.sif)
Data formatted according to a validatable standard? Please provide the output of the validator:
[environment]
cpu_count = 32
exec_env = "posix"
free_mem = 331.4
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.10.0"
templateflow_version = "25.0.3"
version = "0.12.0"
[execution]
fmri_dir = "/scratch/sinha.s/sampleECDMT/ECDMT10002/derivatives"
aggr_ses_reports = 4
bids_database_dir = "/scratch/sinha.s/sampleECDMT/ECDMT10002/xcpd/20260416-175930_1e0a12d9-0afe-4031-a391-3cba6573bcda/bids_db"
bids_description_hash = "fe1ec2b2fd19361c74d61a741636115cd51d6972c0e2a83c7de0b6bdfce07662"
boilerplate_only = false
confounds_config = "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/data/nuisance/36P.yml"
debug = []
layout = "BIDS Layout: ...leECDMT/ECDMT10002/derivatives | Subjects: 1 | Sessions: 1 | Runs: 1"
log_dir = "/scratch/sinha.s/sampleECDMT/ECDMT10002/xcpd/logs"
log_level = 25
low_mem = false
md_only_boilerplate = false
notrack = false
output_layout = "auto"
parameters_hash = "f0a0ec6b"
report_output_level = "session"
reports_only = false
output_dir = "/scratch/sinha.s/sampleECDMT/ECDMT10002/xcpd"
atlases = []
run_uuid = "20260416-175930_1e0a12d9-0afe-4031-a391-3cba6573bcda"
participant_label = [ "ECDMT10002",]
processing_list = [ "ECDMT10002:1:1",]
templateflow_home = "/home/sinha.s/.cache/templateflow"
work_dir = "/scratch/sinha.s/sampleECDMT/ECDMT10002/xcpd"
write_graph = false
[workflow]
mode = "abcd"
file_format = "cifti"
dummy_scans = 6
input_type = "fmriprep"
despike = true
smoothing = 2.0
output_interpolated = true
combine_runs = true
motion_filter_type = "notch"
band_stop_min = 15.0
band_stop_max = 25.0
motion_filter_order = 4
head_radius = "auto"
fd_thresh = 0.3
min_time = 1.0
bandpass_filter = true
high_pass = 0.01
low_pass = 0.08
bpf_order = 2
min_coverage = 0.5
correlation_lengths = [ "all",]
process_surfaces = true
abcc_qc = true
linc_qc = true
[nipype]
crashfile_format = "txt"
get_linked_libs = false
memory_gb = 30.0
nprocs = 16
omp_nthreads = 8
plugin = "MultiProc"
resource_monitor = false
stop_on_first_crash = false
[seeds]
master = 14660
[execution.datasets]
[execution.dataset_links]
preprocessed = "/scratch/sinha.s/sampleECDMT/ECDMT10002/derivatives"
templateflow = "/home/sinha.s/.cache/templateflow"
[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false
Relevant log outputs (up to 20 lines):
#last few lines of the "proc_ECDMT10002_6550204.out" file
260416-18:11:39,479 nipype.workflow IMPORTANT:
XCP-D finished successfully!
260416-18:11:39,491 nipype.workflow IMPORTANT:
Works derived from this XCP-D execution should include the boilerplate text found in /scratch/sinha.s/sampleECDMT/ECDMT10002/xcpd/logs/CITATION.md.
XCPD: error found!
--------------------------------------------------------------
Begin Slurm Epilogue Thu Apr 16 18:11:53 CDT 2026 1776381113
End Slurm Epilogue Thu Apr 16 18:11:53 CDT 2026 1776381113
--------------------------------------------------------------
#last few lines of the "proc_ECDMT10002_6550204.err" file
++ Final vox neighborhood: ellipsoid with Nneigh=27 and radii (1.900,1.900,1.900).
++ ReHo (Kendall's W) calculated.
++ Final vox neighborhood: ellipsoid with Nneigh=27 and radii (1.900,1.900,1.900).
++ ReHo (Kendall's W) calculated.
grep: /scratch/sinha.s/sampleECDMT//ECDMT10002/xcpd/sub-ECDMT10002.html: No such file or directory