Missing functional files found during post-XCPD operation

Summary of what happened:

Hello,

I am reaching out regarding an issue that I am encountering while I attempt to create parcellations based on output files I have generated. Essentially, I have run a pipeline that:
(a) first conducts fMRIprep processing,
(b) which is then followed by XCPD processing, by using the --skip-parcellation option (as the default parcellation options does not include the “cabp718” parcellation scheme), and
(c) finally followed by a post-XCPD operation that computes three parcellations (schaefer456, gordon333, cabp718) and extracts FD information.

I used this pipeline to process >1000 HCD subjects. However, it looks like some processed subjects (~60) do not have the necessary *dtseries.nii files and which were identified while running the post-XCPD operation. I have pasted part of the function call that handles locating these various *dtseries.nii files. I was wondering if there is a way to fix this error.

Thanks,
Saptarshi

Command used (and if a helper script was used, a link to the helper script or the command generated):

mapfile -t subs < ${sub_list}
count=0
for sub in ${subs[@]} ; do

    echo "### Processing ${sub} ###"

    sub_id=$(echo ${sub} | awk -F"_" '{print $1}')
    
    # check scan pattern
    if [[ " ${run1_subs[*]} " == *" $sub "* ]] ; then        
        dtseries_name=ses-01_task-rest_run-01_space-fsLR_den-91k_desc-denoised_bold.dtseries.nii
        outlier_file_name=ses-01_task-rest_dir-AP_run-01_outliers.tsv
        mtion_file_name=ses-01_task-rest_dir-AP_run-01_motion.tsv
    elif [[ " ${PA_subs[*]} " == *" $sub "* ]] ; then
        dtseries_name=ses-01_task-rest_dir-PA_space-fsLR_den-91k_desc-denoised_bold.dtseries.nii
        outlier_file_name=ses-01_task-rest_dir-PA_outliers.tsv
        mtion_file_name=ses-01_task-rest_dir-PA_motion.tsv
    else
        # typical subs 
        dtseries_name=ses-01_task-rest_space-fsLR_den-91k_desc-denoised_bold.dtseries.nii
        outlier_file_name=ses-01_task-rest_outliers.tsv
        mtion_file_name=ses-01_task-rest_motion.tsv
    fi

    # check if xcpd completed        
    func_dir=${ceph_dir}/${sub}/xcpd/sub-${sub_id}/ses-01/func    
    func_file=${func_dir}/sub-${sub_id}_${dtseries_name}
    outlier_file=${func_dir}/sub-${sub_id}_${outlier_file_name}
    mtion_file=${func_dir}/sub-${sub_id}_${mtion_file_name}    
    
    if [ -f ${func_file} ]; then      
        # Specify output directory
        out_dir=${ceph_dir}/${sub}/post_xcpd
        if [ ! -d ${out_dir} ]; then
            mkdir ${out_dir}
        fi

        # Call function to generate FC matrix
        for p in ${parcels[@]}
        do
            gen_fc_matrix ${p} ${func_dir} ${sub_id} ${out_dir} ${func_file} ${outlier_file}                 
        done  

        # Call function to extract FD column  
        extract_fd_column ${sub} ${mtion_file} ${out_dir}
        
        
        count=$((count+1))
        echo "!!! Processed: $count "

    else
        echo "... No func file!"
    fi    

done

Version:

xcp_d-0.10.6.sif

Environment (Docker, Singularity / Apptainer, custom installation):

Singularity / Apptainer

Data formatted according to a validatable standard? Please provide the output of the validator:

PASTE VALIDATOR OUTPUT HERE

Relevant log outputs (up to 20 lines):

PASTE LOG OUTPUT HERE

Screenshots / relevant information:


Hi @ssinha,

You can add custom parcellations to XCPD: Running XCP-D — xcp_d 0.11.1.dev57+g73edd86 documentation

May you please provide commands and error logs for fmriprep and XCPD?

Best,
Steven

Hi @ssinha ,

It’s possible that subjects that were missing files post XCPD did not pass your —min-time threshold.

Best,
Steven

Thank you for sharing this information, Steven.

I will try some trial runs with a higher value for the --min-time option. In the current code, the value is set to 240, which I just saw as being the default value anyway. Would you recommend any particular higher value in this regard?

I should also note that the --fd-thresh option in the code is currently set to 0.3.

Hi @ssinha

There’s no gold standard for this parameter, and it depends on your analysis goals and sample. I don’t know enough about your dataset or goals to provide an explicit recommendation.

Best,
Steven