I am reaching out regarding a question involving the “framewise_displacement” values (that is available as one of the columns generated by XCP-D in the *_motion.tsv files) and the entries of the *outliers.tsv files (also generated by XCP-D).
I have been running some experiments, with an FD threshold of 0.3 for our subjects, and have been observing inconsistency in the results. In my experiments, I am calculating:
---- the number of low-motion frames (counted from the above “framewise_displacement” column by selecting those entries which are <= 3), and
---- the number of low-motion frames (counted from the above *outliers.tsv file by selecting those entries that are 0s).
These two values should be the same as they are referring to the same aspect. However, in my experiments, I am noticing significant differences between these two values. I have attached a spreadsheet that highlights these differences (in the last column) across a sample set of subjects.
I wanted to find out if I am misinterpreting these two values and if there are any other factors that go into the process of outlier detection.
Thank you,
Saptarshi
Command used (and if a helper script was used, a link to the helper script or the command generated):
#!/bin/bash
# Define the directory to process
DIRECTORY="/ceph/chpc/shared/deanna_barch_group/sinha.s/HCD_typicalMOD2"
# Define the output TSV file
OUTPUT_TSV="/ceph/chpc/shared/deanna_barch_group/sinha.s/FDanalysis_typicalMOD2.tsv"
# Create or clear the output file
> "$OUTPUT_TSV"
# Add header to the TSV file (optional)
echo -e "SUB-id\tMeanFD-LowMotionFrames(FD.tsv)\t#LowMotionFrames(FD.tsv)\t#LowMotionFrames(*outliers.tsv)\tDiff=Col3-Col4" > "$OUTPUT_TSV"
# Loop through each folder in the specified directory
for sub in "$DIRECTORY"/*; do
cd ${sub}/post_xcpd/
# Extract SUB-id
subid=$(basename "$sub")
# Find number of low-motion frames
number_lmf=$(awk '{ if ($1 <= 0.3) { total += $1; count++ } } END { if (count > 0) print count}' FD.tsv)
# Find average FD values of low-motion frames
avg_lmf=$(awk '{ if ($1 <= 0.3) { total += $1; count++ } } END { if (count > 0) print total/count}' FD.tsv)
# Find number of low-motion frames from *outliers.tsv file
sub_name="${subid:0:10}"
cd ../xcpd/sub-${sub_name}/ses-01/func/
outlier_lmf=$(tr -cd '0' < sub-${sub_name}_ses-01_task-rest_outliers.tsv | wc -m)
# Compute difference between Col3 & Col4
diff_col3col4=$((number_lmf-outlier_lmf))
cd /ceph/chpc/shared/deanna_barch_group/sinha.s
# Write the information to the TSV file, separated by tabs
echo -e "$subid\t$avg_lmf\t$number_lmf\t$outlier_lmf\t$diff_col3col4" >> "$OUTPUT_TSV"
done
echo "TSV file '$OUTPUT_TSV' created successfully."
In the future, please use the Software Support post category and template to help organize your post and provide us with information to help us debug. You can see I have edited your post accordingly, and you can edit to add missing information.
Thanks for sharing this information. I was unsure about the category as I am not seeking a software support. I am trying to understand the reason behind the discrepancies I am getting. I have now edited the post and have provided some of the additional information available to me.
I will try the 0.12.0 version of XCP-D and get back to this post. Since I am also running fMRIprep-23.1.4 on our subjects (prior to running XCP-D), it may take me some time but I will reply here once I have the results.
Did you apply motion filtering? The outliers file might be using filtered motion confounds, and you might separately be counting based on unfiltered FD.
UPDATE: I wanted to follow-up and let you know that I completed running XCP-D with the latest version (xcp_d-0.12.0) and as a summary, the discrepancies widened. The output from the *outliers.tsv file remained the same. However, the output from the FD.tsv file decreased across all the 10 sample subjects, which contributed to the widened discrepancy. I have attached a screenshot for reference. The ones in red correspond to xcp_d-0.10.6 and those in blue correspond to xcp_d-0.12.0.
Our PI reached out to the UPenn folks over the weekend and Taylor is currently assisting us, but just thought about posting the update here for everyone’s reference.
I used a script developed by one of our collaborators for this purpose. This particular task is handled by a function call which extracts the framewise_displacement column (instead of the framewise_displacement_filtered column) from the concatenated *motion.tsv file. This could be a reason for the discrepancy and we are currently working on this.
UPDATE: I wanted to follow-up and share that I finished running a new trial. This time I generated the FD.tsv file using the framewise_displacement_filtered column, instead of the framewise_displacement column, and I now have matching values. I should also note that I could not find this filtered column in the previous XCP-D version (xcp_d-0.10.6) but the recent outputs (created using xcp_d-0.12.0) do have this column. Feel free to mark this as resolved.