Duplicate T1w image data across sub-NDAREF662XZT and sub-NDARVN370YL8 in the RBC HBN release

, ,
1

Posting here at the suggestion of @tientong, who asked me to move this over from
ReproBrainChart/HBN_BIDS#2.

Two subjects in the HBN release have identical T1w image data but conflicting demographics.

sub-NDAREF662XZT sub-NDARVN370YL8
session ses-HBNsiteCBIC ses-HBNsiteCBIC
acq acq-HCP acq-HCP
age 18.59 8.91
race White Asian
handedness Left Right

The two volumes have identical voxel arrays (max absolute difference 0), identical NIfTI headers and identical affines. The .nii.gz files differ only in gzip framing.

The json sidecars are byte identical, including fields recorded per acquisition:

  • ShimSetting: [189, -10385, -5860, 89, -9, -50, -19, -1]
  • SAR: 0.066642
  • TxRefAmp: 214.489
  • ImageOrientationPatientDICOM
  • VoxelSizeDim1: 0.8000012040138245

Another CBIC acq-HCP subject (sub-NDARAM197GFY) differs on all of these, so these values are not constant across the site. Shim currents are set per session, so this looks like one acquisition written out under two GUIDs.

Verified on files pulled from the fcp-indi remote via git annex get, so it is present in the released data. Both sidecars date to commit e296738.

I hashed the T1w volumes for 1000 subjects and this was the only duplicate pair among them. I have not checked the full release.

Anyone training a model on this data gets the same brain in both train and test, and the ages differ by 9.7 years so at least one label is wrong. We are excluding both subjects since there is no way to tell which GUID owns the scan.

Happy to share the hashing script if it would help check the full release.

1 Like
2

I have downloaded the HBN_FreeSurfer back in december 2026. Just checked that the cortical thickness data for both of these subjects are vastly different. I suppose you can use the001.mgz files in …/mri/orig/as replacements ?

3

Thanks for checking. I pulled both freesurfer.tar.xz archives from the fcp-indi remote and compared mri/orig/001.mgz directly: identical voxel arrays (max absolute difference 0, correlation 1.0), identical affines, same 224x320x320 shape. The two files differ by one byte, which is the filename embedded in the mgz header.

So 001.mgz is the same image for both subjects and can’t serve as a replacement. The duplication is already present at the FreeSurfer input stage.

On the cortical thickness differing: in my copy they’re within about 1%. Mean aparc thickness is 2.788 vs 2.751 and total gray volume 574479 vs 576959, which is consistent with run to run variation in recon-all on the same input rather than two different brains. The archives differ in size by 15MB, but that’s surface files and transforms, not anatomy.

Which metric were you comparing? If you’re seeing a large difference it may be a different release than the one I have.

4

i was looking at the cortical thickness resampled to the fsaverage5 space. the mean vertex-wise thickness for both subjects are 2.439448 2.417708. the correlation between both subject’s vertex-wise CT is 0.95.

i guess you’re right

I did a pairwise correlation across every unique pair of subjects in the dataset. there was only one other pair of subjects whose correlation is higher ; r=0.98

can you check the T1w data for sub-NDARVV704MND sub-NDARYC287UFV as well? thanks

1 Like
5

Thanks for flagging it, confirmed that’s a second duplicate. sub-NDARVV704MND and sub-NDARYC287UFV (both CBIC, acq-VNavNorm) have identical voxel arrays, identical NIfTI headers and identical affines. Demographics conflict the same way as the first pair: ages 8.37 and 6.03, BMI 15.2 and 20.8, race White and Black.

One thing worth noting, the files are not byte identical. They differ by exactly 157 bytes, all inside an AFNI extension block, and the only difference is the self_idcode AFNI assigns per file. So a file level md5 misses this pair while a hash over the decoded voxel array catches it. Both come out as 658efe8001b4d5dc794dc44641e873ea on the voxels.

My earlier scan did hash voxels, but only across the 1000 subject cohort we use. NDARYC287UFV is in it and NDARVV704MND is not, so the hash appeared once and no duplicate group formed. It was scoped to our subset rather than the release.

Also this pair carries an AFNI extension while NDAREF662XZT and NDARVN370YL8 do not, so the two cases may have come about differently.

I can run the voxel hash across the full release if that helps. Our two approaches are complementary, hashing is cheap but only finds exact matches while your correlation picks up near duplicates.

6

I am running the correlation analyses on a N=1379 cohort. Here are the 10 largest inter-subject correlations; the rest of the inter-subject correlations are all <.85

               subj1            subj2         r
1   sub-NDARVV704MND sub-NDARYC287UFV 0.9813238
2   sub-NDAREF662XZT sub-NDARVN370YL8 0.9520936
3   sub-NDARDR591AUC sub-NDARXU253JHC 0.8487429
4   sub-NDARWG831JJ8 sub-NDARZN610GTY 0.8472385
5   sub-NDARTE785ZMJ sub-NDARXU253JHC 0.8471779
6   sub-NDARDR591AUC sub-NDARZN610GTY 0.8467223
7   sub-NDARMW178UDD sub-NDARXU253JHC 0.8453929
8   sub-NDARWN709HA2 sub-NDARZN610GTY 0.8452211
9   sub-NDARTP022VL6 sub-NDARZN610GTY 0.8451568
10  sub-NDARJG740PM8 sub-NDARKD064EVC 0.8438943
7

Thanks, checked all 10. Only the top two are duplicates, both with byte identical voxel arrays:

sub-NDARVV704MND / sub-NDARYC287UFV, ages 8.37 and 6.03, race White and Black
sub-NDAREF662XZT / sub-NDARVN370YL8, ages 18.59 and 8.91, race White and Asian

Ranks 3 to 10 are all distinct images, voxel correlations 0.42 to 0.76 with differing affines. Both duplicate pairs are CBIC, all eight others are RU.

As I said earlier, my scan only covered the 1000 subject cohort we use. Here’s the script if you want to run it over your full N=1379, takes a few minutes and needs the annex content fetched:

import glob, os, hashlib, numpy as np, nibabel as nib, pandas as pd

ROOT = "/path/to/HBN_BIDS"

rows = []
for d in sorted(glob.glob(f"{ROOT}/sub-*")):
    s = os.path.basename(d)[4:]
    p = f"{d}/ses-*/anat/*T1w.nii"
    hits = sorted(glob.glob(p + ".gz")) or sorted(glob.glob(p))
    if not hits or not os.path.exists(os.path.realpath(hits[0])):
        continue
    d_arr = np.ascontiguousarray(np.asanyarray(nib.load(hits[0]).dataobj))
    rows.append((s, hashlib.md5(d_arr.tobytes()).hexdigest()))

df = pd.DataFrame(rows, columns=["sub", "voxel_md5"])
dups = df[df.duplicated("voxel_md5", keep=False)].sort_values("voxel_md5")
print(f"hashed {len(df)}, {dups.voxel_md5.nunique()} duplicate groups")
print(dups.to_string(index=False))

It hashes the decoded voxel array rather than the file, which matters here: neither pair is byte identical as a file. The first differs in gzip framing and the second in an AFNI extension block, so md5sum on the files finds nothing. File size doesn’t work either, the uncompressed .nii files are all the same size for a given matrix.

8

Hi @rlbachen ,

Thank you very much for flagging this issue! We’ll work with the HBN researchers to look into this and will follow up soon.

Best,

Tien

9

Thanks Tien, glad it’s being looked into. Note there are two pairs: sub-NDAREF662XZT / sub-NDARVN370YL8 and sub-NDARVV704MND / sub-NDARYC287UFV, both from CBIC. Happy to send the hashes or the script if that helps.