HBN RBC Datalad instance too large to clone on HPC with inode limits

, ,
1

First, thanks for sharing this preprocessed resource.

I am having trouble using it, as the datalad instance contains too many files to clone. The HPC system I have access to (DRAC) has inode limits. The scratch space limit is 1M files. I would like to raise the issue and hope to find a solution. I would really prefer to use the released resource rather than reprocessing data and not fully use RBC to it’s full potential.

I know @yarikoptic has also opened an issue on an older release (see: https://github.com/ReproBrainChart/HBN_CPAC/issues/2) raising concerns that the dataset is hard to use.

Are there any plans for how to move forward?

2

Would a versioned per-subject export be a workable interim option while the repository layout is being discussed? Keeping the release commit and file checksums alongside it would make the subset traceable. One distinction worth checking is where the inodes are being used: Git’s sparse checkout reduces the working tree, but does not itself shrink the repository history. If the metadata alone hits the limit, a smaller checkout would still leave that blocker in place.

Git reference: Git - git-sparse-checkout Documentation

3

Just adding to the context - This repo is so big that sparse checkout is not working

4

as @bcidecoded alluded, I gues the best is to get a version with one subdataset per subject.

1 Like