ABCD data bids open neuro on AWS

grace-shearrer · May 23, 2018, 2:52pm

Hey, I getting ready to do some analyses on ABCD data. I was wondering if anyone had preprocessed the data into bids format and used fMRIprep/open neuro on it already using AWS/ developed a pipeline? I am a noob on AWS and if someone has already figured this out I would be pretty excited.
Grace

satra · May 23, 2018, 4:12pm

@tsalo - may have done this. (not the AWS part, but likely the ABCD dicoms to BIDS part)

grace-shearrer · May 23, 2018, 4:28pm

Ah thanks @satra ! @tsalo would love to pick your brain/AWS your code!

earl · May 23, 2018, 5:14pm

@grace-shearrer I am not aware of anyone having done this yet, and I am part of the ABCD consortium!

@ChrisGorgolewski showed me a tool recently you may want to check out or try?

Also, I started some heudiconv code for trying to make good BIDS filenames from ABCD DICOMs, but did not get very far.

~Eric

grace-shearrer · May 23, 2018, 5:29pm

Thanks! This is very cool. I am mostly struggling to get oriented on AWS right now, but when i get that figured out I will give this a shot

tsalo · May 23, 2018, 5:29pm

@grace-shearrer I’m sorry to say that I have only worked on ABCD data acquired at FIU. I haven’t tried downloading the data from NDAR yet. I do have a heuristics file for heudiconv that works for FIU’s ABCD data, but I am not sure if data from NDAR will have the same file structure. If you’re interested in trying it out, you can see it here. We’ve also written some code to assign the IntendedFor field for field maps based on the sequence number, which might be useful.

Please let me know if there’s any way I can help.

grace-shearrer · May 23, 2018, 5:30pm

Awesome! Thanks so much! I am gonna limp through this AWS/NDAR and I will give your heuristics file a try with the pyBIDS above

esalman · May 25, 2018, 8:40pm

Hi, I have been working to download data from NDAR- I was wondering if you have any tips on selecting and downloading only the rsfMRI or DTI files? Currently I have to open one page at a time and select them individually to download them.

earl · May 25, 2018, 10:11pm

@esalman What interface do you have access through? There may be a MUCH easier way with AWS or NDA maybe?

esalman · May 25, 2018, 10:28pm

I am accessing from the web interface at http://ndar.nih.gov. I am not aware about any other ways to access actually.

earl · May 29, 2018, 2:02pm

@esalman Yes, then you’re already accessing it the standard way and I think you’re right there’s no way to filter by MRI data type. If you email the NDA helpdesk, they can get you going with pulling the data using AWS instead of their web interface. Then you can pick and choose what data you want in an automated command-line way with the AWS command-line interface (CLI). Let me know if you need more links or email addresses. NDA helpdesk is just NDAHelp@mail.nih.gov. And at the bottom of this page they pretty much say what I just said:

https://data-archive.nimh.nih.gov/abcd

grace-shearrer · May 29, 2018, 4:08pm

Hey @esalman, I am also trying to work through the NDAR/miNDAR/AWS. It is annoying you can’t filter by MR sequence in the web interface. BUT you can download the text file only ABCD, and then you have the links to the individual scan sequences. With the code available at their github page you can get it to download only the sequences you want by giving it a list of the aws links https://github.com/NDAR/AWS-Download

esalman · May 29, 2018, 9:40pm

Hey @earl and @grace-shearrer, thank you so much for your help. I did make a list of files that I actually want to download- I am going to take a look at the script on GitHub to see if I can download through the list.

grace-shearrer · June 1, 2018, 5:08pm

NP! Good to know there are others in the same position as me

esalman · June 1, 2018, 8:27pm

I was also wondering about that until I saw this question pop up in the email digest. I had a brief look at the script and it looks like I can program it to download through my list. I will post an update if I can manage to do it.

chelsmar · January 10, 2019, 8:58pm

Hi @grace-shearrer, I wonder if you’ve had any success with using fMRIprep to process ABCD data using AWS? I’m trying to figure out AWS right now and it is a painful process. Any tips you have would be hugely appreciated!
thank you!

alexlicohen · June 12, 2020, 11:00am

In case anyone stumbles upon this thread looking for ABCD data in BIDS format, I want to point out that Damien Fair’s group put together this dataset that including both BIDS-formatted inputs, and derivatives/outputs of their modified HCPPipelines:
https://nda.nih.gov/edit_collection.html?id=3165

Documentation: https://collection3165.readthedocs.io/en/stable/

Description: This data collection from the Developmental Cognition and Neuroimaging (DCAN) Labs contains a regularly updated dataset of ABCD Brain Imaging Data Structure (BIDS) version 1.2.0 pipeline inputs and derivatives. Source data are currently comprised of all the ABCD Study participants baseline year 1 arm 1 DICOM imaging data that passed initial acquisition quality control from the ABCD Data Analysis and Informatics Center (DAIC) and were processed by DCAN Labs. The input DICOM data to this BIDS version 1.2.0 (https://www.nature.com/articles/sdata201644) data collection were retrieved from the NIMH Data Archive (NDA) share of ABCD fast-track data (https://nda.nih.gov/edit_collection.html?id=2573) and were last accessed on May 1, 2019. BIDS input data were converted from DICOMs using Dcm2Bids (https://github.com/cbedetti/Dcm2Bids). BIDS derivatives data were derived from the DCAN Labs ABCD-BIDS MRI processing pipeline which outputs Human Connectome Project (HCP) Minimal Preprocessing Pipelines-style data in both volume and surface spaces (https://doi.org/10.5281/zenodo.2587210, https://doi.org/10.1016/j.neuroimage.2013.04.127). This collection is independent from ABCD Data Collection 2573. Users may access ABCD DICOM files via the ABCD fast-track imaging data release in Collection 2573.