fMRIPrep HCP-Aging Issue: FreeSurfer orig/001.mgz as T1w

Hello Neurostars,

I’m processing the HCP-Aging 2.0 dataset with fMRIPrep and running into an issue. This issue is likely the result of a decision I made to use the output from HCP’s PreFreeSurferPipeline.sh as structural input for fMRIPrep.

I’ve made this decision because (1) I would like to do do all fMRI preprocessing using a pipeline that I’ve customized for a separate dataset, but (2) I would like to use the FreeSurfer output from HCP’s FreeSurfer pipeline because it has been thoroughly QCed. These files are very minimally processed; according to the code description:

# The primary purposes of the PreFreeSurfer Pipeline are:
#
# 1. To average any image repeats (i.e. multiple T1w or T2w images available)
# 2. To create a native, undistorted structural volume space for the subject
#    * Subject images in this native space will be distortion corrected
#      for gradient and b0 distortions and rigidly aligned to the axes
#      of the MNI space. "Native, undistorted structural volume space"
#      is sometimes shortened to the "subject's native space" or simply
#      "native space".
# 3. To provide an initial robust brain extraction
# 4. To align the T1w and T2w structural images (register them to the native space)
# 5. To perform bias field correction
# 6. To register the subject's native space to the MNI space

I’ve symlinked the FreeSurfer input (i.e., $sub/mri/orig/{001,T2*}.mgz) to the subjects BIDS directory (i.e., $sub/ses-?/anat/$sub_ses-?_run-1_{T1,T2}w.nii.gz).

I’m not sure how to create a BIDS JSON file for these derivative images, so I used copies of a version of the images from an earlier HCP Pipeline (i.e., T1w_MPR_vNav_4e_e1e2_mean/${hcp_sub}_V1_MR_T1w_MPR_vNav_4e_e1e2_mean.json).

While running fmriprep using a containerized version of 22.1.0+0.gce344b39.dirty, I get the following error:

Command:

singularity run \
        --cleanenv -B /projects \
        /projects/p30952/environment/singularity_images/${FMRIPREP_VER}.simg \
        ${BIDS_DIR} \
        ${FMRIPREP_DIR} \
        participant --participant-label ${subject} \
        -w ${WORK_DIR} \
        --skip_bids_validation \
        --omp-nthreads 8 \
        --nthreads 12 \
        --mem_mb 24000 \
        --fs-license-file ${FS_LICENSE} \
	    --fs-subjects-dir ${BIDS_DIR}/derivatives/freesurfer \
        --output-spaces anat fsnative MNI152NLin2009cAsym fsLR fsaverage \
        --cifti-output 91k \
        --fd-spike-threshold 0.4 \
        --dvars-spike-threshold 1.5 \
	    --dummy-scans 4

Error:


Node Name: fmriprep_22_1_wf.single_subject_HCA6002236_wf.anat_preproc_wf.anat_template_wf.t1w_ref_dimensions

File: /projects/b1145/bram/hcp-aging/derivatives/fmriprep-22.1.0/sub-HCA6002236/log/20230224-122507_f0b9a088-0c7d-4934-9bf2-1a88c2b34db5/crash-20230224-123908-brd2241-t1w_ref_dimensions-6fe94ef1-e8aa-4685-8de9-ef9333e2602e.txt
Working Directory: /projects/b1145/bram/work/fmriprep_22_1_wf/single_subject_HCA6002236_wf/anat_preproc_wf/anat_template_wf/t1w_ref_dimensions
Inputs:

    max_scale: 3.0
    t1w_list:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node t1w_ref_dimensions.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/images.py", line 436, in _run_interface
	    orig_imgs = np.vectorize(nb.load)(in_names)
	  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2304, in __call__
	    return self._vectorize_call(func=func, args=vargs)
	  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2382, in _vectorize_call
	    ufunc, otypes = self._get_ufunc_and_otypes(func=func, args=args)
	  File "/opt/conda/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2342, in _get_ufunc_and_otypes
	    outputs = func(*inputs)
	  File "/opt/conda/lib/python3.9/site-packages/nibabel/loadsave.py", line 103, in load
	    matches, msg = _signature_matches_extension(filename, sniff)
	  File "/opt/conda/lib/python3.9/site-packages/nibabel/loadsave.py", line 65, in _signature_matches_extension
	    if sniff.startswith(expected_signature):
	AttributeError: 'tuple' object has no attribute 'startswith'

My Questions:

  1. Is my logic in using FreeSurfer input for fMRIPrep flawed? If so, what files from HCP should I use?
  2. Is the solution as simple as creating more appropriate JSON files for the fMRIPrep input? How can I accomplish this for derivatives from a prior processing pipeline?
  3. I’m worried that the issue is within the header of the image files I’m using as structural inputs to fMRIPrep; should/how can I modify this?

Cheers,
Bram

Hi @bramdiamond,

  • You skip bids validation in your command. Are your data BIDS valid? What is the output of the BIDS validator?
  • Based on the error you got, I doubt this is the case - but fMRIPrep 21+ uses freesurfer 7.2+, so maybe there is something about the outputs between the modified v5.3 that fMRIPrep does not expect?
  • Can you share one of the JSONs from HCP?
  • Unrelated, but I doubt 24GB is enough memory given the number of HCP scans, size of HCP data, and number of output spaces you are producing.

Did you convert to nifti (e.g. with FS mris_convert) or just symlink? I recommend converting.

Best,
Steven

Thanks for taking a look @Steven!

This is what I get when not ignoring BIDS formatting:

dim[0] is out-of-range, we'll simply try continuing to read the file, but this will most likely fail horribly.
dim[0] is out-of-range, we'll simply try continuing to read the file, but this will most likely fail horribly.
dim[0] is out-of-range, we'll simply try continuing to read the file, but this will most likely fail horribly.
/opt/conda/lib/python3.9/site-packages/nitransforms/base.py:207: UserWarning: Reference space not set
  warnings.warn("Reference space not set")

The dim[0] is out-of-range... line keeps repeating.

I believe that HCP-Aging 2.0 uses a modified version of FreeSurfer v6.0. This is from the $sub/scripts/lastcall.build-stamp.txt: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.0-2beb96c. I was under the impression that fMRIPrep should work for FreeSurfer v6.0.

cat ${sub}_ses-1_run-1_T1w.json
{
  "Modality": "MR",
  "MagneticFieldStrength": 3,
  "Manufacturer": "Siemens",
  "ManufacturersModelName": "Prisma",
  "InstitutionName": "CMRR_3T-C",
  "InstitutionalDepartmentName": "Department",
  "InstitutionAddress": "6th_Street_SE_2021_Minneapolis_MN_US_55454",
  "DeviceSerialNumber": "166007",
  "StationName": "TRIOC",
  "BodyPartExamined": "BRAIN",
  "PatientPosition": "HFS",
  "ProcedureStepDescription": "HCP_PRIVATE_HCP_2017.05.08_saveRaw",
  "SoftwareVersions": "syngo_MR_E11",
  "MRAcquisitionType": "3D",
  "SeriesDescription": "T1w_MPR_vNav_4e",
  "ProtocolName": "T1w_MPR_vNav_4e",
  "ScanningSequence": "GR_IR",
  "SequenceVariant": "SK_SP_MP",
  "ScanOptions": "IR_WE",
  "SequenceName": "tfl3d4_16ns",
  "ImageType": [
    "ORIGINAL",
    "PRIMARY",
    "M",
    "ND",
    "MEAN"
  ],
  "SeriesNumber": 25,
  "AcquisitionTime": "11:39:18.190000",
  "AcquisitionNumber": 1,
  "SliceThickness": 0.8,
  "SAR": 0.0287266,
  "RepetitionTime": 2.5,
  "InversionTime": 1,
  "FlipAngle": 8,
  "PartialFourier": 1,
  "BaseResolution": 320,
  "ShimSetting": [
    -891,
    -11542,
    -3879,
    282,
    -112,
    6,
    127,
    50
  ],
  "TxRefAmp": 267.567,
  "PhaseResolution": 1,
  "ReceiveCoilName": "Head_32",
  "ReceiveCoilActiveElements": "HEA;HEP",
  "PulseSequenceDetails": "%CustomerSeq%_tfl_mgh_epinav_ABCD",
  "ConsistencyInfo": "N4_VE11C_LATEST_20160120",
  "PercentPhaseFOV": 93.75,
  "PhaseEncodingSteps": 299,
  "AcquisitionMatrixPE": 300,
  "ReconMatrixPE": 320,
  "ParallelReductionFactorInPlane": 2,
  "PixelBandwidth": 745,
  "DwellTime": 2.1e-06,
  "ImageOrientationPatientDICOM": [
    -0.0115737,
    0.930533,
    -0.366026,
    0.0944014,
    -0.363399,
    -0.926839
  ],
  "InPlanePhaseEncodingDirectionDICOM": "ROW",
  "ConversionSoftware": "dcm2niix",
}

This is only for one participant. What do you recommend I change in my sbatch command?

Good point! Just a symlink. Converted with mri_convert and now running again to see if I have the same issue.

It appears that even with converting to .nii.gz, I’m still running into the same issue:

/opt/conda/lib/python3.9/site-packages/nitransforms/base.py:207: UserWarning: Reference space not set
  warnings.warn("Reference space not set")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/sdcflows/transform.py:398: UserWarning: Image's and B-Spline's grids are not aligned.
  warn("Image's and B-Spline's grids are not aligned.")
/opt/conda/lib/python3.9/site-packages/nitransforms/base.py:207: UserWarning: Reference space not set
  warnings.warn("Reference space not set")

Do you have access to the native space T1 image that results from prefreesurfer pipeline? It seems like that would be an okay input if you skip the skull stripping in fmriprep since it appears that is already done. I don’t think the double bias correction would be that much of an issue.

I believe that is the file I am already using (i.e., ${T1wFolder}/${T1wImage}_acpc_dc_restore. What do you suggest?


(Glasser et al., 2013)

It’s my impression that the final box (i.e., “Native Volume Space”) is the same as ${T1wFolder}/${T1wImage}_acpc_dc_restore. Is that not the case?

I thought you were using the .MGZ file as per your earlier answer

Is the $sub/mri/orig/{001,T2*}.mgz simply a mri_convert (.nii.gz.mgz) version of $sub/ses-?/anat/$sub_ses-?_run-1_{T1,T2}w.nii.gz?

I do not know, but in either case there might be information that is lost/distorted during file type conversion, so it’s safer to try cutting out intermediary steps.

1 Like

Okay, I’ll have a go at that - thanks!

@Steven, you’re suggestions appear to be working! How many GB do you recommend per scan?

It seems that I am still getting an error:

Node Name: fmriprep_22_1_wf.single_subject_HCA6670479_wf.anat_preproc_wf.anat_derivatives_wf.raw_sources

File: /projects/b1145/bram/hcp-aging/derivatives/fmriprep-22.1.0/sub-HCA6670479/log/20230227-173241_37eb8067-e4b8-4ebc-b342-54016c5c3ce5/crash-20230227-173808-brd2241-raw_sources-a6a06608-0044-4b96-9ace-8d2d3cae7aeb.txt
Working Directory: /projects/b1145/bram/work/fmriprep_22_1_wf/single_subject_HCA6670479_wf/anat_preproc_wf/anat_derivatives_wf/raw_sources
Inputs:

    bids_root: /projects/b1145/bram/hcp-aging
    function_str: def _bids_relative(in_files, bids_root): from pathlib import Path if not isinstance(in_files, (list, tuple)): in_files = [in_files] in_files = [str(Path(p).relative_to(bids_root)) for p in in_files] return in_files
    in_files:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node raw_sources.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
	    out = function_handle(**args)
	  File "", line 6, in _bids_relative
	  File "", line 6, in 
	  File "/opt/conda/lib/python3.9/pathlib.py", line 939, in relative_to
	    raise ValueError("{!r} is not in the subpath of {!r}"
	ValueError: '/home/brd2241/projects/b1145/bram/hcp-aging/sub-HCA6670479/ses-1/anat/sub-HCA6670479_ses-1_run-1_T1w.nii.gz' is not in the subpath of '/projects/b1145/bram/hcp-aging' OR one path is relative and the other is absolute.

@bramdiamond. This does not look to be related to the earlier error. Is this with or without the premade bids index? Also are your data in /projects as suggested by your fMRIPrep invocation, or in /home/brd2241/projects?

This is with the premade BIDS index.

I have a symlink to our projects’ directories in my home directory because they are impossible to remember. /home/brd2241/projects/ has symlinks to the relevant projects directories, while /projects/ has true links to hundreds of projects directories. See below:

[brd2241@quser31 hcp-aging]$ pwd
/home/brd2241/projects/b1145/bram/hcp-aging
[brd2241@quser31 hcp-aging]$ readlink -f .
/projects/b1145/bram/hcp-aging

Yah that may be the problem. Based on our personal correspondance, you made the pybids layout with /home address. So all of your layout items are probably based off of that. You only have /projects mounted. Without seeing your full script, I do not know what you have set as your $BIDS_DIR and other directory argument. Try without premade first to see if this is indeed the issue.

I would then try remaking the layout while only using true paths, and in fmriprep command only use true paths. Symlinks and singularity are certainly possible but it becomes a bit more of headache when thinking of what needs to be mounted, best to avoid it altogether if possible.

Do you mean per subject? It depends on a lot of things… I don’t know what your HPC cluster is, but if it uses SLURM you can just run one subjects with lots of memory, and then run seff $JOBID to see how much memory it used and base future jobs off of that.

I think you are exactly right about the pybids command. I’m re-running w/out the bids-directory index.

I’m also re-running the pybids command with full paths.

closing this thread as it seems further progress is being discussed at fMRIPrep Ignoring Pre-Processed FreeSurfer from HCP-Aging 2.0