Error when running bandpass filter with AFNI in nipype

,
1

Hello, this is my first time posting on neurostars so please let me know if I’m missing any information. I am trying to use nipype to preprocess some task data from an fMRI study we collected that has 28 subjects and 5 runs/subject. The pipeline runs correctly for runs 1-4 for all 28 subjects, but doesn’t work properly for the fifth run for all 28 subjects (so it’s not a subject-specific problem).

The error appears to be in writing the output of afni bandpass. The error log says that afni bandpass “failed to find header file for ‘/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gzq’”.

Another curveball is that, after the error message, it says it saved the file. I can also print the header information using “nifti_tool -disp_hdr” and if I load the output file from afni’s bandpass using nibabel. But, if I try to run the next preprocessing step (which uses afni’s 3dtproject to regress out head motion confounds), I get another error message that the header file in the bandpass output file is missing.

I have copied the command I used to run afni bandpass and the logging output for afni bandpass below.

Has anyone run into this problem before? Or otherwise, know why the output might not be writing and why it says the header file is missing?

Thanks,

Steve

Command:

Import packages

from future import division
import os, re
from os.path import join as opj

#import nipype
import nipype
nipype.config.set(‘logging’, ‘workflow_level’, ‘INFO’)
nipype.config.set(‘logging’, ‘interface_level’, ‘INFO’)
nipype.logging.update_logging(nipype.config)
import nipype.interfaces.fsl as fsl
import nipype.interfaces.afni as afni
import nipype.interfaces.ants as ants

#Set file paths
bp_inFile = dmt_outFile
bp_outFile = ‘bp_{}’.format(bp_inFile)

#create node and set node options
bandpass = afni.Bandpass()
bandpass.inputs.in_file = opj(funcPath,bp_inFile)
bandpass.inputs.out_file = opj(funcPath,bp_outFile)
bandpass.inputs.mask = opj(funcPath,maskFile)
bandpass.inputs.no_detrend = True
bandpass.inputs.highpass = 0.01
bandpass.inputs.lowpass = 0.12
bandpass.inputs.outputtype = ‘NIFTI_GZ’

#run node
result = bandpass.run()

Output:
…Running bandpass filter…
170922-16:14:21,632 interface INFO:
stderr 2017-09-22T16:14:21.632298:++ 3dBandpass: AFNI version=AFNI_2011_12_21_1014 (Jun 16 2014) [64-bit]
170922-16:14:22,133 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Authored by: RW Cox
170922-16:14:22,133 interface INFO:
stderr 2017-09-22T16:14:22.133510:*+ WARNING: If you are performing spatial transformations on an oblique dset,
170922-16:14:22,133 interface INFO:
stderr 2017-09-22T16:14:22.133510: such as /data/picsl/stompson/SNL/data/subjs_raw/SNL_039/func/stc_mc_dsp_BOLD_run10.nii.gz_mask.nii.gz,
170922-16:14:22,133 interface INFO:
stderr 2017-09-22T16:14:22.133510: or viewing/combining it with volumes of differing obliquity,
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510: you should consider running:
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510: 3dWarp -deoblique
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510: on this and other oblique datasets in the same session.
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510: See 3dWarp -help for details.
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Oblique dataset:/data/picsl/stompson/SNL/data/subjs_raw/SNL_039/func/stc_mc_dsp_BOLD_run10.nii.gz_mask.nii.gz is 37.899982 degrees from plumb.
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Number of voxels in mask = 136933
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Oblique dataset:/data/picsl/stompson/SNL/data/subjs_raw/SNL_039/func/dmt_stc_mc_dsp_BOLD_run10.nii.gz is 37.899982 degrees from plumb.
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Data length = 310 FFT length = 320
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510: + bandpass: ntime=310 nFFT=320 dt=1 dFreq=0.003125 Nyquist=0.5 passband indexes=3…38
170922-16:14:22,134 interface INFO:
stderr 2017-09-22T16:14:22.133510:++ Loading input dataset time series
170922-16:14:24,677 interface INFO:
stderr 2017-09-22T16:14:24.677218:++ Checking dataset for initial transients [use ‘-notrans’ to skip this test]
170922-16:14:25,688 interface INFO:
stderr 2017-09-22T16:14:25.688224: + No widespread initial positive transient detected :slight_smile:
170922-16:14:26,189 interface INFO:
stderr 2017-09-22T16:14:26.189098:++ Bandpassing data time series
170922-16:14:26,689 interface INFO:
stderr 2017-09-22T16:14:26.689598:++ Creating output dataset in memory, then writing it
170922-16:14:38,422 interface INFO:
stderr 2017-09-22T16:14:38.421918:++ Output dataset /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz
** ERROR (nifti_image_read): failed to find header file for ‘/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gzq’

** FATAL ERROR: Can’t open dataset ‘/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz^A^A’
** Program compile date = Jun 16 2014

Saving bandpass filtered file /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz

2

It looks like ‘^A^A’ is getting appended to the output file name somehow. Can you post the values of dmt_outFile, pb_outFile and bandpass.cmdline?

3

The .gzq extension is sticking out to me. Is there a typo somewhere that might be adding a “q” to your filename?

4

Thanks! I’m not sure where the ^A^A or .gzg are coming from. They don’t appear in my code anywhere and don’t appear in the inputs to bandpass (see cmdline, dmt_outFile, bp_outFile below).

dmt_outFile:
‘dmt_stc_mc_dsp_BOLD_run5.nii.gz’

bp_outFile:
‘bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz’

bandpass.cmdline:
u’3dBandpass -prefix /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz -mask /data/subjs_raw/test/func/stc_mc_dsp_BOLD_run5.nii.gz_mask.nii.gz -nodetrend 0.010000 0.120000 /data/subjs_raw/test/func/dmt_stc_mc_dsp_BOLD_run5.nii.gz’

5

Yeah, that’s really weird. Just to beat a dead horse, can you post the results of running:
bp_outFile[-1] == 'z'

6

In [3]: bp_outFile[-1] == ‘z’
Out[3]: True

7

Value of bandpass._list_outputs()?
I’m just trying to figure out where those characters are getting added.

8

Are you able to run the cmdline string directly in the shell?

3dBandpass -prefix /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz \
    -mask /data/subjs_raw/test/func/stc_mc_dsp_BOLD_run5.nii.gz_mask.nii.gz \
    -nodetrend 0.010000 0.120000 \
    /data/subjs_raw/test/func/dmt_stc_mc_dsp_BOLD_run5.nii.gz
9

Thanks! I appreciate you helping me troubleshoot. I’m struggling with this error in part because if it was a simple bug in the code you’d expect it to appear for all of the runs, not just run 5.

In [5]: bandpass._list_outputs()
Out[5]: {‘out_file’: u’/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run1.nii.gz’}

10

That’s what it says for run_1. Run_5 looks the same?

11

Yes, sorry I copied the wrong code! I was just trying to check whether it looked the same for both. Run 5 looks the same.

12

I agree with effigies, I’d like to see if it runs on the commandline. Also, it looks like maybe in your original post you’ve posted the output from processing run10 along with the error from processing run5. Shouldn’t matter, but can you post the output from run5, along with the report.rst for run5?

13

Hmm interesting. When I run the cmdline string directly in the shell it doesn’t give me the error message, but I still can’t run the next step. When I try to run the next step directly in the shell, it says:

** ERROR (nifti_image_read): failed to find header file for ‘/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gzq’
** FATAL ERROR: Can’t open dataset ‘/data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz’

Why would one part of the file have the correct path (.nii.gz) and the other error (nifti_image_read) have the ‘.nii.gzq’?

Two other things to note: I don’t have a report.rst file because I haven’t been using workflows for my pipeline (yet). This is my first time trying to preprocess data using nipype and i was told that it might be easier to learn without workflows.

Also, the run5/run10 distinction is just a label that’s separately applied to different subjects. For some subjects the tasks I’m analyzing are runs 1-5 and for some it’s runs 6-10. The error logs are identical regardless of whether the last run I’m looking at is run 5 or run 10.

14

What’s the next interface you’re running? It looks like bandpass is completing fine, maybe the issue is with the subsequent interface. Can you post the code you’re using to run the following interface?

15

Yes! I couldn’t find a good nipype interface for confound regression, so I was just using subprocess to call 3dTproject using the shell command.

Command:
rg_inFile = bp_outFile
rg_outFile = ‘rg_{}’.format(rg_inFile)

rg_inPath = opj(funcPath,rg_inFile)
rg_outPath = opj(funcPath,rg_outFile)
maskPath = opj(funcPath,maskFile)
allConfoundsPath = opj(funcPath,’{}_36Params.1D’.format(rg_inFile))

subprocess.check_call(‘3dTproject -input {} -prefix {} -mask {} -ort {} -polort 1’.format(rg_inPath,rg_outPath,maskPath,allConfoundsPath),shell=True)

The completed cmdline call with file paths is:
‘3dTproject -input /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz -prefix /data/subjs_raw/test/func/rg_bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz -mask /data/subjs_raw/test/func/stc_mc_dsp_BOLD_run5.nii.gz_mask.nii.gz -ort /data/subjs_raw/test/func/bp_dmt_stc_mc_dsp_BOLD_run5.nii.gz_36Params.1D -polort 1’

16

And when you paste that into the commandline, I assume it works fine?

17

A touch off-topic from the strange .gzq issue, but unless I’m misunderstanding what you’re trying to do with 3dTproject, you should be able to do confound regression in the same step as your bandpassing. Simply set bandpass.inputs.orthogonalize_file to point to your *_36Params.1D file.

If you’re able to remove the 3dTproject node you might see a temporary resolution to the issue…

18

Hmm I hadn’t considered that option. That would certainly make things simpler.

As for the mysterious .gzq issue, I think I was able to figure it out. It appears that afni was having trouble with the length of the file name. All the prefixes I had appended at each step made things go haywire. I didn’t think the path was that short to begin with, but shortening it even more resolves the issue. Apparently this is an issue with older versions of AFNI. Thanks all for your help debugging!