fmriPrep error codes?

Justin_Smith1 · January 23, 2019, 8:17pm

Hello Neurostars,
I’m using fmriPrep 1.2.6 and was wondering if there’s a place where error codes are listed. I already deleted the .html file and forget the code, something to the effect of “Error: Node name whateverfileiscalled_summary_wf” or something to that effect.

I ran through a few subjects without problems, then re-did heudiConv on a larger subject pool and included field maps and DSI scans. As far as I know, we never used the field maps in our pre-processing before. I believe we took them for the DSI scans… I’m re-running with --ignore fieldmaps (no idea if this will change anything or not…). Hopefully this fixes the error, but if not it would be nice to know what the error is.

Help is appreciated

JUSTIN_SMITH · January 23, 2019, 9:01pm

UPDATE- (sorry for two accounts, my school computer is logged on to my school email)

Re-ran it, still got an error. It won’t let me upload the .html but it says “qform matrix overwritten” a bunch of times at the top and then gives me that same error at the bottom.

My command to run it:

singularity run --cleanenv -B /data2/users/liz/MLINDIV/fmriPrep:/work ~/fmriprep-1.2.6.simg /work/dataBids /work/processed participant -w /work/workDir --fs-license-file /work/license.txt --skip_bids_validation --ignore fieldmaps

And the end of the output

…
return super(SubjectSummary, self)._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 63, in _run_interface
segment = self._generate_segment()
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in _generate_segment
for series in bold_series)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 566, in init
self.update(*args, **kwds)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 653, in update
_count_elements(self, iterable)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in
for series in bold_series)

TypeError: ‘NoneType’ object is not subscriptable
190123-20:35:25,400 nipype.workflow WARNING:
[Node] Error on “fmriprep_wf.single_subject_020_wf.summary” (/work/workDir/fmriprep_wf/single_subject_020_wf/summary)
190123-20:35:25,402 nipype.workflow ERROR:
Node summary failed to run on host hopper.
190123-20:35:25,411 nipype.workflow ERROR:
Saving crash info to /work/processed/fmriprep/sub-020/log/20190123-201548_ab833868-4d9f-4c48-8582-986252f05b5b/crash-20190123-203525-jsmith-summary-121c01d5-ae8a-418f-a57b-2b014e82b960.txt
Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 473, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 557, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 637, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 369, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 96, in _run_interface
return super(SubjectSummary, self)._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 63, in _run_interface
segment = self._generate_segment()
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in _generate_segment
for series in bold_series)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 566, in init
self.update(*args, **kwds)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 653, in update
_count_elements(self, iterable)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in
for series in bold_series)
TypeError: ‘NoneType’ object is not subscriptable

Preprocessing did not finish successfully. Errors occurred while processing data from participants: 020 (1), 024 (1), 025 (1), 027 (1), 031 (1), 032 (1), 034 (1), 038 (1), 041 (1), 043 (1). Check the HTML reports for details.
Sentry is attempting to send 9 pending error messages
Waiting up to 2.0 seconds
Press Ctrl-C to quit
Captured warning (<class ‘ResourceWarning’>): unclosed <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=(‘128.111.106.39’, 59414), raddr=(‘35.188.42.15’, 443)>

JUSTIN_SMITH · January 23, 2019, 9:07pm

Error Log File:

Node: fmriprep_wf.single_subject_020_wf.summary
Working directory: /work/workDir/fmriprep_wf/single_subject_020_wf/summary

Node inputs:

bold = [’/work/dataBids/sub-020/func/sub-020_run-1_task-boldEx2_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldEx_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun1_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun2_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun3_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun4_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun5_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-boldRun6_bold.nii.gz’, ‘/work/dataBids/sub-020/func/sub-020_run-1_task-rest_bold.nii.gz’]
output_spaces = [‘template’, ‘fsaverage5’]
subject_id = sub-020
subjects_dir = /work/processed/freesurfer
t1w = [’/work/dataBids/sub-020/anat/sub-020_run-1_T1w.nii.gz’]
t2w = [’/work/dataBids/sub-020/anat/sub-020_run-1_T2w.nii.gz’]
template = MNI152NLin2009cAsym

Traceback (most recent call last):
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 316, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 473, in run
result = self._run_interface(execute=True)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 557, in _run_interface
return self._run_command(execute)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 637, in _run_command
result = self._interface.run(cwd=outdir)
File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 369, in run
runtime = self._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 96, in _run_interface
return super(SubjectSummary, self)._run_interface(runtime)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 63, in _run_interface
segment = self._generate_segment()
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in _generate_segment
for series in bold_series)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 566, in init
self.update(*args, **kwds)
File “/usr/local/miniconda/lib/python3.7/collections/init.py”, line 653, in update
_count_elements(self, iterable)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/interfaces/reports.py”, line 123, in
for series in bold_series)
TypeError: ‘NoneType’ object is not subscriptable

oesteban · January 24, 2019, 12:39am

I’m pretty positive that using that --skip_bids_validation argument is masking the underlying source of the problem here.

Could you report how fMRIPrep breaks if you remove that argument?

Justin_Smith1 · January 24, 2019, 10:04pm

Thank you for the suggestion, I now get the error pasted below. It looks like a problem with allocating memory…

This could be related, or completely due to the fact, that something is taking all of the available storage in /usr/ while this is trying to run. I don’t understand why if I’m using --cleanEnv. It’s hard to track down what it is since the script clears it all once it fails. Singularity was installed to the root directory, I moved it to a drive with more space and I’m still getting the same error.

The singularity image, data folder, output folder, and working folder are all on drives with plenty of space. The machine has >150GB of RAM and my understanding is that singularity utilizes all available resources.

I tried (Error using bids_validator) and I still get the same error. This is what I get after running the patch.

npm WARN saveError ENOENT: no such file or directory, open ‘/data2/users/liz/MLINDIV/fmriPrep/package.json’
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open ‘/data2/users/liz/MLINDIV/fmriPrep/package.json’
npm WARN fmriPrep No description
npm WARN fmriPrep No repository field.
npm WARN fmriPrep No README data
npm WARN fmriPrep No license field.

bids-validator@0.0.0
added 135 packages from 127 contributors and audited 196 packages in 3.649s
found 0 vulnerabilities

And this is what I get after running fmriPrep

$ ./singularityRun
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).

<— Last few GCs —>

[10722:0x2fd0110] 528492 ms: Mark-sweep 1208.4 (1215.3) -> 1208.4 (1214.3) MB, 8.9 / 0.0 ms (average mu = 0.998, current mu = 0.003) last resort GC in old space requested
[10722:0x2fd0110] 528500 ms: Mark-sweep 1208.4 (1214.3) -> 1208.4 (1214.3) MB, 8.2 / 0.0 ms (average mu = 0.998, current mu = 0.002) last resort GC in old space requested

<— JS stacktrace —>

==== JS stack trace =========================================
0: ExitFrame [pc: 0x164a15e5be1d]
Security context: 0x10c2b139e6e1
1: stringSlice(aka stringSlice) [0x33b48bb58c01] [buffer.js:594] [bytecode=0x3d6e6a10c1f9 offset=91](this=0x37026c9026f1 ,buf=0x1625c57ebf61 ,encoding=0x10c2b13bcbe9 <String[4]: utf8>,start=0,end=278649801)
2: toString [0x2e592599d4a1] [buffer.js:667] [bytecode=0x1625c57efaf1 offset=145](this=0x1625c57ebf61…

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: 0x8db900 node::Abort() [bids-validator]
2: 0x8db94c [bids-validator]
3: 0xad6c1e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [bids-validator]
4: 0xad6e54 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [bids-validator]
5: 0xec44e2 [bids-validator]
6: 0xed3cff v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [bids-validator]
7: 0xea38ab v8::internal::NewRawTwoByteString(int, v8::internal::PretenureFlag) [bids-validator]
8: 0xea41b2 v8::internal::NewStringFromUtf8(v8::internal::Vector, v8::internal::PretenureFlag) [bids-validator]
9: 0xae4c69 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [bids-validator]
10: 0x99d9f8 node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Localv8::Value) [bids-validator]
11: 0x8f7ac8 [bids-validator]
12: 0xb5f33f [bids-validator]
13: 0xb5fea9 v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate) [bids-validator]
14: 0x164a15e5be1d
Traceback (most recent call last):
File “/usr/local/miniconda/bin/fmriprep”, line 11, in
sys.exit(main())
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py”, line 354, in main
validate_input_dir(exec_env, opts.bids_dir, opts.participant_label)
File “/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py”, line 535, in validate_input_dir
subprocess.check_call([‘bids-validator’, bids_dir, ‘-c’, temp.name])
File “/usr/local/miniconda/lib/python3.7/subprocess.py”, line 341, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘bids-validator’, ‘/work/dataBids’, ‘-c’, ‘/tmp/tmpj5dq_11u’]’ died with <Signals.SIGABRT: 6>.
Sentry is attempting to send 1 pending error messages
Waiting up to 2.0 seconds

oesteban · January 24, 2019, 11:12pm

Hi @Justin_Smith1,

I don’t see any error “pasted below”. I only the output of trying to install the bids validator (I would imagine that using singularity shell…, but I’m pretty unclear on your settings).

So, long story short: your dataset is not BIDS valid. You’ll need to rename all your files so that the task entity appears BEFORE the run entity (e.g. sub-020/func/sub-020_task-boldEx2_run-1_bold.nii.gz). See the template given with the specification here. Please make sure your input data are BIDS-compliant.

Before fighting with fMRIPrep, you can use the online validator: https://bids-standard.github.io/bids-validator/

Please let us know if you sorted that out.

EDIT: coming back to your issue I understand now that the second code section shows the BIDS validator failing. Yes, it looks indeed a memory issue. Please use the online validator to check your dataset. Are you trying to run fmriprep on a login node?

JUSTIN_SMITH · January 25, 2019, 8:31pm

You are correct, such a simple fix… Bids validator informed me that I switched positions of the run # and the task name. Thank you!!