Data format for clinical electrophysiology research data

1

Clinical electrophysiology data can be extremely large, including several terabytes recorded over >1 week. It should be easy to read/load/analyze these data on a normal computer. Several questions come up here:

  1. Which BIDS allowed data formats are most suitable for this? Can we already read/write this format in Matlab/Python/other programming languages?
  2. Because of the large data size, major hospitals still throw away recordings. Which BIDS allowed format has compression?
  3. Which format allows parallel processing?
  4. Which format is best suited for HIPAA required encryption?
2

Multiscale Electrophysiology Format (v3.0) might be a solution to this.

The python tools for writing and reading already exist. However, the matlab solution supports only reading at the moment.

It supports compression in the ratio 1:7 (roughly).

The data are stored by channel in separate files which allows for parallel processing.

The data can be encrypted.

3

I don’t have an answer, but I just wanted to mention that python-neo has support for reading many different neurophysiology formats (list here), some of which find their way into clinical applications. For at least some of these formats it has the ability to merely map the memory to file and only load relevant data upon request, enabling opening of very large files that wouldn’t otherwise fit into memory.

I think that whatever format you land on, it would be good to make sure it is possible to write to that format from python-neo objects, effectively giving you a converter for ~39 different formats.

4

I’ll check it out. It shouldn’t be too difficult to include pymef (meflib python wrapper) into python-neo