Clarifications on what BIDS schema says about denoting new lines in tsvs

Software Support
,
1

The BIDS schema has an error code relating to new lines

# BIDS Validator Original Issue Code #70
WrongNewLine:
  code: WRONG_NEW_LINE
  message: |
    All TSV files must use Line Feed '\n' characters to denote new lines.
    This files uses Carriage Return '\r'.
  level: error
  selectors:
    - extension == ".tsv"

I’m not sure how to interpret this. Does “This file uses Carriage Return“ imply that validators should check for the presence of carriage returns (that is, separating lines with CRLF is disallowed), or does it mean that validators are checking for the use of carriage returns alone as a line ending?

In bids-examples, there’s a file that mostly uses \r\n, but the last line ends with \n .

❯ cat -e xeeg_hed_score/sub-ieegModulator/ses-ieeg01/ieeg/sub-ieegModulator_ses-ieeg01_electrodes.tsv | head -n 2
name	x	y	z	size	hemisphere	seizure_zone	Destrieux_label	Destrieux_label_text^M$
RO1	13.06511498	67.61359064	-51.92367238	5	R	n/a	12131	rh_G_rectus^M$

❯ cat -e xeeg_hed_score/sub-ieegModulator/ses-ieeg01/ieeg/sub-ieegModulator_ses-ieeg01_electrodes.tsv | tail -n 2
RY15	64.43587541	16.62375016	-14.21599021	5	R	n/a	41	Right_Cerebral_White_Matter^M$
RY16	68.37697562	16.88315244	-13.87652125	5	R	n/a	12129	rh_G_precentral$

This is a somewhat awkward file in that some standard tools under default configurations will fail to read it. For example, here is duckdb

import duckdb

# succeeds
d = duckdb.read_csv(
    "sub-ieegModulator_ses-ieeg01_electrodes.tsv", sep="\t", strict_mode=False
)

# fails
d = duckdb.read_csv("sub-ieegModulator_ses-ieeg01_electrodes.tsv", sep="\t")

outputs

---------------------------------------------------------------------------
InvalidInputException                     Traceback (most recent call last)
Cell In[2], line 1
----> 1 d = duckdb.read_csv("sub-ieegModulator_ses-ieeg01_electrodes.tsv", sep="\t")

InvalidInputException: Invalid Input Error: Error when sniffing file "sub-ieegModulator_ses-ieeg01_electrodes.tsv".
It was not possible to automatically detect the CSV parsing dialect
The search space used was:
Delimiter Candidates: '	'
Quote/Escape Candidates: ['(no quote)','(no escape)'],['"','(no escape)'],['"','"'],['"','''],['"','\'],[''','(no escape)'],[''','''],[''','"'],[''','\']
Comment Candidates: '\0', '#'
Encoding: utf-8
Possible fixes:
* Disable the parser's strict mode (strict_mode=false) to allow reading rows that do not comply with the CSV standard.
* Make sure you are using the correct file encoding. If not, set it (e.g., encoding = 'utf-16').
* Delimiter is set to '	'. Consider unsetting it.
* Set quote (e.g., quote='"')
* Set escape (e.g., escape='"')
* Set comment (e.g., comment='#')
* Set skip (skip=${n}) to skip ${n} lines at the top of the file
* Enable ignore errors (ignore_errors=true) to ignore potential errors
* Enable null padding (null_padding=true) to pad missing columns with NULL values
* Check you are using the correct file compression, otherwise set it (e.g., compression = 'zstd')
* Be sure that the maximum line size is set to an appropriate value, otherwise set it (e.g., max_line_size=10000000)

I’m not super familiar with the relevant standards. Wikipedia’s article points here, which just specifies “EOL” (ambiguous about whether that’s the same EOL). The CSV standard requires “CRLF“, except for the last line, which “may or may not have an ending line break”.

As far as I understand, that issue is not currently triggered on the bids-validator, and so I’m not aware of a de facto answer.

❯ deno run -ERWN jsr:@bids/validator xeeg_hed_score --ignoreWarnings | grep -i error
	[ERROR] EMPTY_FILE Empty files not allowed.
2

It looks like the legacy validator errored on \r, but not \r\n or \n:

The current validator uses TextLineStream from text-line-stream - @std/streams - JSR, which splits on \r\n or \n. It should also fail for \r, but it looks like it might crash or throw an error that is caught and reported as a more general read failure.

If the spec says they must be LF terminated and not CRLF, then we should implement that, but if not, being more strict now will break datasets.

3

To me, the quoted part of the spec above is ambiguous, so I don’t see a reason to require LF and not CRLF for line endings; just disallowing the single-character line ending \r makes sense to me.

How about a new check that disallows multiple line-ending types within the same dataset (or at least within the same table)? That would disallow cases like that sub-ieegModulator_ses-ieeg01_electrodes.tsv file in xeeg_hed_score.

4

Feels like quite an expensive check to do in schema (would need to be something like match(text, '\r\n') != match(text, '[^\r]\n') that would process the text twice). In the validator, you could make a stream splitter that remembers the first line break it sees and raises an error if it hits a different one.

While this one seems manageable with custom code in the validator, I do think there’s room for a checker that goes beyond what the spec states or the validator can reasonably do. We started GitHub - OpenNeuroOrg/miniqc: A BIDS app for performing minimal QC beyond validation · GitHub a while back to detect corrupted NIfTI files. The intent was to add more checks as we encountered the issues, but we haven’t found the need for others yet.

5

IDK, but I admit that seeing miniqc makes me a bit anxious; I understood rules.errors as where one-off checks that can’t be expressed in the schema’s expression DSL go. For example, there’s the rule INVALID_JSON_ENCODING (bids-validator and implementation), which any compliant validator would seem to need to handle uniquely. The anxiety comes from the distinction between a one-off rule and a rule that no validator can reasonably do not being clear to me.