MRIQC metadata error: file not found?

,
1

Hi everyone,

when I try to run MRIQC on my subjects I get this error:

PS C:\Users\45027900> docker run -it --rm -v C:\Users\45027900\Desktop\tmp\mriqc:/scratch -v D:\Andrea\fov\BIDS:/data:ro -v D:\Andrea\fov\BIDS\derivatives\MRiQC:/out poldracklab/mriqc:latest /data /out participant -w /scratch --verbose --no-sub --nproc 19 --mem_gb 45 -f --participant-label 09
201223-03:23:56,876 cli IMPORTANT:

    Running MRIQC version 0.15.2:
      * BIDS dataset path: /data.
      * Output folder: /out.
      * Analysis levels: ['participant'].

201223-03:23:57,595 nipype.utils WARNING:
         Could not check for version updates:
Connection to server could not be made
201223-03:23:58,276 nipype.utils WARNING:
         Could not check for version updates:
Connection to server could not be made
201223-03:24:11,11 nipype.workflow INFO:
         Workflow mriqc_wf settings: ['check', 'execution', 'logging', 'monitoring']
201223-03:24:11,347 nipype.workflow INFO:
         Running in parallel.
201223-03:24:11,356 nipype.workflow INFO:
         [MultiProc] Running 0 tasks, and 20 jobs ready. Free memory (GB): 45.00/45.00, Free processors: 19/19.
201223-03:24:11,445 nipype.workflow INFO:
         [Job 0] Cached (mriqc_wf.funcMRIQC.ComputeIQMs.provenance).
201223-03:24:11,447 nipype.workflow INFO:
         [Node] Setting-up "mriqc_wf.funcMRIQC.ComputeIQMs.metadata" in "/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata".
201223-03:24:11,477 nipype.workflow INFO:
         [Node] Running "metadata" ("niworkflows.interfaces.bids.ReadSidecarJSON")
201223-03:24:16,707 nipype.workflow ERROR:
         Node metadata.a5 failed to run on host 73a040c4a132.
201223-03:24:16,707 nipype.workflow ERROR:
         Saving crash info to /out/logs/crash-20201223-032416-root-metadata.a5-6acff2e4-04c6-4184-8ddb-e45723e8b1df.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json' -> '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 531, in run
    shutil.move(hashfile_unfinished, hashfile_unfinished.replace("_unfinished", ""))
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 571, in move
    copy_function(src, real_dst)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json'

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json' -> '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 531, in run
    shutil.move(hashfile_unfinished, hashfile_unfinished.replace("_unfinished", ""))
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 571, in move
    copy_function(src, real_dst)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/mriqc", line 10, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/run.py", line 69, in main
    mriqc_wf.run(**config.nipype.get_plugin())
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 184, in run
    self._send_procs_to_workers(updatehash=updatehash, graph=graph)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 348, in _send_procs_to_workers
    jobid, graph, result={"result": None, "traceback": traceback}
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/base.py", line 227, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 557, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json' -> '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 531, in run
    shutil.move(hashfile_unfinished, hashfile_unfinished.replace("_unfinished", ""))
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 571, in move
    copy_function(src, real_dst)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/miniconda/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/mriqc_wf/funcMRIQC/ComputeIQMs/_in_file_..data..sub-09..func..sub-09_task-loc_run-1_bold.nii.gz/metadata/_0x6c22ac695a181c85d5bcbccf4e0d53ec_unfinished.json'

The strange thing is that MRIQC runs without any problem on other subjects, and all my folders (including the problematic subjects) have all the JSON files you expect to see in a BIDS dataset. The JSON has been generated with heudiconv.

Any idea on why I get this and how to solve it?

Thanks in advance.

2

This totally looks like two parallel and competing processes run against the same work directory (seemingly /scratch)

3

Thanks for your response.

That was actually the only process running on that subject. I get this error no matter if mriqc is running on a specific subject or on the whole BIDS folder, and even after deleting the working or output folders…

4

can you post your command line?

5

here is the command line. can this issue be related to this one?

6

Indeed, the error was related to the above issue. I inspected the JSON files that were giving this problem (they were all T1w JSON files) and compared with good files, and I noticed that one of the items in these files was made of null characters. Specifically, the offending field was ‘DataSetTrailingPadding’ inside of ‘global’, ‘slices’ (I had a similar problem with the bold JSONs, in that case the offending field was inside ‘time’, ‘samples’).

For future reference in case someone else get stuck with the same problem, this is the code I run to clean the JSON files before running MRIQC:

## CORRECT NULL BYTES IN ANATOMICALS
root_path = r"D:\Andrea\fov\BIDS\*\*\*T1w.json"
backup_dir = r"D:\Andrea\fov\misc\json_backup"
  
for name in glob.glob(root_path):
    os.chmod(name, stat.S_IWRITE)
    with open(name, 'r') as a:
        j = json.load(a)
        if 'global' in j:
            if 'DataSetTrailingPadding' in j['global']['slices']:
                copyfile(name, os.path.join(backup_dir, os.path.split(name)[1]))
                del j['global']['slices']['DataSetTrailingPadding']
                print(name + ' has been fixed')
    with open(name, 'w', encoding='utf-8') as a:
        os.chmod(name, stat.S_IWRITE)
        json.dump(j, a, ensure_ascii=False, indent=4)

A similar procedure can be used to clean the bold files (more details in the issue linked in my previous reply).

Andrea

7

I’m glad you had it fixed, but I honestly doubt this is not a race condition.

You are definitely having encoding issues here and there in your JSON files, maybe due to some glitch in the Docker mounting from Windows, and these issues might be related somehow (i.e., nodes in the compute graph of MRIQC spend much longer than expected when dealing with JSON files), but it doesn’t change the fact that the issue you reported in GitHub is just about file encoding errors.

However, this is an internal file of Nipype that is being renamed by some process and another process is trying to do the same rename shortly after. I’m pretty positive this particular error will not happen if you run one subject at once. An alternative possibility is that your bold runs have some BIDS entity that MRIQC is not handling correctly. How does the func/ folder look for this sub-09 across all sessions?

That said, glad you went past this.

8

I can tell you how I fixed it, for the reason of the issue I will definitely leave it to the experts (you) :slight_smile:

The sub-09/func folder looks just fine to me. My idea was that, since the anatomical and the bold JSON files for this subject had invalid fields with invalid characters in it, for some reason MRIQC was not able to open/edit/save those files, and that’s why the expected files went missing and the error got fixed after cleaning the files. I was running only one subject when this error came up, but I had other docker containers running on the same working folder - so I wouldn’t exclude some sort of racing issue.

Anyways, thanks to your help I managed to fix the issue, so a big thank you for the efforts you guys put on these amazing tools!

9

Can I ask you to post the directory listing of the func/ folder of this subject? I did not mean that the BIDS structure is wrong in any ways. If you were not (maybe unintentionally) running several processes at the same time, then it is MRIQC that is mismanaging parallelism, but the two problems are different (possibly related, as I mention above), but still independent in their root cause.

10

This is the directory listing for sub-09/func:

D:\ANDREA\FOV\BIDS\SUB-09\FUNC
    sub-09_task-exp_run-1_bold.json
    sub-09_task-exp_run-1_bold.nii.gz
    sub-09_task-exp_run-1_events.tsv
    sub-09_task-exp_run-2_bold.json
    sub-09_task-exp_run-2_bold.nii.gz
    sub-09_task-exp_run-2_events.tsv
    sub-09_task-exp_run-3_bold.json
    sub-09_task-exp_run-3_bold.nii.gz
    sub-09_task-exp_run-3_events.tsv
    sub-09_task-exp_run-4_bold.json
    sub-09_task-exp_run-4_bold.nii.gz
    sub-09_task-exp_run-4_events.tsv
    sub-09_task-exp_run-5_bold.json
    sub-09_task-exp_run-5_bold.nii.gz
    sub-09_task-exp_run-5_events.tsv
    sub-09_task-loc_run-1_bold.json
    sub-09_task-loc_run-1_bold.nii.gz
    sub-09_task-loc_run-1_events.tsv
1 Like
11

Hi all :slight_smile:

I am using a singularity container on a cluster and I was receiving the same error every time I submitted a SLURM job array. However, it worked as supposed to when I ran the command on an interactive node for a single subject.

This is exactly what was happening:

This totally looks like two parallel and competing processes run against the same work directory.

I solved the issue by binding the work directory to the local tmpdir created by SLURM for each (sub)job using --bind $SLURM_TMPDIR:/work and -w /work.

I adapted this fMRIprep script (also here) for MRIQC but -B $L_SCRATCH:/work didn’t work as $L_SCRATCH seemed to be empty. :man_shrugging:.

How is $L_scratch defined? Is it specific to the SLURM system the script was based on?

Thanks!

1 Like
12

Hi @SamGuay !

$L_SCRATCH is defined on Sherlock (the Stanford computing system, where these scripts were originally developed) as the local scratch space for the compute node. You can find more detail here: https://www.sherlock.stanford.edu/docs/storage/filesystems/#l_scratch

If you’re working on compute Canada, you can define a similar directory using $SLURM_TMPDIR (which you’ve already found !). Just in case it’s useful, more information is available here under ‘Solutions.’

HTH !

Elizabeth

1 Like