FitLins crashes with "'Dict' nodes are not implemented" error

Summary of what happened:

When trying to run fitLins using any model (lightly modified example model, model generated with pyBIDS automodel, etc.), fitLins crashes during the BIDS model loading process with the error message:

NotImplementedError: 'Dict' nodes are not implemented

Command used (and if a helper script was used, a link to the helper script or the command generated):

apptainer run --cleanenv --home /arc/burst/st-toddrebe-1/ \
  /arc/project/st-toddrebe-1/software_envs/fitlins/fitlins_latest.sif \
  "${BIDS_dir}" \
  "${out_dir}" \
  dataset \
  -vvv \
  -m "${model}" \
  -d "${BIDS_dir}"/derivatives/fmriprep \
  --space "MNI152NLin6Asym" \
  --n-cpus "${num_proc}" \
  -w "${work_dir}"

Version:

0.11.0

Environment (Docker, Singularity / Apptainer, custom installation):

Apptainer via HPC system

Data formatted according to a validatable standard? Please provide the output of the validator:

BIDS-formatted dataset with derivatives folder from fMRIprep. The following warning did not present any issues for fMRIprep. The data dictionaries were presenting other, unrelated issues so I did not include them - hence the second warning.

1: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)
2: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)

Relevant log outputs (up to 20 lines):

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node loader.
...
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/computation/expr.py", line 263, in f
	    raise NotImplementedError(f"'{node_name}' nodes are not implemented")
	NotImplementedError: 'Dict' nodes are not implemented

Screenshots / relevant information:

Model file:

{
    "Name": "MixedMotivation",
    "BIDSModelVersion": "1.0.0",
    "Input": {
        "subject": [
            "010",
            "011",
            "012"
        ],
        "task": "ADAPTAVOID"
    },
    "Nodes": [
        {
            "Level": "Run",
            "Name": "run_level",
            "GroupBy": [
                "run",
                "subject"
            ],
            "Model": {
                "X": [
                    1,
                    "trial_type.GoAppetitive",
                    "trial_type.GoAversive",
                    "trial_type.NoGoAppetitive",
                    "trial_type.NoGoAversive"
                ],
                ",Type": "glm"
            },
            "Contrasts": [
                {
                    "Name": "AppetitiveAversiveGo",
                    "ConditionList": [
                        "trial_type.GoAppetitive",
                        "trial_type.GoAversive"
                    ],
                    "Weights": [
                        1,
                        -1
                    ],
                    "Test": "t"
                }
            ]
        },
        {
            "Level": "Subject",
            "Name": "subject_level",
            "GroupBy": [
                "subject",
                "contrast"
            ],
            "Model": {
                "X": [
                    1
                ],
                "Type": "meta"
            },
            "DummyContrasts": {
                "Test": "t"
            }
        },
        {
            "Level": "Dataset",
            "Name": "one-sample_dataset",
            "GroupBy": [
                "contrast"
            ],
            "Model": {
                "X": [
                    1
                ],
                "Type": "glm"
            },
            "DummyContrasts": {
                "Test": "t"
            }
        }
    ]
}

Events file example:

onset duration task run trial_type response_time corr_resp rt_criterion rt_mean outcome num_presses_centered outcome_type
19.5338435 6.012443701 ADAPTAVOID 01 GoAversive 20.0594266 0.0 20.7451566 20.9974999 31.5606178 5.052631579 Go_punishment
36.1745694 5.9803594 ADAPTAVOID 01 GoAversive n/a 0.0 n/a 37.6382258 48.8014002 n/a Go_punishment
55.3823226 6.0251623 ADAPTAVOID 01 GoAppetitive 55.9894561 0.0 56.6296086 56.845979 64.4120456 6.052631579 other

I’m just trying to test with a simple model on a few participants to quickly debug, and have gotten the same error with different contrasts, HRF setups, etc. Any help or suggestions would be greatly appreciated, thank you!

Hi @brandon.forys, and welcome to neurostars!

You have an extra comma. before "Type":

Best,
Steven

1 Like

Could be helpful: BIDS Stats Model Validator — BIDS Stats Models Specification

Thanks @Steven and @effigies! After correcting the typo and also checking the BIDS model against the validator and the reference here: BIDS Stats Models Object Reference — BIDS Stats Models Specification I slightly revised the model to add the Edges list to see if it would help, removed the explicitly defined participant list, and enclosed the task name in an array. Unfortunately, the exact same error arises:

	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/pandas/core/computation/expr.py", line 263, in f
	    raise NotImplementedError(f"'{node_name}' nodes are not implemented")
	NotImplementedError: 'Dict' nodes are not implemented

arising from

	Traceback (most recent call last):
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run
	    runtime = self._run_interface(runtime)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/fitlins/interfaces/bids.py", line 246, in _run_interface
	    graph.load_collections(**selectors)
	  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/site-packages/bids/modeling/statsmodels.py", line 198, in load_collections
	    collections = self.layout.get_collections(node.level, drop_na=drop_na,

Below is the updated model:

{
  "Name": "MixedMotivation",
  "BIDSModelVersion": "1.0.0",
  "Input": {
    "task": ["ADAPTAVOID"]
  },
  "Nodes": [
    {
      "Level": "Run",
      "Name": "run",
      "GroupBy": ["run", "subject"],
      "Model": {"X": ["trial_type.GoAppetitive", "trial_type.GoAversive", "trial_type.NoGoAppetitive", "trial_type.NoGoAversive", 1],
        "Type": "glm"},
      "Contrasts": [
        {
          "Name": "AppetitiveAversiveGo",
          "ConditionList": ["trial_type.GoAppetitive", "trial_type.GoAversive"],
          "Weights": [1, -1],
          "Test": "t"
        }
      ]
    },
    {
      "Level": "Subject",
      "Name": "subject",
      "GroupBy": ["contrast", "subject"],
      "Model": {
        "X": [1],
        "Type": "meta"},
      "DummyContrasts": {"Test": "t"}
    },
    {
      "Level": "Dataset",
      "Name": "dataset",
      "GroupBy": ["contrast"],
      "Model": {
        "X": [1],
        "Type": "glm"},
      "DummyContrasts": {"Test": "t"}
    }
  ],
  "Edges": [
    {"Source": "run", "Destination": "subject"},
    {"Source": "subject", "Destination": "dataset"}
  ]
}

What else would you recommend checking?

Looking at your Run node, where are trial_type.* coming from? You probably need to factor your trial_type column into dummy variables (and probably to convolve them):

    {
      "Level": "Run",
      "Name": "run",
      "GroupBy": ["run", "subject"],
      "Transformations":{
         "Transformer":"pybids-transforms-v1",
         "Instructions":[
            {
               "Name":"Factor",
               "Input":[
                  "trial_type"
               ]
            },
            {
               "Name":"Convolve",
               "Input":[
                  "trial_type.GoAppetitive",
                  "trial_type.GoAversive",
                  "trial_type.NoGoAppetitive",
                  "trial_type.NoGoAversive",
               ],
               "Model":"spm"
            }
         ]
     },
     "Model": {"X": ["trial_type.GoAppetitive", "trial_type.GoAversive", "trial_type.NoGoAppetitive", "trial_type.NoGoAversive", 1],
        "Type": "glm"},
    ...

Update: I was able to get the model running successfully on a local Linux machine using the docker implementation of fitLins, so I suspect these errors are related to something with how the jobs are scheduled using apptainer/SLURM. Thanks for your help and advice! Original message below…

Thanks again for your help @effigies! I think I have gotten the dataset-level model running (with this model for reference):

{
  "Name": "MixedMotivation",
  "BIDSModelVersion": "1.0.0",
   "Description":"",
   "Input":{
      "task": ["ADAPTAVOID"]
   },
   "Nodes":[
      {
         "Level":"Run",
         "Name":"run_level",
         "GroupBy": ["run", "subject"],
         "Transformations":{
            "Transformer": "pybids-transforms-v1",
            "Instructions":[
               {
                  "Name":"Factor",
                  "Input":[
                     "trial_type"
                  ]
               },
               {
                  "Name":"Convolve",
                  "Input":[
                    "trial_type.GoAppetitive",
                    "trial_type.GoAversive",
                    "trial_type.NoGoAppetitive",
                    "trial_type.NoGoAversive"
                  ],
                  "Model":"spm"
               }
            ]
         },
         "Model":{
            "X":[
              1,
              "trial_type.GoAppetitive",
              "trial_type.GoAversive",
              "trial_type.NoGoAppetitive",
              "trial_type.NoGoAversive",
              "framewise_displacement",
              "trans_x",
              "trans_y",
              "trans_z",
              "rot_x",
              "rot_y",
              "rot_z"
            ],
           "Type": "glm"
         },
         "Contrasts":[
            {
               "Name":"appetitive_vs_aversive_go",
               "ConditionList":[
                  "trial_type.GoAppetitive",
                  "trial_type.GoAversive"
               ],
               "Weights":[
                  1,
                  -1
               ],
               "Test":"t"
            }
         ]
      },
     {
       "Level": "Dataset",
       "Name": "t-test",
       "GroupBy": [
         "contrast"
       ],
       "Model": {
         "X": [
           1
         ],
         "Type": "glm"
       },
       "DummyContrasts": {
         "Test": "t"
       }
     }
   ]
}

However, I am noticing that this type of output keeps occurring, along with the software appearing to hang for as much as 10 hours without crashing:

[MultiProc] Running 16 tasks, and 218 jobs ready. Free memory (GB): 120.77/168.77, Free processors: 0/16.
                     Currently running:
                       * _l1_model31
                       * _l1_model30
                       * _l1_model29
                       * _l1_model28
                       * _l1_model27
                       * _l1_model26
                       * _l1_model25
                       * _l1_model24
                       * _l1_model23
                       * _l1_model22
                       * _l1_model21
                       * _l1_model20
                       * _l1_model19
                       * _l1_model18
                       * _l1_model17
                       * _l1_model16

And a corresponding error (that appears for each process):

exception calling callback for <Future at 0x7fa4369a4190 state=finished raised BrokenProcessPool>
Traceback (most recent call last):
...
  File "/opt/miniconda-latest/envs/neuro/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I am currently running fitLins as a SLURM job with 16 cores allocated (using --n-cpus). I got the same kinds of outputs when trying to run the task at the run or subject level, without the dataset-level node. Do I need to allocate the same number of cores as jobs all at once, or are jobs supposed to cycle and free up unused cores once the operation is complete?