FileNotFoundError after check_for_s3

Software Support
1

Hello everyone,

I’m running into an issue with C-PAC (v1.8.7.dev1) for pre-processing and analyzing data in BIDS format using Singularity (v3.11.9) on our HPC. The pipeline and data configuration file worked smoothly on my local machine, successfully completing with a few test subjects.

However, when I try to execute it on our HPC using Singularity, I encounter the following error across all subjects, causing the pipeline to crash (the issue also affects func files):

Node: cpac_sub-X_ses-X.anat_T1w_gather_sub-X_ses-X.check_for_s3
Working directory: /outputs/working/pipeline_cpac-pipeline/cpac_sub-X_ses-X/anat_T1w_gather_sub-X_ses-X/check_for_s3

Node inputs:

creds_path = None
dl_dir = /outputs/working/pipeline_cpac-pipeline
file_path = /path/to/BIDS/site/sub-X/ses-X/anat/sub-X_ses-X_run-X_T1w.nii.gz
function_str = CPAC.utils.datasource.check_for_s3
img_type = anat

Traceback (most recent call last):
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 47, in run
    node.run(updatehash=updatehash)
  File "/code/CPAC/pipeline/nipype_pipeline_engine/engine.py", line 443, in run
    return super().run(updatehash)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/share/fsl/6.0/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node check_for_s3.

Traceback:
	Traceback (most recent call last):
	  File "/code/CPAC/pipeline/nipype_pipeline_engine/monkeypatch.py", line 40, in run
	    runtime = self._run_interface(runtime)
	  File "/code/CPAC/utils/interfaces/function/function.py", line 216, in _run_interface
	    out = function_handle(**args)
	  File "/code/CPAC/utils/datasource.py", line 851, in check_for_s3
	    raise FileNotFoundError(f'File {local_path} does not exist!')

FileNotFoundError: File /path/to/BIDS/site/sub-X/ses-X/anat/sub-X_ses-X_run-X_T1w.nii.gz does not exist!

However, the file does exist in the specified path, and I have read/write permissions in all related directories.

I’ve tried several adjustments to the Singularity command without success. My latest attempt was this:

singularity run \
-B /path/to/BIDS:/bids_dataset \
-B /path/to/output:/outputs \
-B /path/to/configs:/configs \
-B /path/to/scratch/space:/outputs/working \
c-pac.sif /bids_dataset /outputs participant --pipeline-file /configs/pipeline_config.yml --data-config-file /configs/data_config_file.yml

If anyone has encountered a similar issue or has any suggestions, your guidance would be much appreciated!

Thank you in advance!

2

Hi @luisagulleiro , it looks like this error is coming from the input path you are giving C-PAC. Can you try the following command and let me know how it goes?

singularity run \
  --cleanenv \
  -B "$INPUT":"$INPUT":ro \
  -B "$OUTPUT":"$OUTPUT" \
 "$IMAGE" \
  "$INPUT" \
  "$OUTPUT" \
  "$ANALYSIS" \
  --skip_bids_validator \
  --n_cpus 1 \
  --mem_gb 7.0 \
  --participant_label "$PARTICIPANT" \
  --pipeline-file "$CONFIG"

INPUT = path to your BIDS dataset
OUTPUT = path to your desired output folder (make sure it exists)
IMAGE = the the cpac .sif image you are using
ANALYSIS = analysis level (in your use case, “participant”)
CONFIG = the .yml config file

let me know if I can clarify further or you have any questions!
-Tamsin