Miniconda unicode error in latest fmriprep docker push

1

I was running the April docker push as a singularity container on a HPC and it was working well. I just pulled the new docker container and it’s throwing this internal miniconda error. The singularity call is below in case I’m doing something wrong (and let me know if you want a tree structure or anything). It doesn’t seem to throw the unicode error if I run with the --anat-only flag on the cluster or if I run the same image as a docker container locally. Any idea what’s amiss?

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==0.5.3', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 157, in main
    create_workflow(opts)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 264, in create_workflow
    ignore_aroma_err=opts.ignore_aroma_denoising_errors)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 67, in init_fmriprep_wf
    ignore_aroma_err=ignore_aroma_err)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/base.py", line 158, in init_single_subject_wf
    ignore_aroma_err=ignore_aroma_err)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/workflows/epi.py", line 71, in init_func_preproc_wf
    metadata = layout.get_metadata(bold_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/bids_layout.py", line 44, in get_metadata
    param_dict = json.load(open(json_file_path, "r"))
  File "/usr/local/miniconda/lib/python3.6/json/__init__.py", line 296, in load
    return loads(fp.read(),
  File "/usr/local/miniconda/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 37: invalid start byte
srun: error: node027: task 0: Exited with exit code 1


PYTHONPATH="" \
singularity run /software/poldracklab_fmriprep_latest-2017-07-18-ce203ed0021e.img \
--participant_label ${participant} \
--nthreads 16 \
--omp-nthreads 16 \
--mem_mb 200000 \
--ignore slicetiming \
--bold2t1w-dof 9 \
--output-space template \
--template MNI152NLin2009cAsym \
--skull-strip-ants \
--write-graph \
-w ${exp_path}${workdir} \
${exp_path}${sourcedir} \
${exp_path}${outdir} \
participant
2

Could it possibly be that one of your JSON sidecar files is not encoded in utf-8?

3

No obvious encoding issues based on the snippet of code below.

shopt -s globstar
for f in **/*.json; do
    echo "${f}"
    tr -d \\000-\\177 < "${f}" | wc -c # test if only ASCII characters
    iconv -f utf-8 -t ucs-4 < "${f}" >/dev/null # test for unexpected characters
done

I reset the local docker and pulled the container again. Now I’m getting a different conda error, this one about loading the conda libraries. Still doesn’t seem to be an issue with the local docker container. I’ll try building the singularity container a different way and see if that changes anything.

170719-03:02:48,430 workflow INFO:
         Executing: mask_reportlet ID: 25
170719-03:02:48,437 workflow INFO:
         Executing node mask_reportlet in dir: ~/work/fmriprep_wf/single_subject_008_wf/func_preproc_task_rest_run_001_wf/epi_hmc_wf/enhance_and_skullstrip_epi_wf/mask_reportlet
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.so or libmkl_def.so.
4
  1. To confirm when you run the same FMRIPREP version using Docker on the same data you don not get either of those errors?
  2. How are you creating the Singularity container?
5

  1. I should test more locally before I say definitively, but I have yet to see either locally.

  2. Just with docker2singularity. I can try bootstraping the image or using singularity to import the docker container as well.

    docker run --privileged -t --rm
    -v /var/run/docker.sock:/var/run/docker.sock
    -v ~/Downloads:/output
    singularityware/docker2singularity
    poldracklab/fmriprep:latest

I had not see the mkl errors with the previous version, but they don’t stop the pype execution, just the report generation (I think).

6

Is the LD_LIBRARY_PATH variable set on the system running singularity?

I wonder if this is related: https://github.com/singularityware/singularity/issues/795#issuecomment-314235821

7

good shout, likely related at some level but I am definitely using the docker2singularity import already

fyi:

openmind7:~🐹  echo $LD_LIBRARY_PATH
/cm/shared/apps/gcc/4.8.4/lib:/cm/shared/apps/gcc/4.8.4/lib64

Running PYTHONPATH="" singularity exec docker://poldracklab/fmriprep:latest bash -c 'echo $LD_LIBRARY_PATH && ldconfig && fmriprep -v'
gives
/usr/lib/fsl/5.0::/.singularity.d/libs

But I get the same library error using this:

PYTHONPATH="" singularity exec docker://poldracklab/fmriprep:latest bash -c 'echo $LD_LIBRARY_PATH && ldconfig && echo $LD_LIBRARY_PATH && fmriprep --participant_label $0 --nthreads 16 --omp-nthreads 16 --mem_mb 200000 --ignore slicetiming --bold2t1w-dof 9 --output-space T1w --template MNI152NLin2009cAsym --skull-strip-ants --no-freesurfer --write-graph -w $1$2 $1$3 $1$4 participant' $participant $exp_path $workdir $sourcedir $outdir
e.g.

Docker image path: index.docker.io/poldracklab/fmriprep:latest
Creating container runtime...
/usr/lib/fsl/5.0::/.singularity.d/libs
/usr/lib/fsl/5.0::/.singularity.d/libs
...
170719-07:31:43,261 workflow INFO:
         [Job finished] jobname: apply_mask jobid: 24
170719-07:31:43,263 workflow INFO:
         Executing: mask_reportlet ID: 25
170719-07:31:43,272 workflow INFO:
         Executing node mask_reportlet in dir: ~/work/fmriprep_wf/single_subject_008_wf/func_preproc_task_rest_rec_acqrun007_run_001_wf/epi_hmc_wf/enhance_and_skullstrip_epi_wf/mask_reportlet
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.
Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.