SSL Error for MRIQC Singularity image

Suyash_Bhogawar · July 10, 2020, 2:57pm

i have converted mriqc docker image to singularity image but its giving me following SSL error.
I am not sure if i can add trusted domains like we do for pip install to conda env in mriqc docker image…?
Command:
unset PYTHONPATH; singularity run $HOME/images/mriqc_latest
$bids_root_dir $bids_root_dir/derivatives/mriqc/sub-${subj}
participant
–n_proc $nthreads
–hmc-fsl
–correct-slice-timing
–mem_gb $mem
–float32
–ants-nthreads $nthreads
-w $HOME/derivatives/mriqc/sub-${subj}
Running MRIQC on participant

200709-18:23:45,550 cli IMPORTANT:

    Running MRIQC version 0.15.2:
      * BIDS dataset path: /home/vhasfcbhogas/BIDS_tutorial.
      * Output folder: /home/vhasfcbhogas/BIDS_tutorial/derivatives/mriqc/sub-0tMLqrsbce.
      * Analysis levels: ['participant'].

200709-18:23:46,330 nipype.utils WARNING:
         Could not check for version updates:
Connection to server could not be made
200709-18:23:46,915 nipype.utils WARNING:
         Could not check for version updates:
Connection to server could not be made
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 444, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 356, in connect
    ssl_context=context)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 450, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/workflow.py", line 21, in build_workflow
    retval["workflow"] = init_mriqc_wf()
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/core.py", line 22, in init_mriqc_wf
    workflow.add_nodes([anat_qc_workflow()])
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 87, in anat_qc_workflow
    norm = spatial_normalization()
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 189, in spatial_normalization
    get_template(tpl_id, resolution=resolution, desc='brain', suffix='mask'))
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 67, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 178, in _s3_get
    r = requests.get(url, stream=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 511, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])")))

oesteban · July 10, 2020, 3:13pm

Hi @Suyash_Bhogawar,

There are a couple of reasons why this may be happening.

Real Internet access problems (or no access at all from your compute node)
Singularity attempting to download data to the $TEMPLATEFLOW_HOME folder, but it is not writeable.

Both issues are discussed here: https://www.nipreps.org/apps/singularity/#accessing-the-hosts-filesystem (for the case of fMRIPrep, but it works the same). You may also find interesting reading after the following Q&A: https://fmriprep.org/en/stable/faq.html#what-is-templateflow-for

HTH
Cheers,
Oscar

Suyash_Bhogawar · July 10, 2020, 9:08pm

Thank you Oscar! @oesteban

I tried both the ways as mentioned here- https://www.nipreps.org/apps/singularity/#accessing-the-hosts-filesystem

In my case, we definitely have internet connection but because i am working behind the VA firewall, our ssl certs are signed by VA proxy. Hence i tried downloading all the templates beforehand as suggested to ~/templateflow folder.
However i am getting same error… Probably there is mistake in mounting templateflow path.
My command line is:
unset PYTHONPATH; singularity run -B ${TEMPLATEFLOW_HOME:-$HOME/.cache/templateflow}:/home/vhasfcbhogas/templateflow
–cleanenv $HOME/images/mriqc_latest
$bids_root_dir $bids_root_dir/derivatives/mriqc/sub-${subj}
participant
–n_proc $nthreads
–hmc-fsl
–correct-slice-timing
–mem_gb $mem
–float32
–ants-nthreads $nthreads
-w $HOME/derivatives/mriqc/sub-${subj}
error:

200710-13:40:52,506 cli IMPORTANT:

Running MRIQC version 0.15.2:
  * BIDS dataset path: /home/vhasfcbhogas/BIDS_tutorial.
  * Output folder: /home/vhasfcbhogas/BIDS_tutorial/derivatives/mriqc/sub-0tMLqrsbce.
  * Analysis levels: ['participant'].

200710-13:40:53,176 nipype.utils WARNING:
     Could not check for version updates:
Connection to server could not be made
200710-13:40:53,764 nipype.utils WARNING:
     Could not check for version updates:
Connection to server could not be made
Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 444, in wrap_socket
cnx.do_handshake()
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1907, in do_handshake
self._raise_ssl_error(self._ssl, result)
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
_raise_current_error()
  File "/usr/local/miniconda/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
self._validate_conn(conn)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
conn.connect()
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 356, in connect
ssl_context=context)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 450, in wrap_socket
raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/cli/workflow.py", line 21, in build_workflow
retval["workflow"] = init_mriqc_wf()
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/core.py", line 22, in init_mriqc_wf
workflow.add_nodes([anat_qc_workflow()])
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 87, in anat_qc_workflow
norm = spatial_normalization()
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 189, in spatial_normalization
get_template(tpl_id, resolution=resolution, desc='brain', suffix='mask'))
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 67, in get
_s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 178, in _s3_get
r = requests.get(url, stream=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 72, in get
return request('get', url, params=params, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 511, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')])")))