Encoding of Neurovault CURL output

1

See https://github.com/jeroen/curl/issues/160. The error is related to neurovault by https://ci.appveyor.com/project/neuroconductor-releases/neurovault/build/job/gmb9iv0f0oy6sbju#L1441

curl --head -v https://neurovault.org/media/images/264/JHU-ICBM-labels-2mm.nii.gz
*   Trying 54.149.190.168...
* TCP_NODELAY set
* Connected to neurovault.org (54.149.190.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=neurovault.org
*  start date: Oct 26 15:18:09 2019 GMT
*  expire date: Jan 24 15:18:09 2020 GMT
*  subjectAltName: host "neurovault.org" matched cert's "neurovault.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fecf1000000)
> HEAD /media/images/264/JHU-ICBM-labels-2mm.nii.gz HTTP/2
> Host: neurovault.org
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< server: nginx/1.15.6
server: nginx/1.15.6
< date: Wed, 11 Dec 2019 03:47:14 GMT
date: Wed, 11 Dec 2019 03:47:14 GMT
< content-type: application/octet-stream
content-type: application/octet-stream
< content-length: 9611
content-length: 9611
< last-modified: Tue, 16 Dec 2014 00:57:24 GMT
last-modified: Tue, 16 Dec 2014 00:57:24 GMT
< content-encoding: utf-8
content-encoding: utf-8
< strict-transport-security: max-age=31536000
strict-transport-security: max-age=31536000
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-allow-credentials: true
access-control-allow-credentials: true
< access-control-allow-methods: GET, POST, OPTIONS
access-control-allow-methods: GET, POST, OPTIONS
< access-control-allow-headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
access-control-allow-headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type

< 
* Connection #0 to host neurovault.org left intact

where you see content-encoding: utf-8, which apparently is not a correct content encoding based on https://github.com/jeroen/curl/issues/160#issuecomment-396287899, which notes https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding.

What can be done with problems such as these’

2

It seems to be working with wget https://neurovault.org/media/images/264/JHU-ICBM-labels-2mm.nii.gz which might provide a workaround.

If you can find the root of the problem - please sent a Pull Request to https://github.com/NeuroVault/NeuroVault

3

I agree that wget works, but it pretty much cURL does not work with it, which is what httr in R uses, which is what my package neurovault relies on. It only happens on Windows cURL which is odd: https://ci.appveyor.com/project/muschellij2/neurovault/builds/29473096#L731 (with full header info here: https://ci.appveyor.com/project/muschellij2/neurovault/builds/29472976#L730) Otherwise, this package won’t work with windows for Neurovault, but it seems like the options would be:

  1. Change the encoding on the file directly.
  2. Change all headers to have Content-Encoding: deflate or Content-Encoding: gzip for these files.
4

THis is also a case for images such as:
https://neurovault.org/media/images/77/zstat-young.nii.gz, which you can see with

curl -v -s https://neurovault.org/media/images/77/zstat-young.nii.gz 1> /dev/null
*   Trying 54.149.190.168...
* TCP_NODELAY set
* Connected to neurovault.org (54.149.190.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [220 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [98 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2573 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=neurovault.org
*  start date: Oct 26 15:18:09 2019 GMT
*  expire date: Jan 24 15:18:09 2020 GMT
*  subjectAltName: host "neurovault.org" matched cert's "neurovault.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc982800800)
> GET /media/images/77/zstat-young.nii.gz HTTP/2
> Host: neurovault.org
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< server: nginx/1.15.6
< date: Wed, 11 Dec 2019 16:53:10 GMT
< content-type: application/octet-stream
< content-length: 537724
< last-modified: Mon, 04 Aug 2014 09:04:35 GMT
< content-encoding: utf-8
< strict-transport-security: max-age=31536000
< access-control-allow-origin: *
< access-control-allow-credentials: true
< access-control-allow-methods: GET, POST, OPTIONS
< access-control-allow-headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
< 
{ [8038 bytes data]
* Connection #0 to host neurovault.org left intact