Datalad - setting up a special remote with google cloud platform storage bucket

datalad

#1

Hi,

Having issues publishing my datalad datasets to google cloud platform as special remotes.

Here are the steps I’ve taken:

1.) Obtain an interoperability Access Key and Secret from this link. And select a project you would like to use as a special remote. Next select the the ‘Interoperability’ tab and write down the access key and associated secret.

2.) Export these into your bash environment
export AWS_ACCESS_KEY_ID="YOUR-KEY"
export AWS_SECRET_ACCESS_KEY="YOUR-SECRET"

3.) Now cd into this repository and
initialize a special remote using git annex.

git annex initremote <pickaname> type=S3 encryption=none host=storage.googleapis.com port=80 chunk=10Mb bucket=neuroyouths fileprefix=annex

4.) git annex enableremote <pickaname>

5.) git annex testremote <pickaname> fails with this repeating error log

git annex sync does nothing…

Halp please :cold_sweat:

testremote gcp (generating test keys...) Remote Tests unavailable remote removeKey: OK (0.09s) storeKey: HttpExceptionRequest Request { host = "gcp-7382bf3d-14cb-4167-aa7b-4d4f58ad2c9c.!dne!" port = 80 secure = False requestHeaders = [("Date","Mon, 02 Jul 2018 13:23:50 GMT"),("Authorization","<REDACTED>"),("x-amz-storage-class","STANDARD")] path = "/annexSHA256E-s1048576-S10000000-C1--1aa2757d35343fbdf47c07d8ecdef5eb100b4b044edfc579207db1d66d027d29.this-is-a-test-key" queryString = "" method = "PUT" proxy = Nothing rawBody = False redirectCount = 10 responseTimeout = ResponseTimeoutDefault requestVersion = HTTP/1.1 } (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "gcp-7382bf3d-14cb-4167-aa7b-4d4f58ad2c9c.!dne!", service name: Just "80"): does not exist (nodename nor servname provided, or not known)) OK (0.15s) checkPresent: OK (0.08s) retrieveKeyFile: OK (0.08s) retrieveKeyFileCheap: OK key size Just 1048576; NoChunks; encryption none removeKey when not present: FAIL (0.21s) ./Command/TestRemote.hs:172: failed


#2

Could never get GCS to work with datalad. So I wrote some hooks to sync a repository with google cloud storage.

Check out the solution here