Datalad - setting up a special remote with google cloud platform storage bucket




Having issues publishing my datalad datasets to google cloud platform as special remotes.

Here are the steps I’ve taken:

1.) Obtain an interoperability Access Key and Secret from this link. And select a project you would like to use as a special remote. Next select the the ‘Interoperability’ tab and write down the access key and associated secret.

2.) Export these into your bash environment

3.) Now cd into this repository and
initialize a special remote using git annex.

git annex initremote <pickaname> type=S3 encryption=none port=80 chunk=10Mb bucket=neuroyouths fileprefix=annex

4.) git annex enableremote <pickaname>

5.) git annex testremote <pickaname> fails with this repeating error log

git annex sync does nothing…

Halp please :cold_sweat:

testremote gcp (generating test keys...) Remote Tests unavailable remote removeKey: OK (0.09s) storeKey: HttpExceptionRequest Request { host = "gcp-7382bf3d-14cb-4167-aa7b-4d4f58ad2c9c.!dne!" port = 80 secure = False requestHeaders = [("Date","Mon, 02 Jul 2018 13:23:50 GMT"),("Authorization","<REDACTED>"),("x-amz-storage-class","STANDARD")] path = "/annexSHA256E-s1048576-S10000000-C1--1aa2757d35343fbdf47c07d8ecdef5eb100b4b044edfc579207db1d66d027d29.this-is-a-test-key" queryString = "" method = "PUT" proxy = Nothing rawBody = False redirectCount = 10 responseTimeout = ResponseTimeoutDefault requestVersion = HTTP/1.1 } (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "gcp-7382bf3d-14cb-4167-aa7b-4d4f58ad2c9c.!dne!", service name: Just "80"): does not exist (nodename nor servname provided, or not known)) OK (0.15s) checkPresent: OK (0.08s) retrieveKeyFile: OK (0.08s) retrieveKeyFileCheap: OK key size Just 1048576; NoChunks; encryption none removeKey when not present: FAIL (0.21s) ./Command/TestRemote.hs:172: failed


Could never get GCS to work with datalad. So I wrote some hooks to sync a repository with google cloud storage.

Check out the solution here