Skip to content

Multiple Bundle Stores (in development)

Unreleased feature

This feature is still in development. It does not fully work yet and will only work once #3710 is resolved.

CodaLab allows you to specify multiple bundle stores.

Workflow steps

This workflow describes how to set up a bundle store on GCP.

To create a bundle store, run the following:

cl store add --name store1 --storage-type disk --storage-format uncompressed

You can list bundle stores by running

cl store ls

You can also delete a bundle store by running

cl store rm [bundle store uuid]

Steps to test locally with Azurite

First, make sure you start Azurite locally by running codalab-service start -bds default azurite. Then run:

cl store add --name blob --url azfs://devstoreaccount1/bundles
cl upload --store blob mkdocs.yml
cl run --store blob "echo hello"

Steps to set up and test with GCP

First, make sure the CODALAB_GOOGLE_APPLICATION_CREDENTIALS env var is set on the server that runs the REST server. It should be set to a path to a credentials JSON file (see this page for more information on how to create it).

export CODALAB_GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
codalab-service start -bd

Then make a bucket on your GCP account (in this case, it's called ashwin123123123). You can then store bundles on GCP:

cl store add --name gcp2 --url gs://ashwin123123123
cl upload --store gcp2 mkdocs.yml
cl run --store gcp2 "echo hello"

Setting a default bundle store

Once a bundle store is created, you can restart the server and set the CODALAB_DEFAULT_BUNDLE_STORE_NAME environment variable. This will make bundles be uploaded, by default, to the specified bundle store.

CODALAB_DEFAULT_BUNDLE_STORE_NAME=blob codalab-service start ...

Todo (unreleased features)

  • cl replicate
  • Later: by default it should always use Blob / GCS instead of disk if someone just runs cl upload mkdocs.yml.