Data I/O: Accessing the Cloud
This guide demonstrates how to interact with data using the supported cloud providers.
Goal
By following this guide, you will be able to access data from the cloud in the RKGMS database and vice versa using the different available options.
This guide complements the CSV Import/CSV Export and JSON Import/Export guides, where you can find all the relevant information for importing and exporting specific data types.
Cloud Storage Providers
Currently, Amazon S3 and Azure Blob Storage are supported.
Note that S3 data must be in buckets in the (N. Virginia) us-east-1
region.
Cloud Parameters
To interact with the cloud storage service, you need to define a config
module that specifies the data configuration. There are two relevant options for describing how to access the cloud:
Option | Description |
---|---|
path | A string that specifies the location and name of the file you want to import/export. Currently, this can point to s3://... (Amazon S3) or azure://... (Microsoft Azure) URLs. |
integration | Credentials needed to access the data. |
Public Data
If the cloud storage location is publicly available (e.g.: this file), you only need to provide the path in the configuration option path
.
module config
def path = "s3://relationalai-documentation-public/csv-import/simple-import-4cols.csv"
end
For security reasons, "http://"
and "https://"
URL addresses are not supported.
Private Data
In order to access private data, you need to specify cloud credentials in integration
.
Currently, only private Azure Blob Storage is supported. To access it, you must provide a valid SAS token and URL address, corresponding to a valid Azure account.
The URL is provided via the path
option.
Within the integration
submodule, the following information needs to be provided:
- the cloud storage provider in the
provider
field, and - the access token as a string in the
credentials
field, along with the token name (:azure_sas_token
).
module config
def path = "azure://myaccount.blob.core.windows.net/sascontainer/myfile.csv"
module integration
def provider = "azure"
def credentials = (:azure_sas_token, "sv=2014-02-14&sr=b&si=TestPolicy&sig=o%2B5%2F0C%2BLm7tWWftNKvQEGKHlSt%2Bfs8No7FZkUk5T%2Bv0%3D")
end
end
See Also
Now that you know how to access a service cloud provider, you can check our I/O how-to guides to learn how to interact with data. Check CSV Import and CSV Export for CSV data. If you are interested in JSON, go to JSON Import/Export.