Connect to S3
Collect the connection details and certificates
To connect to S3 buckets through the Cyral sidecar, follow these steps:
Get the sidecar endpoint address and port number used for S3 connections:
- As a Cyral admin, open the Cyral control plane UI.
- Click Sidecars and click on the name of your sidecar.
- Copy the sidecar host from the top of the page, right below the sidecar name.
- The port can be collected from the Bindings tab.
- As an end-user:
Go to the Cyral data access portal and select the desired S3 repository.
Follow the steps to configure the AWS CLI with the Cyral CLI.
Open the AWS configuration file, find the profile created (or updated) by the Cyral CLI and copy the proxy endpoint, as illustrated below:
[profile <your-profile-name>]
ca_bundle = /home/<user>/.aws/cyral_ca_bundle.pem
s3 =
proxy = http://<sidecar-host>:<port>
s3api =
proxy = http://<sidecar-host>:<port>
- As a Cyral admin, open the Cyral control plane UI.
Download the certificate authority (CA) bundle to verify TLS/SSL connections.
The data traffic is encrypted with the sidecar's own certificate. To properly operate, client applications should use the sidecar's CA bundle to verify TLS/SSL connections. This step is optional, but recommended. Client applications can also be configured to not verify the Cyral-provided TLS/SSL certificates.
In case you already made the AWS CLI configuration using the Cyral CLI, the CA bundle file should be already present in your local machine. Its location is specified in the AWS configuration file for the profile you created with the Cyral CLI tool, referenced by the
ca_bundle
key. Please check the example configuration payload above for a reference to the CA bundle file.Alternatively, you can ask your admin to provide you the CA bundle to use with your encrypted connections.
Connect to S3
Make sure you have followed the steps described above.
Connect to S3 using your preferred client. User authentication relies on the credentials you collected from the Cyral Control Plane UI. If present, Cyral policies are enforced. All traffic is sent through the Cyral sidecar, which logs all data activity to the log location configured in your Cyral installation.
- Python SDK
- Ruby SDK
- Java SDK
- NodeJS SDK
- Golang SDK
- CLI + Plugins
- CLI
This section describes the manual steps to configure S3 access through the AWS CLI without any extra Python plugin. This can be used as an alternative to the AWS CLI configuration flow using the Cyral CLI tool.
Store the endpoint address as the proxy address:
In a shell session, set the
HTTP_PROXY
andHTTPS_PROXY
environment variables to the hostname and port of your sidecar. For example, for a sidecar reachable athttp://mysidecar.example.com
, and a S3 repository bound to it at port 453, we might assign:export HTTP_PROXY=http://mysidecar.example.com:453
export HTTPS_PROXY=http://mysidecar.example.com:453If your sidecar runs as an AWS EC2 instance, you must also set the
NO_PROXY
variable to the IP address of the AWS EC2 instance metadata service, 169.254.169.254. This address does not vary.export NO_PROXY=169.254.169.254
Detailed information and settings for configuring the proxy endpoint for different systems is available in the AWS documentation section, Using an HTTP proxy
In your AWS credentials file, add a profile containing a valid (
aws_access_key_id
) and secret key (aws_secret_access_key
) associated with the IAM username that you'll use to connect to S3. This file is usually found at~/.aws/credentials
.Example credential file with native S3 IAM credentials:
[profile mysidecar]
aws_access_key_id=AKIAIOSFODNCYRALEXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYRALEXAMPLEKEYExample credential file with a Cyral Access Token:
[profile mysidecar]
aws_access_key_id=<user>:<access-token>:<role>
aws_secret_access_key=nonetip
By default, for native credentials, Cyral treats the
aws_access_key_id
as the username for logging and policy enforcement.Access S3 with the previously created profile:
aws s3 --region us-east-1 --profile mysidecar list-tables
This section describes the manual steps to configure S3 access through the AWS CLI
using the awscli-plugin-proxy
Python plugin. This can be used as an alternative
to the AWS CLI configuration flow using the Cyral CLI tool.
- Install the plugin,
awscli-plugin-proxy
, using python pip:
python3 -m pip install awscli-plugin-proxy --user
If you are using the AWS CLI v2 (
aws --version
), you will need to define the appropriate location for the plugin:a. Get the plugin location:
python -m pip show awscli-plugin-proxy
It is expected to have a Location key in the generated output, for example:
Linux: Location: /home/<username>/.local/lib/python3.8/site-packages
Mac: Location: /Users/<username>/.local/lib/python3.8/site-packages
Windows: Location: c:\Users\<username>\Application Data\python\python38\site-packagesb. Set the plugin location with the command:
aws configure set plugins.cli_legacy_plugin_path <LOCATION>
which for the example Mac output above, would be:
aws configure set plugins.cli_legacy_plugin_path /Users/<username>/.local/lib/python3.8/site-packages
- Configure using the AWS CLI:
aws configure set plugins.s3-proxy awscli_plugin_s3_proxy
- Add the sidecar as a proxy for S3:
aws configure --profile aws_sidecar_plugin_profile set s3.proxy http://<YOUR_SIDECAR_ADDRESS>:<PORT>
replacing <YOUR_SIDECAR_ADDRESS>
with the actual address.
- Add the Cyral CA bundle to the same profile:
aws configure --profile aws_sidecar_plugin_profile set ca_bundle /path/to/cyral_ca_bundle.pem
- Connect to S3:
aws s3 --region us-east-1 --profile aws_sidecar_plugin_profile list-tables
Connect using the AWS SDK for Ruby:
Set http_proxy
to the hostname of your
sidecar and the port where the S3 repository is
bound. This example assumes a sidecar is reachable at
example-sidecar-endpoint.com
, and a S3
repository bound to it at port 453.
Set ssl_ca_bundle
to the path to the Cyral CA bundle.
require 'aws-sdk-s3'
def sidecar_example
sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
sidecar_endpoint = 'example-sidecar-endpoint.com'
sidecar_port = 453
region = 'us-east-2'
# static credentials:
credentials = Aws::Credentials.new('YOUR_AWS_ACCESS_KEY_ID', 'YOUR_AWS_SECRET_ACCESS_KEY')
# alternative credential configuration available at:
# https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html
s3_client = Aws::S3::Client.new(
credentials: credentials,
region: region,
http_proxy: sprintf("http://%s:%d", sidecar_endpoint, sidecar_port),
ssl_ca_bundle: sidecar_certificate_bundle,
)
puts 'Running List Tables command through the sidecar'
result = s3_client.list_tables()
puts result
end
sidecar_example if $PROGRAM_NAME == __FILE__
Alternatively, the following environment variables are applicable to the Ruby AWS SDK
http_proxy
, https_proxy
, and AWS_CA_BUNDLE
. If a shell session is configured with
these variables, all data traffic will go through the sidecar, potentially including
traffic not related to S3 itself, so the code snipped presented above is preferable
in favor of environment variables.
Connect using the AWS SDK for Python:
In the proxies
configuration block (in the code below), specify the host and port of the
sidecar in the format http://<sidecar_endpoint>:<sidecar_port>
for both http
and https
keys. The scheme in the URL must be http
. The final connection between client applications
and the S3 servers will still be TLS encrypted.
For the example code below, it is assumed that the sidecar host and port are
example-sidecar-endpoint.com:453
.
Set verify
to the path of the Cyral CA bundle.
import boto3
from botocore.config import Config
# reference AWS documentation:
# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
def sidecar_example():
sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
sidecar_endpoint = 'example-sidecar-endpoint.com'
sidecar_port = 453
region = 'us-east-2'
s3_client = boto3.client('s3',
aws_access_key_id="YOUR_AWS_ACCESS_KEY_ID",
aws_secret_access_key="YOUR_AWS_SECRET_ACCESS_KEY",
region_name=region,
verify=sidecar_certificate_bundle,
config=Config(
proxies={
"http": f"http://{sidecar_endpoint}:{sidecar_port}",
"https": f"http://{sidecar_endpoint}:{sidecar_port}",
}),
)
print('Running List Tables command through the sidecar')
result = s3_client.list_tables()
print(result)
if __name__ == "__main__":
sidecar_example()
Alternatively, the following environment variables are applicable to the Python AWS SDK
HTTP_PROXY
, HTTPS_PROXY
, and AWS_CA_BUNDLE
. If a shell session is configured with
these variables, all data traffic will go through the sidecar, potentially including
traffic not related to S3 itself, so the code snipped presented above is preferable
in favor of environment variables.
It is recommended to keep certificate validation always enabled. However, it can be
temporarily disabled, to perform connectivity validation, by setting verify=False
.
Connect using the AWS SDK for NodeJS:
In HttpsProxyAgent
, specify the hostname of your
sidecar, the port where the S3 repository is
bound, and the certificate file.
import { S3Client, ListTablesCommand } from '@aws-sdk/client-s3';
import { NodeHttpHandler } from '@aws-sdk/node-http-handler';
import { HttpsProxyAgent } from 'hpagent';
import fs from 'fs';
import util from 'util';
const sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
const sidecar_endpoint = 'example-sidecar-endpoint.com'
const sidecar_port = 453
var region = 'us-east-2'
// load the certificate bundle file
const certs = [
fs.readFileSync(sidecar_certificate_bundle)
];
// create a proxy agent pointing to the sidecar that uses the custom certificate bundle
const agent = new HttpsProxyAgent({
proxy: util.format('http://%s:%d', sidecar_endpoint, sidecar_port),
ca: certs
});
(async () => {
const client = new S3Client({
region: region,
requestHandler: new NodeHttpHandler({
httpAgent: agent,
httpsAgent: agent
}),
});
const command = new ListTablesCommand({});
try {
console.log("Running List Tables command through the sidecar")
const results = await client.send(command);
console.log(results.TableNames.join("\n"));
} catch (err) {
console.error(err);
}
})();
A proxy agent is required for a proper configuration and can be installed with the command:
npm install hpagent --save
.
Different SDK versions might require different proxy modules. Please refer to the official
AWS documentation at: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-proxies.html
Connect using the AWS SDK for Java:
In ClientConfiguration
, specify the hostname of your
sidecar, the port where the S3 repository is
bound, and the proxy protocol, HTTP
.
This example assumes a sidecar is reachable at
example-sidecar-endpoint.com
, and a S3
repository bound to it at port 453.
In the AWS SDK for JAVA, the certificate bundle needs to be directly included in the JVM certificate trust store.
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.services.s3v2.AmazonS3;
import com.amazonaws.services.s3v2.AmazonS3ClientBuilder;
import com.amazonaws.services.s3v2.document.S3;
import com.amazonaws.services.s3v2.document.Table;
import com.amazonaws.services.s3v2.document.TableCollection;
import com.amazonaws.services.s3v2.model.ListTablesResult;
public class ExampleSidecarConn {
static String sidecar_certificate_bundle = "/path/to/cyral_ca_bundle.pem";
static String sidecar_endpoint = "example-sidecar-endpoint.com";
static Integer sidecar_port = 443; // port selected when binding the s3 repo to the sidecar
static String region = "us-east-2";
public static void main(String[] args) throws Exception {
BasicAWSCredentials basic = new BasicAWSCredentials("YOUR_AWS_ACCESS_KEY_ID", "YOUR_AWS_SECRET_ACCESS_KEY");
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(basic);
// sidecar proxy configuration
ClientConfiguration config = new ClientConfiguration();
config.setProxyHost(sidecar_endpoint);
config.setProxyPort(sidecar_port);
config.setProxyProtocol(Protocol.HTTP);
AmazonS3 client = AmazonS3ClientBuilder.
standard().
withCredentials(credentials).
withRegion(region).
withClientConfiguration(config).build();
S3 dynamoDB = new S3(client);
System.out.println("Running List Tables command through the sidecar");
TableCollection<ListTablesResult> result = dynamoDB.listTables();
for (Table a: result) {
System.out.println(a.getTableName());
}
}
}
Connect using the AWS SDK for Golang:
package main
import (
"context"
"fmt"
"log"
"net/http"
"net/url"
"os"
awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
// For more details, please refer to the official AWS documentation:
// https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/custom-http/#configuring-a-proxy
const (
sidecar_certificate_bundle = "/path/to/cyral_ca_bundle.pem"
sidecar_endpoint = "example-sidecar-endpoint.com"
sidecar_port = 453 // port selected when binding the s3 repo to the sidecar
region = "us-east-2"
)
func assertNil(err error) {
if err != nil {
log.Fatal(err)
}
}
func main() {
httpClient := awshttp.NewBuildableClient().WithTransportOptions(func(tr *http.Transport) {
proxyURL, err := url.Parse(fmt.Sprintf("http://%s:%d", sidecar_endpoint, sidecar_port))
assertNil(err)
tr.Proxy = http.ProxyURL(proxyURL)
})
ctx := context.TODO()
certReader, err := os.Open(sidecar_certificate_bundle)
assertNil(err)
cfg, err := config.LoadDefaultConfig(
ctx,
config.WithRegion(region),
config.WithHTTPClient(httpClient),
config.WithCustomCABundle(certReader),
)
assertNil(err)
client := s3.NewFromConfig(cfg)
log.Println("Running List Tables command through the sidecar")
result, err := client.ListTables(ctx, &s3.ListTablesInput{})
assertNil(err)
log.Println(result)
}
Alternatively, the following environment variables are applicable to the Golang AWS SDK
HTTP_PROXY
, HTTPS_PROXY
, and AWS_CA_BUNDLE
. If a shell session is configured with
these variables, all data traffic will go through the sidecar, potentially including
traffic not related to S3 itself, so the code snipped presented above is preferable
in favor of environment variables.
Note on other AWS SDKs
AWS offers SDKs for a wide range of programming languages, while here we only show configuration examples for a subset of them. If examples for your programming language are missing from this page, we encourage you to either look for examples in AWS official documentation at https://aws.amazon.com/developer/tools/, or contact Cyral support for help.