Connect to S3

Collect the connection details and certificates

To connect to S3 buckets through the Cyral sidecar, follow these steps:

1. Get the sidecar endpoint address and port number used for S3 connections:

   • As a Cyral admin, open the Cyral control plane UI.
     • Click Sidecars and click on the name of your sidecar.
     • Copy the sidecar host from the top of the page, right below the sidecar name.
     • The port can be collected from the Bindings tab.
   • As an end-user:

     • Go to the Cyral data access portal and select the desired S3 repository.

     • Follow the steps to configure the AWS CLI with the Cyral CLI.

     • Open the AWS configuration file, find the profile created (or updated) by the Cyral CLI and copy the proxy endpoint, as illustrated below:

       [profile <your-profile-name>]
       ca_bundle = /home/<user>/.aws/cyral_ca_bundle.pem
       s3 = 
            proxy = http://<sidecar-host>:<port>
       s3api = 
            proxy = http://<sidecar-host>:<port>

2. Download the certificate authority (CA) bundle to verify TLS/SSL connections.

   The data traffic is encrypted with the sidecar's own certificate. To properly operate, client applications should use the sidecar's CA bundle to verify TLS/SSL connections. This step is optional, but recommended. Client applications can also be configured to not verify the Cyral-provided TLS/SSL certificates.

   • In case you already made the AWS CLI configuration using the Cyral CLI, the CA bundle file should be already present in your local machine. Its location is specified in the AWS configuration file for the profile you created with the Cyral CLI tool, referenced by the ca_bundle key. Please check the example configuration payload above for a reference to the CA bundle file.

   • Alternatively, you can ask your admin to provide you the CA bundle to use with your encrypted connections.

Connect to S3

1. Make sure you have followed the steps described above.

2. Connect to S3 using your preferred client. User authentication relies on the credentials you collected from the Cyral Control Plane UI. If present, Cyral policies are enforced. All traffic is sent through the Cyral sidecar, which logs all data activity to the log location configured in your Cyral installation.

Python SDK

This section describes the manual steps to configure S3 access through the AWS CLI without any extra Python plugin. This can be used as an alternative to the AWS CLI configuration flow using the Cyral CLI tool.

1. Store the endpoint address as the proxy address:

   • In a shell session, set the HTTP_PROXY and HTTPS_PROXY environment variables to the hostname and port of your sidecar. For example, for a sidecar reachable at http://mysidecar.example.com, and a S3 repository bound to it at port 453, we might assign:

     export HTTP_PROXY=http://mysidecar.example.com:453
     export HTTPS_PROXY=$HTTP_PROXY

   • If your sidecar runs as an AWS EC2 instance, you must also set the NO_PROXY variable to the IP address of the AWS EC2 instance metadata service, 169.254.169.254. This address does not vary.

     export NO_PROXY=169.254.169.254

   Detailed information and settings for configuring the proxy endpoint for different systems is available in the AWS documentation section, Using an HTTP proxy

2. In your AWS credentials file, add a profile containing a valid (aws_access_key_id) and secret key (aws_secret_access_key) associated with the IAM username that you'll use to connect to S3. This file is usually found at ~/.aws/credentials.

   Example credential file with native S3 IAM credentials:

   [profile mysidecar]
   aws_access_key_id=AKIAIOSFODNCYRALEXAMPLE
   aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYRALEXAMPLEKEY

   Example credential file with a Cyral Access Token:

   [profile mysidecar]
   aws_access_key_id=<user>:<access-token>:<role>
   aws_secret_access_key=none

   tip

   By default, for native credentials, Cyral treats the aws_access_key_id as the username for logging and policy enforcement.

3. Access S3 with the previously created profile:

aws s3 --region us-east-1 --profile mysidecar list-tables

Ruby SDK

This section describes the manual steps to configure S3 access through the AWS CLI using the awscli-plugin-proxy Python plugin. This can be used as an alternative to the AWS CLI configuration flow using the Cyral CLI tool.

1. Install the plugin, awscli-plugin-proxy, using python pip:

python3 -m pip install awscli-plugin-proxy --user

2. If you are using the AWS CLI v2 (aws --version), you will need to define the appropriate location for the plugin:

   a. Get the plugin location: python -m pip show awscli-plugin-proxy

   It is expected to have a Location key in the generated output, for example:

   Linux: Location: /home/<username>/.local/lib/python3.8/site-packages
   Mac: Location: /Users/<username>/.local/lib/python3.8/site-packages
   Windows: Location: c:\Users\<username>\Application Data\python\python38\site-packages

   b. Set the plugin location with the command:

   aws configure set plugins.cli_legacy_plugin_path <LOCATION>

   which for the example Mac output above, would be:

   aws configure set plugins.cli_legacy_plugin_path /Users/<username>/.local/lib/python3.8/site-packages

3. Configure using the AWS CLI:

aws configure set plugins.s3-proxy awscli_plugin_s3_proxy

4. Add the sidecar as a proxy for S3:

aws configure --profile aws_sidecar_plugin_profile set s3.proxy http://<YOUR_SIDECAR_ADDRESS>:<PORT>

replacing <YOUR_SIDECAR_ADDRESS> with the actual address.

5. Add the Cyral CA bundle to the same profile:

aws configure --profile aws_sidecar_plugin_profile set ca_bundle /path/to/cyral_ca_bundle.pem

6. Connect to S3:

aws s3 --region us-east-1 --profile aws_sidecar_plugin_profile list-tables

Java SDK

Connect using the AWS SDK for Ruby:

Set http_proxy to the hostname of your sidecar and the port where the S3 repository is bound. This example assumes a sidecar is reachable at example-sidecar-endpoint.com, and a S3 repository bound to it at port 453.

Set ssl_ca_bundle to the path to the Cyral CA bundle.

require 'aws-sdk-s3'

def sidecar_example
    sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
    sidecar_endpoint = 'example-sidecar-endpoint.com'
    sidecar_port = 453

    region = 'us-east-2'

    # static credentials:
    credentials = Aws::Credentials.new('YOUR_AWS_ACCESS_KEY_ID', 'YOUR_AWS_SECRET_ACCESS_KEY')
    # alternative credential configuration available at:
    # https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html

    s3_client = Aws::S3::Client.new(
        credentials: credentials,
        region: region,
        http_proxy: sprintf("http://%s:%d", sidecar_endpoint, sidecar_port),
        ssl_ca_bundle: sidecar_certificate_bundle,
    )

    puts 'Running List Tables command through the sidecar'
    result = s3_client.list_tables()
    puts result
end

sidecar_example if $PROGRAM_NAME == __FILE__

Alternatively, the following environment variables are applicable to the Ruby AWS SDK http_proxy, https_proxy, and AWS_CA_BUNDLE. If a shell session is configured with these variables, all data traffic will go through the sidecar, potentially including traffic not related to S3 itself, so the code snipped presented above is preferable in favor of environment variables.

NodeJS SDK

Connect using the AWS SDK for Python:

In the proxies configuration block (in the code below), specify the host and port of the sidecar in the format http://<sidecar_endpoint>:<sidecar_port> for both http and https keys. The scheme in the URL must be http. The final connection between client applications and the S3 servers will still be TLS encrypted.

For the example code below, it is assumed that the sidecar host and port are example-sidecar-endpoint.com:453.

Set verify to the path of the Cyral CA bundle.

import boto3
from botocore.config import Config

# reference AWS documentation:
# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

def sidecar_example():
    sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
    sidecar_endpoint = 'example-sidecar-endpoint.com'
    sidecar_port = 453

    region = 'us-east-2'

    s3_client = boto3.client('s3',     
        aws_access_key_id="YOUR_AWS_ACCESS_KEY_ID",
        aws_secret_access_key="YOUR_AWS_SECRET_ACCESS_KEY",
        region_name=region,
        verify=sidecar_certificate_bundle,
        config=Config(
            proxies={
                "http": f"http://{sidecar_endpoint}:{sidecar_port}",
                "https": f"http://{sidecar_endpoint}:{sidecar_port}",
            }),
    )

    print('Running List Tables command through the sidecar')
    result = s3_client.list_tables()
    print(result)

if __name__ == "__main__":
    sidecar_example()

Alternatively, the following environment variables are applicable to the Python AWS SDK HTTP_PROXY, HTTPS_PROXY, and AWS_CA_BUNDLE. If a shell session is configured with these variables, all data traffic will go through the sidecar, potentially including traffic not related to S3 itself, so the code snipped presented above is preferable in favor of environment variables.

It is recommended to keep certificate validation always enabled. However, it can be temporarily disabled, to perform connectivity validation, by setting verify=False.

Golang SDK

Connect using the AWS SDK for NodeJS:

In HttpsProxyAgent, specify the hostname of your sidecar, the port where the S3 repository is bound, and the certificate file.

import { S3Client, ListTablesCommand } from '@aws-sdk/client-s3';
import { NodeHttpHandler } from '@aws-sdk/node-http-handler';
import { HttpsProxyAgent } from 'hpagent';
import fs from 'fs';
import util from 'util';

const sidecar_certificate_bundle = '/path/to/cyral_ca_bundle.pem'
const sidecar_endpoint = 'example-sidecar-endpoint.com'
const sidecar_port = 453
var region = 'us-east-2'

// load the certificate bundle file
const certs = [
    fs.readFileSync(sidecar_certificate_bundle)
];

// create a proxy agent pointing to the sidecar that uses the custom certificate bundle
const agent = new HttpsProxyAgent({
    proxy: util.format('http://%s:%d', sidecar_endpoint, sidecar_port),
    ca: certs
});

(async () => {
    const client = new S3Client({
        region: region,
        requestHandler: new NodeHttpHandler({
            httpAgent: agent,
            httpsAgent: agent
        }),
    });
    
    const command = new ListTablesCommand({});
    try {
        console.log("Running List Tables command through the sidecar")
        const results = await client.send(command);
        console.log(results.TableNames.join("\n"));
    } catch (err) {
        console.error(err);
    }
})();

A proxy agent is required for a proper configuration and can be installed with the command: npm install hpagent --save. Different SDK versions might require different proxy modules. Please refer to the official AWS documentation at: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-proxies.html

CLI + Plugins

Connect using the AWS SDK for Java:

In ClientConfiguration, specify the hostname of your sidecar, the port where the S3 repository is bound, and the proxy protocol, HTTP. This example assumes a sidecar is reachable at example-sidecar-endpoint.com, and a S3 repository bound to it at port 453.

In the AWS SDK for JAVA, the certificate bundle needs to be directly included in the JVM certificate trust store.

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.services.s3v2.AmazonS3;
import com.amazonaws.services.s3v2.AmazonS3ClientBuilder;
import com.amazonaws.services.s3v2.document.S3;
import com.amazonaws.services.s3v2.document.Table;
import com.amazonaws.services.s3v2.document.TableCollection;
import com.amazonaws.services.s3v2.model.ListTablesResult;


public class ExampleSidecarConn {

    static String  sidecar_certificate_bundle = "/path/to/cyral_ca_bundle.pem";
    static String  sidecar_endpoint = "example-sidecar-endpoint.com";
    static Integer sidecar_port = 443; // port selected when binding the s3 repo to the sidecar
    static String  region = "us-east-2";

    public static void main(String[] args) throws Exception {
        BasicAWSCredentials basic = new BasicAWSCredentials("YOUR_AWS_ACCESS_KEY_ID", "YOUR_AWS_SECRET_ACCESS_KEY");
        AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(basic);

        // sidecar proxy configuration
        ClientConfiguration config = new ClientConfiguration();
        config.setProxyHost(sidecar_endpoint);
        config.setProxyPort(sidecar_port);
        config.setProxyProtocol(Protocol.HTTP);

        AmazonS3 client = AmazonS3ClientBuilder.
            standard().
            withCredentials(credentials).
            withRegion(region).
            withClientConfiguration(config).build();
        S3 dynamoDB = new S3(client);


        System.out.println("Running List Tables command through the sidecar");

        TableCollection<ListTablesResult> result = dynamoDB.listTables();
        for (Table a: result) {
            System.out.println(a.getTableName());
        }
    }
}

CLI

Connect using the AWS SDK for Golang:

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "net/url"
    "os"

    awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

// For more details, please refer to the official AWS documentation:
// https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/custom-http/#configuring-a-proxy

const (
    sidecar_certificate_bundle = "/path/to/cyral_ca_bundle.pem"
    sidecar_endpoint           = "example-sidecar-endpoint.com"
    sidecar_port               = 453 // port selected when binding the s3 repo to the sidecar
    region                     = "us-east-2"
)

func assertNil(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

func main() {
    httpClient := awshttp.NewBuildableClient().WithTransportOptions(func(tr *http.Transport) {
        proxyURL, err := url.Parse(fmt.Sprintf("http://%s:%d", sidecar_endpoint, sidecar_port))
        assertNil(err)
        tr.Proxy = http.ProxyURL(proxyURL)
    })

    ctx := context.TODO()
    certReader, err := os.Open(sidecar_certificate_bundle)
    assertNil(err)

    cfg, err := config.LoadDefaultConfig(
        ctx,
        config.WithRegion(region),
        config.WithHTTPClient(httpClient),
        config.WithCustomCABundle(certReader),
    )
    assertNil(err)

    client := s3.NewFromConfig(cfg)
    log.Println("Running List Tables command through the sidecar")
    result, err := client.ListTables(ctx, &s3.ListTablesInput{})
    assertNil(err)

    log.Println(result)
}

Alternatively, the following environment variables are applicable to the Golang AWS SDK HTTP_PROXY, HTTPS_PROXY, and AWS_CA_BUNDLE. If a shell session is configured with these variables, all data traffic will go through the sidecar, potentially including traffic not related to S3 itself, so the code snipped presented above is preferable in favor of environment variables.

Note on other AWS SDKs

AWS offers SDKs for a wide range of programming languages, while here we only show configuration examples for a subset of them. If examples for your programming language are missing from this page, we encourage you to either look for examples in AWS official documentation at https://aws.amazon.com/developer/tools/, or contact Cyral support for help.