Skip to main content
Version: v4.8

Send Cyral logs to ELK stack

Follow the steps below to configure Cyral to output repository activity logs to ELK.

Set up an ELK stack​

Deploy AWS Elasticsearch Service​

Contact your Cyral support representative to get the deployment template for your cloud environment, and then follow the steps below.

Use the ELK CloudFormation template provided by the Cyral support team to deploy the AWS Elasticsearch service using AWS CloudFormation, taking note of the following steps:

First, set the RemoteAccessCIDR range to your corporate IP addresses while deploying the stack to restrict access to Kibana.

Next, the IAM role AWSServiceRoleForAmazonElasticsearchService is required to install the ELK template. If it does not yet exist, create it with the following command:

aws iam create-service-linked-role --aws-service-name es.amazonaws.com

Note:

If you’d like to make the ELK deployment more robust and enable HA, we suggest making the following changes to the template:

       ElasticsearchClusterConfig:
           DedicatedMasterEnabled: false
           InstanceCount: 2
           InstanceType: t3.medium.elasticsearch
           WarmEnabled: false
           ZoneAwarenessEnabled: true
       EBSOptions:
           EBSEnabled: true
           VolumeType: gp2
           VolumeSize: 50

Connect Cyral to your ELK stack​

Now that the AWS Elasticsearch service is deployed, let's integrate it with Cyral.

  1. In the Cyral management console, click Integrations, find the ELK card, and click Setup.

  1. In the ELK configuration form, you will be prompted to enter the following:

    • Integration Name: Integration name of your choice.
    • Kibana URL: The Kibana server's hostname. If you're using CloudFormation, this matches the KibanaURL output parameter.
    • Elasticsearch URL: Address where Elasticsearch is available. If you're using CloudFormation, this matches the ElasticsearchURL output parameter.
  1. Click Save to connect the integration.

NOTE: Changes to this configuration will take effect immediately, and the sidecar does not need to be redeployed.

caution

Each sidecar that will send logs to this log destination will need to be deployed with its Log Integration set to the Integration Name you specified above.

When deploying new sidecars, make sure to choose the name of this Log Integration when you generate the template.

NOTE: Sidecars that are already deployed will need to be redeployed.

Monitor data activity in Kibana​

Once the ELK integration is configured, administrators will be able to view and monitor data repository activity logs in Kibana. Data activity logs will be indexed under the Elasticsearch index pattern cyral-data-activity-logs-<YYYY-MM-dd>.

Prerequisites​

Generate sample log data​

If you haven't already done so, run a few queries to generate query logs.

  • If your repository is set to log all activity, any query will do.
  • If you have a policy that logs only certain types of activity, run a query that falls within the scope of the policy.

View logs​

Access the Kibana dashboard as shown here for your cloud platform:

In your browser, visit the Kibana dashboard address provided in the Terraform or Cloudformation output from your deployment. See Set up an ELK stack for deployment details.

To limit the view to show only query logs by, search for the log field endUser in the search bar.

Import the performance insights dashboard into Kibana​

Cyral provides a dashboard you can import into your Kibana to observe performance insights based on the query logs generated by your sidecars. Importing this also creates an index matching the pattern filebeat*,cyral* to capture logs produced by sidecars.

  1. Download the Performance Insights Dashboard file by clicking Integrations: ELK: Configure, clicking the Download Kibana Dashboards button and choosing the appropriate version.

  2. Navigate to the Saved Objects tab in the Management page (in the sidebar), and click Import. You can expand the sidebar using the button in the bottom left of the screen to make navigation easier.

  3. Select the cyral-dashboard-<VERSION>.ndjson file, where <VERSION> corresponds to the Kibana Dashboard version downloaded. Check the toggle to Automatically overwrite all saved objects, and click open.

  4. Download and install custom Cyral index template to Elasticsearch. This index template tells Elasticsearch how to configure the Cyral log indexes when they are created. From Kibana, navigate to Dev Tools located under the Management section (or alternatively, use the Elasticsearch REST API), and add the following request, plus the index template JSON from above:

PUT _index_template/cyral_index_template
<index template JSON>

Execute the request, either by clicking to send request button (it looks like a "play button" in the top-right corner of the page), or by using a REST API client or choice such as cURL.

Note that if you have indexes which have been created by Cyral before the index template was installed, they will need to be "re-indexed" using the Elasticsearch REST API. This can be using the Dev Tools again and a request similar to one the below:

POST _reindex
{
  "source": {
    "index": "cyral-data-activity-logs-2023.05.05"
  },
  "dest": {
    "index": "cyral-data-activity-logs-2023.05.05-new"
  }
}

This example command would tell Elasticsearch to copy the index called cyral-data-activity-logs-2023.05.05 into an index called cyral-data-activity-logs-2023.05.05-new, and re-index the fields in it using the Cyral index template. This will need to be done for each index which should be re-indexed.

Once an index has been re-indexed, you’ll want to delete the old indexes so that you no longer receive any errors. You can issue a delete API request Dev Tools (or REST API) each old index to delete it, e.g.:

DELETE cyral-data-activity-logs-2023.05.05

The Performance Insights dashboard should now be available, and you can find it by navigating to the Dashboards page (in the sidebar).

Next steps​