Skip to main content
Version: v4.14

Customize how Cyral logs are sent to a logging destination of your choice

Cyral sidecars have an embedded Fluent Bit engine, which can be utilized to customize how both Cyral data activity logs and sidecar diagnostic logs are shipped to various log management destinations. The Custom logging integration is the way to customize how logs are shipped using the sidecar's embedded Fluent Bit engine.

Like other the supported logging platforms, a Custom integration must be created and mapped to a sidecar (or sidecars) for it to function.

  1. Navigate to the Integrations page in the sidebar.

  2. Click Setup or Configure on the Logging card, and click the New Integration button.

  3. Select Custom from the list of integration platforms.

  4. Give this integration an identifiable Name of your choice.

  5. In the Config text region, provide a valid Fluent Bit configuration (more on this below).

  6. Click Create.

  1. For each sidecar that will send logs to this destination, configure the sidecar's advanced logging settings and select this integration for Data Activity Logs and/or Diagnostic Logs. For more information, see "Manage Sidecars -> Logging".

Creating a custom Fluent Bit configuration

The Config text must be a valid "classic mode" Fluent Bit configuration. However, Cyral only supports FILTER and OUTPUT sections in the config. Any other sections will be considered invalid and lead to an error. All the standard Fluent Bit filter plugins and output plugins are supported. Filters and outputs can be combined in any way supported by Fluent Bit, including combining multiple filters and outputs.

Matching and Routing

Fluent Bit requires that plugins are configured to match log records for them to apply.

Cyral log records are tagged based on their output stream and the sidecar service that generated them, in the format <stdout|stderr>-<service name>. For example stdout-dispatcher and stderr-pg-wire. Cyral data activity logs are sent to the stdout stream, while diagnostic logs are sent to the stderr stream.

Records can be matched and routed by their tag, using wildcards. For example, to match all Cyral data activity logs from all services, one could write a match rule Match stdout*. Alternatively, to match only diagnostic logs, one could write the match rule Match stderr*. To match all sidecar logs, one could use Match * or Match std*, for example.

Examples

Below are a few examples of possible Fluent Bit configurations that can be used to customize how sidecar logs are shipped to various log management platforms. Please note that these are examples only and are not functional by themselves without customization for your environment. Also note that the examples are not exhaustive - there are many more filter and output plugins which can be used. Please see the Fluent Bit documentation for more details.

Amazon S3

The following example config will send only sidecar diagnostic logs (note the stderr* match) to Amazon S3 (to the bucket example-bucket in the us-east-2 region), using the Amazon S3 output plugin:

[OUTPUT]
Name s3
Match stderr*
Region us-east-2
Bucket example-bucket
Total_file_size 1M

Note that for S3, the sidecar will need permissions to put objects to S3 (s3:PutObject), and AWS credentials must be available to the sidecar via environment variables or other standard methods (IAM instance profile, etc.). See the Fluent Bit Amazon S3 output plugin documentation for more details.

ELK

The following example config will send all Cyral data activity logs (note the stdout* match) to an ElasticSearch, using the Elasticsearch output plugin. Data activity logs will be indexed under the Elasticsearch index pattern cyral-data-activity-logs-<YYYY-MM-dd> as defined by the Logstash_Prefix configuration:

[OUTPUT]
Name es
Match stdout*
Host 192.168.2.3
Port 9200
Logstash_Format On
Logstash_Prefix cyral-data-activity-logs
warning

The ELK integration configuration may be different depending on where it is hosted. See your ELK-hosting provider instructions and refer to the Elasticsearch output plugin documentation for additional information.

Kafka

The following example config will send all sidecar logs (both data activity logs and diagnostic logs - note the * match) to Kafka, using the Kafka output plugin:

[OUTPUT]
Name kafka
Match *
Brokers 192.168.1.3:9092
Topics test

Slack

The following example config will send all Cyral data activity logs (note the stdout* match) to a Slack channel, using the Slack output plugin:

[OUTPUT]
Name slack
Match stdout*
Webhook https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
Monitor data activity in Kibana

Once the ELK integration is configured, administrators will be able to view and monitor data repository activity logs in Kibana.

Prerequisites
Generate sample log data

If you haven't already done so, run a few queries to generate query logs.

  • If your repository is set to log all activity, any query will do.
  • If you have a policy that logs only certain types of activity, run a query that falls within the scope of the policy.
Import the performance insights dashboard into Kibana

Cyral provides a dashboard you can import into your Kibana to observe performance insights based on the query logs generated by your sidecars. Importing this also creates an index matching the pattern cyral* to capture logs produced by sidecars.

To import the dashboard, follow the steps on the Kibana Dashboards and Elastic Index Templates quickstart guide.

Grep Filter and Standard Output

The following example uses the grep filter plugin to filter only data activity logs generated by the end user bob_hardy, and send them via HTTP POST to the URL http://example.com/logs, using the HTTP output plugin:

[FILTER]
Name grep
Match stdout*
Regex $identity['endUser'] bob_hardy

[OUTPUT]
Name http
Match stdout*
Host example.com
Port 80
URI /logs

Next steps