Skip to main content
Version: v4.11

Sidecar Logging

Cyral sidecars output two types of logs: data activity logs, which can be used to monitor database activity, and diagnostic logs, which can be used to monitor the health and activity of the sidecar itself.

The sidecar emits both data activity logs and diagnostic logs to its stdout and stderr streams respectively. For multi-container sidecars, logs are aggregated by the cyral_log_shipper container and output to that container's standard streams.

Data Activity Logs​

Data activity logs describe queries and other operations performed on your data, and detail who is accessing which data and how they are doing so. Data activity logs can be used to monitor and visualize activity on your data repositories.

Data activity logs are structured JSON documents, which include information about database activity such as executed SQL statements, user identity information, and whether any policies were violated, etc. Please see specification below for a complete breakdown on the structure and content of the data activity logs.

Cyral admins can customize which types of database events generate data activity logs on a per-repository basis. Additionally, Cyral can be customized to redact sensitive values from data activity logs.

Data Activity Logs Specification​

Cyral data activity logs follow the structure shown below. The particular set of logged fields you'll see in your logs depends on what you have specified in your repository's log settings.

For help setting up logging, see log integrations.

The example below shows an SSO user, bob@hhiu.us, who logged in using the repo account dbuser001 and ran a SELECT query on the invoices-staging PostgreSQL repository. The query violated a policy that limits the Finances group to read only one row of the table containing the field associated with the data label CCN and data tag PCI:

{
    "time": "2023-07-06T03:52:42.000000Z",
    "activityId": "127.0.0.1:42118:1688615546256057574:7:1",
    "activityTime": "2023-07-06 03:52:42.858413348 +0000 UTC",
    "activityTimeNanos": 1688615562858413348,
    "activityTypes": [
        "query",
        "fullTableScan"
    ],
    "identity": {
        "endUser": "bob_hardy",
        "endUserEmail": "bob@hhiu.us",
        "group": "Finances",
        "userGroups": [
            "Contractors", 
            "Finances"
        ],
        "repoUser": "dbuser001",
        "dbRole": "readOnly"
    },
    "repo": {
        "id": "2SBMUz9xU0SIwXh3iXekMa6UecG",
        "name": "invoices-staging",
        "type": "postgresql",
        "host": "172.18.0.4",
        "port": 5432
    },
    "client": {
        "connectionId": "127.0.0.1:42118:1688615546256057574",
        "connectionTime": "2023-07-06 03:52:26.256057574 +0000 UTC",
        "connectionTimeNanos": 1688615546256057574,
        "host": "127.0.0.1",
        "port": 42118,
        "applicationName": "psql",
    },
    "connectionAuthorization": {
        "authorized": true,
        "reason": "Authorized by access rule for group Finances"
    },
    "sidecar": {
        "id": "2SBEUstTIgl2d1LDAVR3rVDE3Hq",
        "name": "sidecar-east-2",
        "autoScalingGroupInstance": "SINGLETON"
    },
    "request": {
        "statement": "SELECT * FROM playground.playground.transactions",
        "statementType": "SELECT",
        "isSensitive": true,
        "datasetsAccessed": [
            {
                "database": "playground",
                "dataset": "playground.transactions",
                "accessType": "read"
            }
        ],
        "fieldsAccessed": [
            {
                "database": "playground",
                "field": "playground.transactions.card_number",
                "label": "CCN",
                "tags": [
                  "PCI"
                ],
                "accessType": "read"
            },
            {
                "field": "playground.transactions.email",
                "label": "EMAIL",
                "accessType": "read"
            }
        ],
        "searchPath": [
            "public",
            "playground"
        ],
        "sessionParameters": {
            "database_name": "playground"
        }
    },
    "response": {
        "message": "Ok",
        "isError": false,
        "records": 54,
        "bytes": 11214,
        "executionTime": "1.340189ms",
        "executionTimeNanos": 1340189
    },
    "policyViolated": true,
    "policyViolations": [
      {
          "label": "CCN",
          "tag": "PCI",
          "policyName": "row_limit_policy",
          "policyId": "2SE8mSmzCCE2zEYHO26ItCqe4QC",
          "accessType": "read",
          "selectedIdentity": "group:Finances",
          "reasons": [
              "Policy row_limit_policy violated: 54 records accessed exceeding limit of 1"
          ],
          "severity": "high"
      }
    ],
    "stream": "stdout",
    "svc": "pg-wire"
}

  • Each log entry begins with a header section that includes:

    • activityId: Unique identifier for this log entry.
    • activityTime: When this event began.
    • activityTimeNanos: Duration of the event.
    • activityTypes: What sort of event this entry describes. This may be query for a database query, authenticationFailure when a user fails to connect to a repository, authorizationFailure when a user is not authorized by an access rule, newConnection when a user connects to a repository, and closedConnection when they disconnect. Other possible events are: dml, privileged, fullTableScan, authenticationSuccess, networkAccessViolation, portScan, connectionError, multiplexedConnectionFailure.
    • svc: The sidecar service which generated this log message.

  • The identity block shows who performed the action:

    • endUser: The user who authenticated; usually an SSO user.
    • endUserEmail: Email address of the user.
    • repoUser: The native repo account used for the database session.
    • group: This field will be filled in the following cases:
    • userGroups: End user's SSO groups. For users' group affiliations to be tracked using Looker and Tableau, please refer to the following pages: Looker and Tableau.
    • dbRole: (PostgreSQL, SQLServer, Oracle, Redshift, Denodo, Snowflake and DynamoDB only) The user database role. In some databases, the same user (repoUser) may assume different database roles, e.g. admin or readOnly, with different database-level permissions associated with each one of them.
    • authDB: (MongoDB only) The authentication database for the user, i.e., the database that the user exists in. For MongoDB, the user's name and the authentication database serve as a unique identifier for that user.

  • The cyralContext block has the fields user, userGroup, serviceName and attributes. It will be populated in case of Cyral Context-based service account resolution and it will contain the Cyral Context data passed in by the client along with the command or query executed.

  • The repo block shows:

    • id: Data repository ID in Cyral.
    • name: Data repository name in Cyral.
    • type: Type of repository, such as PostgreSQL or MongoDB.
    • host: Network host of the repository.
    • port: Network port of the repository.

  • The client block shows:

    • connectionId: A unique ID for the client connection.
    • connectionTime: Timestamp when client connection was initiated.
    • connectionTimeNanos: Duration of connection.
    • host: Client's network host.
    • port: Client's network port.
    • applicationName: Client application that sent the command.

  • The connectionAuthorization block shows whether the connection was authorized, why it was authorized or not (reason) and the accessConditionEvalResults, a summary of the evaluation of access conditions. Note that the reason may give some positive reason for authorization while overall authorization still fails because some access condition failed.

  • The connectionError field will contain the error message if a connection error occurred.

  • The networkAccessControl will be populated with the reason that access was granted or denied, unless the Network Shield feature is disabled.

  • The sidecar block shows which Cyral sidecar brokered the request:

    • id: Sidecar ID.
    • name: Sidecar name.
    • autoScalingGroupInstance: ID of the sidecar cluster node that brokered the request.

  • The request block shows details about the command or query that was run/attempted:

    • statement: The command or query itself. If you have turned on log redaction, then sensitive data will be omitted from the logged statement and replaced with a value of ${cyral-redact}.
    • preparedStatement: If the command was a PREPARE statement, statement will contain a PREPARE prefix and preparedStatement will have the prepared command.
    • cursorStatement: The actual statement associated with the cursor that will be executed through fetch statements.
    • statementType: Type of command.
    • preparedStatementType: If the command was a PREPARE statement, statementType will be PREPARE and preparedStatementType will contain the type of the statement being prepared (e.g. SELECT, INSERT, etc.).
    • cursorStatementType: The type of statement associated with the cursor that will be executed.
    • storedProcedureName: The name of the stored procedure that is being invoked by the request.
    • The statementData block contains extra information specific to the executed/attempted action.
    • isSensitive: Did this action affect a table or field considered to be sensitive in your Cyral configuration?
    • The datasetsAccessed block lists the tables and collections affected by this action. Inside this block, each accessed data location is listed as a database and a dataset with an accessType showing whether the action was a read, update, delete, or other action. The database field is populated only when the query explicitly has this information, otherwise is empty.
    • The fieldsAccessed block lists the labeled columns and fields, i.e., the fields present in your data map, within each dataset affected by this action. Each is a database and a field with an accessType and a label or tags in your data map. The database field is populated only when the query explicitly has this information, otherwise is empty. If masking has been applied to the field, the mask function used is also shown here.
    • rewrittenStatement: The rewritten command or query when dataset rewrites and/or masking have been applied. For Oracle, PostgreSQL, Redshift, Denodo, Snowflake and SQLServer this field is also populated in cases where row limiting and rate limiting are enforced, which are achieved by adding a suitable LIMIT clause for each dialect to the query.
    • The filters block lists the filters included in the statement (e.g. in the WHERE clause).
    • limit (MongoDB, MySQL, PostgreSQL, SQLServer, Oracle, Redshift and Denodo only) contains the values for the limit and offset elements of the command.
    • error: If an error occurred when processing this action, this field will describe the error. The type of the error can be Parse Error, Analysis Error, Empty Request Error or Request Too Big.
    • dremioQueryId: (Dremio only) The ID generated by the Dremio server for the query being executed.
    • defaultSchema: (Dremio only) The currently selected schema in your session.
    • database: (MongoDB only) The MongoDB database on which the current query is being executed.
    • baseStatement: (MongoDB only) The actual command associated with the cursor that will be executed through GET MORE requests.
    • baseStatementType: (MongoDB only) The type of command associated with the cursor that will be executed (e.g. FIND, AGGREGATE).
    • schema: (MySQL only) The currently selected database in the MySQL session.
    • informationSchemaAccessed (MySQL only) indicates whether the INFORMATION_SCHEMA database was accessed by the command.
    • The userConfigParameters block (PostgreSQL, SQLServer, Oracle, Redshift and Denodo only) captures custom configuration parameters that the database user has provided using SET or SELECT set_config statements in their queries. This facility captures parameters that are written as dot-separated strings. For example, userConfigParameters will show the contents of the JWT token if the API client set it before running the query.
    • searchPath: (PostgreSQL, SQLServer, Oracle, Redshift and Denodo only) List of schemas the database system looks in when unqualified tables are queried.
    • sessionParameters (PostgreSQL, SQLServer, Oracle, Redshift and Denodo only) stores session-related data. The key is the session parameter ID, for example, client_identifier or database_name, while the value shows the actual data associated with that session variable in the database.

  • The response block describes the database response to the command, including any message returned and whether this command resulted in an error (isError). This block indicates how many records and bytes were returned and how long the command took to run (executionTime and executionTimeNanos).

  • The policyViolated field shows whether this command violated any of your Cyral policy rules.

  • If this command violated any of your Cyral policy rules, the policyViolations block will describe the reasons.

  • blockedQuery will indicate whether the query had its execution blocked.

Diagnostic Logs​

Diagnostic logs are sidecar application-level logs, also sometimes referred to as "debug logs". These logs may be useful for diagnosing sidecar issues, analyzing sidecar performance, and general troubleshooting.

While diagnostic logs are still structured JSON documents, their structure is simpler than data activity logs. In general, a sidecar diagnostic log message may look like:

{
  "time": "2006-01-02T15:04:05.000000Z",
  "instanceId": "some-sidecar-instance-id",
  "level": "info",
  "msg": "Some log message",
  "stream": "stderr",
  "svc": "pg-wire"
}
  • time is the timestamp the log message was generated.
  • instanceId is the sidecar instance identifier which generated the log (useful in multi-instance sidecar clusters)
  • level is the log level. Valid log levels are trace, debug, info, warning, error, fatal, and panic.
  • msg is the log message.
  • stream is the standard stream on which the log was generated. This should always be stderr.
  • svc is the name of the sidecar microservice which generated the log message.

Log Management​

Cyral gives you the ability to send sidecar logs to various destinations such as Splunk, AWS CloudWatch, and many others. Please refer to the "Integrations -> Logs" section for complete details.

When using Cyral's logging integrations features, the log management destination for both the sidecar's data activity logs and diagnostic logs can be configured independently in the sidecar's Logging settings: on the Control Plane, navigate to Sidecars -> {sidecar name} -> Settings (tab) -> Logging.

Additionally, sidecar logs can be aggregated and managed using any log management tooling such as FluentD, Fluent Bit, etc. deployed to your infrastructure by consuming the standard output/error streams of the sidecars.

Learn more​