Database Activity Monitoring: DIY or Buy?  

Overview

Database Activity Monitoring (DAM) is a critical component in safeguarding data, providing real-time monitoring, threat detection, and compliance adherence. This white paper serves as a guide for practitioners evaluating whether to purchase a DAM solution or build an in-house system using available tools.

This document systematically examines the considerations, trade-offs, and key elements essential in making an informed decision based on 

• Business needs
• Technology needs
• Existing tooling
• Operational considerations

It explores several intricate details of DAM solutions, exploring their functionalities, benefits, and potential limitations. Simultaneously, it outlines considerations for those contemplating a DIY approach, especially customers moving to the cloud with the ability to leverage the native logging and monitoring capabilities provided by platforms like AWS, Azure, etc.

Introduction to Database Activity Monitoring

Solution

Database Activity Monitoring (DAM) is a security solution dedicated to the continuous surveillance and analysis of database activities in real-time. It monitors all user actions, queries, and transactions within the database infrastructure, flagging potential threats, unauthorized access, and anomalous behaviors. DAM solutions play a critical role in maintaining data integrity, ensuring compliance with industry regulations, and fortifying defenses against evolving cyber threats.

At its core, a DAM provides the following key features

1. Monitor and log all database activity, at the field level
2. Allow logs to be stored externally for assurance 
3. Provide consistent monitoring and logging
4. Help respond to threats and unauthorized access

Drivers for adoption

• Compliance Adherence: Various regulatory standards, such as Payment Card Industry Data Security Standard (PCI DSS), Health Insurance Portability and Accountability Act (HIPPA), and Sarbanes-Oxley Act (SOX), mandate stringent data security and privacy measures, necessitating continuous monitoring that DAM can provide.
• Customer Assurance: Protecting the integrity of sensitive data is critical. DAM enables businesses to provide assurance to their customers about their internal processes and practices by providing an auditable activity trail.
• Audit and Forensics: DAM solutions provide robust audit trails and forensic analysis, crucial for investigations, compliance audits, and understanding the sequence of events in case of security incidents or breaches.

Use cases

• Threat Detection: With the rising sophistication of cyber threats, DAM offers real-time threat detection and swift response mechanisms, averting potential breaches.
• Insider Threat Mitigation: Monitoring user behavior helps in identifying and preventing potentially harmful actions from within the organization, reducing the risk posed by insider threats.
• Security Incident Response: Visibility into database activity logs enables quick response and remediation, minimizing potential damage and mitigating risks to the organization.
• Protection of Sensitive Data: Organizations handling proprietary or confidential information can use DAM to safeguard their valuable assets and prevent unauthorized access or misuse.

Typical approaches to commercial DAM solutions

Baseline architecture

A typical DAM comprises three core components:

• Log generator and collection: At the heart of DAM lies the log generation and collection component. This element operates within the database infrastructure, capturing and collecting detailed records of all activities and transactions occurring within the system. It comprehensively logs user actions, queries, access attempts, modifications, and other database activities, providing a granular overview of the operations performed.
• Visibility and reporting: The visibility and reporting component is the front-facing aspect of a DAM solution. It processes the aggregated data, conducts analyses, and generates reports for administrators or security personnel. 
• Enforcement and alerting: DAM solutions should enable administrators to use policies to prevent unauthorized access and provide alerting on malicious behavior.

This three-tier architecture forms the fundamental framework of a DAM solution, providing a systematic approach to monitoring, analyzing, and responding to database activities, ensuring comprehensive security and threat detection within the organization’s database infrastructure.

Types of commercial DAM solutions

DAM solutions differ from each other based on how the log generator and collectors are deployed, which impacts their efficacy and user experience. The three deployment models are as follows:

While each approach has its merit, in an ecosystem that is increasingly becoming cloud-centric, a proxy-based approach generally provides several advantages when it comes to uniformity of log generation and collection. This is explained in the following table below:

Manageability and efficacy in a cloud environment
	Log generation and collection	Visibility and reporting	Enforcement and alerting	Performance and availability
Network-based	Generally only able to monitor unencrypted database connections, which are rare.	Visibility and reporting is very limited because most traffic is encrypted.	No ability to enforce policies in the cloud, alerting is done post-facto.	Degrades performance and availability due to resource contention within the database.
Agent-based	Cannot be deployed for SaaS databases, so logs generated from the database directly and collected using scraping.	Visibility and reporting is inconsistent because of reliance on native database logs.	No ability to enforce policies in the cloud, alerting is done post-facto.	Degrades performance and availability due to resource contention within the database.
Proxy-based	Can be deployed in all environments and across all types of databases.	Consistent visibility and reporting across all databases.	Consistent policy enforcement and real-time alerting.	No impact to availability and performance.

The DIY way for cloud

As organizations are moving critical datasets to the cloud, they are in need of a DAM solution that works for the cloud. There are two key considerations that practitioners face when it comes to cloud, which were irrelevant for on-prem workloads:

1. The inefficacy and challenges for network-based and agent-based DAM solutions when it comes to the cloud. 
2. The availability of tools and services available in a typical cloud environment that were difficult to gain in an on-prem setting.

Services available to practitioners in the cloud

A DIY DAM is generally built by stitching a range of cloud native components together. Since the comparison for most teams is against network-based and agent-based DAM solutions, they only care about the following underlying functionalities: 

• Log generation: administrators can turn on native database logging to obtain logs. 
• Log collection: logs can be collected by using standard log management tools.
• Reporting and alerting: this can be done via a myriad of standalone services.

A sample architecture for AWS

Logs are typically aggregated together for two purposes. The first is to forward to some form of cheap long term storage for audit & compliance purposes – often S3. The second is for monitoring & reporting. This is typically done by looking for keywords in the logs which indicate an event of interest. This could be done using a variety of tools including Lambda, ELK or Splunk. Alerting to tools like Slack can then provide security notifications to investigate. Equivalent tools exist in GCP & Azure also.

DIY vs proxy-based DAM

While a DIY solution is cheap and can tick some DAM requirements, it is typically limited to alerting or reactive actions. Adding real-time response requires a proxy-based DAM architecture that can provide policy enforcement and consistent visibility, ideally without any impact to performance or scalability. A detailed checklist of considerations where a commercial DAM solution is valuable are the following:

• Do the logs need to be used for forensics?
  • A proxy-based DAM is useful because it can generate enriched logs with additional context on where the request came from, who made it, and other attributes (which are often vendor dependent).
• Do the logs need to cover application users?
  • A proxy-based DAM is required because activity of all application users shows up as requests from a single service account, making the logs noisy and ineffective (only few vendors have the ability to disambiguate service account users)
• Is policy enforcement a use case?
  • A proxy-based DAM is recommended because policy enforcement is very difficult to accomplish using database-level policies. 
• Does the DAM need to provide data protection?
  • A proxy-based DAM is required because the policies to prevent data insertion, deletion and/or update needs to be externally managed. 
• Does the DAM need to support privacy initiatives?
  • A proxy-based DAM is highly recommended because privacy initiatives mandate the ability to dynamically mask data which is extremely complex to accomplish using database policies.