Database Activity Monitoring: DIY or Buy?

Overview

Database Activity Monitoring (DAM) is a critical component in safeguarding data, providing real-time monitoring, threat detection, and compliance adherence. This guide is for practitioners evaluating whether to purchase a DAM solution or build an in-house system.

Here we systematically examine the considerations, trade-offs, and key elements essential in making an informed decision based on 

• Business needs
• Technology needs
• Existing tooling
• Operational considerations

We explore several intricate details of DAM solutions, including their functionalities, benefits, and potential limitations. We also outline considerations for those contemplating a DIY approach, especially organizations moving to the cloud with the ability to leverage the native logging and monitoring capabilities provided by platforms like AWS, Azure, etc.

What is Database Activity Monitoring?

Database Activity Monitoring (DAM) is a security solution dedicated to the continuous surveillance and analysis of database activities in real-time. It provides the following key features

1. Monitor and log all database activity, at the field level
2. Allow logs to be stored externally for assurance 
3. Provide consistent monitoring and logging
4. Help respond to threats and unauthorized access

Typical approaches to commercial DAM solutions

A typical DAM comprises three core components:

• Log generator and collection: This element operates within the database infrastructure, capturing and collecting detailed records of all activities and transactions occurring within the system. It comprehensively logs user actions, queries, access attempts, modifications, and other database activities, providing a granular overview of the operations performed.
• Visibility and reporting: This front-facing aspect of a DAM solution processes the aggregated data, conducts analyses, and generates reports for administrators or security personnel. 
• Enforcement and alerting: Enables administrators to use policies to prevent unauthorized access and provide alerting on malicious behavior.

This three-tier architecture forms the fundamental framework of a DAM solution, providing a systematic approach to monitoring, analyzing, and responding to database activities, ensuring comprehensive security and threat detection within the organization’s database infrastructure.

Types of commercial DAM solutions

The following table outlines the three popular types of DAM solutions, which differ depending on where the solution is deployed – on the network tap, as an agent on the database, or as a proxy before the database.

Manageability and efficacy in a cloud environment
	Log generation and collection	Visibility and reporting	Enforcement and alerting	Performance and availability
Network-based	Generally only able to monitor unencrypted database connections, which are rare.	Visibility and reporting is very limited because most traffic is encrypted.	No ability to enforce policies in the cloud, alerting is done post-facto.	Degrades performance and availability due to resource contention within the database.
Agent-based	Cannot be deployed for SaaS databases, so logs generated from the database directly and collected using scraping.	Visibility and reporting is inconsistent because of reliance on native database logs.	No ability to enforce policies in the cloud, alerting is done post-facto.	Degrades performance and availability due to resource contention within the database.
Proxy-based	Can be deployed in all environments and across all types of databases.	Consistent visibility and reporting across all databases.	Consistent policy enforcement and real-time alerting.	No impact to availability and performance.

The DIY approach to DAM

A DIY approach is getting especially popular for cloud environments. A DIY DAM is generally built by stitching a range of cloud native components together. Since the comparison for most teams is against network-based and agent-based DAM solutions, they only care about the following underlying functionalities: 

• Log generation: administrators can turn on native database logging to obtain logs. 
• Log collection: logs can be collected by using standard log management tools.
• Reporting and alerting: this can be done via a myriad of standalone services.

A sample architecture for AWS

Logs are typically aggregated together for two purposes. The first is to forward to some form of cheap long term storage for audit & compliance purposes – often S3. The second is for monitoring & reporting. This is typically done by looking for keywords in the logs which indicate an event of interest. This could be done using a variety of tools including Lambda, ELK or Splunk. Alerting to tools like Slack can then provide security notifications to investigate. Equivalent tools exist in GCP & Azure also.

Which approach to use?

While a DIY approach is cheap and can tick some DAM requirements, it is typically limited to alerting or reactive actions. Adding real-time response requires a proxy-based DAM architecture that can provide policy enforcement and consistent visibility, ideally without any impact to performance or scalability. The white paper below explores this topic in more detail and outlines a checklist of scenarios to further evaluate what approach is best suited for you.