How we use OPA at Cyral

Background

The Cyral service evaluates policies to monitor, protect, and govern access to data. A key engineering requirement we’ve had to address while building Cyral is the system’s ability to evaluate these policies in the context of systems with dynamically evolving states. Perhaps one of the simplest examples of such decision-making is the task of authorization, e.g. answering questions like “Does some user have access to some resource?”. This is quite simple when it comes to web APIs for example, as one can use various authorization solutions and protocols (OAuth2, etc.), many of which have support directly included in the web frameworks in use. However at Cyral, we’ve often encountered the need to make similar types of decisions in the context of systems and domains that are not as straightforward as the simple API authorization example. Some examples include:

• Enforcing user and group access to different databases in a consistent manner.
• Classifying the contents of random samples of database data.
• Evaluating network allowlists and denylists based on user IP address.
• Extracting and evaluating user identification information from database connection data.

While it would be possible to encapsulate all decision-making logic in standard application code, we realized quickly that such an approach is inflexible.

When designing the Cyral platform, we also realized that we needed a uniform approach to handling policy decisions across our entire stack. As mentioned above, our policies themselves vary wildly in substance and domains. Additionally, we needed something that could fit well into a mesh of disparate microservices.

We ultimately settled on adopting Open Policy Agent (OPA) into our stack. OPA is an open-source, general-purpose policy engine that lets you specify policy as code in a declarative fashion, and then use those policies to make decisions dynamically. OPA policies are defined using Rego, a declarative and data-driven domain-specific language (DSL).

Limitations and Complexities of Policy Evaluation In Microservice Architectures

Traditional monolithic software systems had a simpler problem to solve when it came to policy evaluation, as it was usually contained within a single part of the system. As the industry embraced microservices as a standard in software architecture, policy enforcement became more complex. 

• Decision-making often required unique behavior across multiple, independent services. 
• Services are often written in different languages and use different technologies, adding further complexity.
• The code to express policies may be complex and difficult to read and understand.
• As services are updated, they may eventually fall out of step with the policies associated with them.

Also, while traditional policy frameworks may handle simpler authorization use-cases well (such as API and user authorization) they are not flexible enough to model more generalized systems and decision making.

Why OPA (and Rego)?

OPA is a general-purpose policy engine, which makes it quite powerful and flexible. Some of its main benefits include:

• OPA is a general-purpose policy engine, and can be used to make decisions at any layer of the stack, so long as the domain and its state can be modeled as JSON (or YAML). This can be leveraged to make decisions about just any domain imaginable.
• Rego is declarative and readable — policies express what the outcome should be, not how to achieve that outcome.
• Rego policies are decoupled and can be managed separately as code, and injected into OPA at any point during their lifecycle.
• OPA is data-driven. External data can be injected into policies and used to make decisions, even as the data changes.
• OPA is lightweight, and can be integrated into the stack as a service or even a simple library.

A Quick Primer on Rego

Rego policies use two pieces of JSON (or YAML) data to make policy decisions, named input and data. Input represents a per-decision query, such as the data associated with a single HTTP request, or a database query, etc. On the other hand, data represents external data which can be used in decision-making, inserted to OPA ahead-of-time via a variety of means (API, etc.).  

OPA itself does not understand what the domain state data (or the schema for that data) means in any context, nor does it need to — it simply provides an interface to interact with the data.

Consider the canonical example policy — a policy to allow/deny requests to a REST API, given some input query containing a user, a path (e.g. /balance/{username} ), and a method (e.g. GET, POST, etc.) (try on the Rego playground) :

# By default, the allow rule is set to false
default allow = false

# allow is true if...
allow = true {
    # The input method is 'GET', and...
    input.method == "GET"
    
    # ...the first element of the input path is 'balance', and...
    input.path[0] == "balance"
    
    # ...the second element of the input path is equal to the user 
    # making the request, and...
    input.path[1] == input.user
    
    # ...the input path is not restricted as defined by the
    # external data.
    not data.paths[input.path[0]].restricted
}

The policy above both uses input to make policy decisions as well as cross-references against existing data. Both input and data are just arbitrary JSON. Input data may look something like below, which represents a single GET request to an API by the user bob and to the path /balance/bob:

{
  "method": "GET",
  "path": ["balance", "bob"],
  "user": "bob"
}

As mentioned above, data is also just arbitrary JSON. For example, imagine we could have a set of API paths we decide should be restricted. These paths could then be defined as external JSON data (in this case, we’ll restrict the balance path):

{
  "paths": {
    "balance": {
      "restricted": false
    }
  }
}

Now regardless of the input query, if the balance path is set to restricted in the external data, the policy will evaluate to false.

Note that the data is just arbitrary JSON. In the example above, we modeled a simple denylist of API endpoints. In reality, your imagination is the limit. If you can model a domain and its state as JSON, you can take advantage of it in a Rego policy. 

Additionally, OPA provides mechanisms for updating a policy’s external data dynamically and in real-time, often as requirements and/or the domain state changes. This flexibility can be taken advantage of to completely alter how policy decisions evaluate, without requiring any underlying changes to the policies themselves. Simply, as the system and its data changes, OPA can adapt its decision-making. This type of data-driven behavior is extremely powerful, yet uncommon in other policy engines (especially those which are custom-built)

How Cyral uses OPA

Once we adopted OPA, we immediately began using it across our entire stack to make both “traditional” policy decisions (such as allow/deny access approval, etc.), and more “non-traditional” policy decisions, leveraging OPA’s flexible and domain-agnostic nature.

Traditional Policy Decisions	Non-Traditional Policy Decisions
Generally involves authorization, e.g. “Can this user do X?”	May involve any sort of general decision, e.g. “Is this data sensitive?”
The result of a decision is usually a single true/false (e.g. allow/disallow).	Decision output is arbitrary — it may be any structured or unstructured data.
Policies are typically evaluated on input data only (e.g. a user token, etc.)	External data may be cross-referenced with input data.

For example, we use OPA and Rego to automatically classify sensitive data within a set of sample data. Given some sample data key/value pairs, we can classify it based on the contents of that data. The example policy below demonstrates a policy which can classify some arbitrary key/value input data as ADDRESS if it meets some defined criteria (try on the Rego playground):

package classifier

# The input data (a key/value pair of strings) is classified as ADDRESS if...
classify(key, val) = "ADDRESS" {
   	 # Any of the following statements are true...
    	any(
   		 [
            	# The lowercase key is equal to "state"...
       		 lower(key) == "state",
            	# ...or, the lowercase key is equal to "zip"...
            	lower(key) == "zip",
            	# ...or, the lowercase key is equal to "zipcode"...
            	lower(key) == "zipcode",
            	# ...or, the lowercase key contains the word "address"...
            	re_match(`\A.*address.*\z`, lower(key)),
            	# ...or, the lowercase key contains the word "street"
         		 re_match(`\Astreet.*\z`, lower(key)),
        	]
    	)
} else = "UNLABELED" {
    	true
}

# Classify each input k/v pair, and put the results in the "output" variable.
# Note we're using a comprehension here: https://www.openpolicyagent.org/docs/latest/policy-language/#object-comprehensions
output := {k: v |
	v := classify(k, input[k])
}

Given some key/value pair input:

{
  "address": "123 N. Example St.",
  "foo": "bar",
  "state": "NY",
  "zip": 12345
}

The policy output may look something like this:

{
  "output": {
    "address": "ADDRESS",
    "foo": "UNLABELED",
    "state": "ADDRESS",
    "zip": "ADDRESS"
  }
}

Additionally, because the output of OPA policies are just JSON documents, we can also use OPA policies to return arbitrary information in response to some input data. For example, we can define an OPA policy which returns something like a rule-set for how to act on some piece of data. The rule-set is just a piece of data which may contain instructions for how the application interacts with some other data (e.g. deserialization logic, etc.). An application can query the policy, passing the data and any other metadata as input, and use the response to understand how to process the data further. This rule-set is defined directly into the policy, and the application making the query to the policy only knows the source of the data and the actual data itself. The policy can use this input data, as well as external data, to return a set of rules to the application which tells it how to process its data. 

Here is a simple example policy which returns a “rule-set” on how to process database connection data, for a specific set of database types (try on the Rego playground):

package dbruleset
import data.params

# Given some connection data, infer a ruleset about how to process said data.
inferRuleset(conn) = {
  "appName": params.appName,
  "encoding": "base64",
  "charset": "utf-8",
  "serde": {
"type": "json",
"usernameField": "un",
"passwordField": "pw",
"hostField": "host",
"portField": "port"
  }
} {
  # Only return the above details if the db type is one of the following...
  applicableDbTypes := ["postgresql", "redshift", "snowflake"]
  conn.db.type == applicableDbTypes[_]
} else = {
  "status": "unknown",
}

# Return the ruleset inferred by the input
ruleset := inferRuleset(input)

These policies can then be checked into source control and managed externally from application code. Additionally, the external data (the params in the example above) can change as necessary and be synced into OPA dynamically.

While the above example may seem a bit contrived, it should still serve as an example of the flexible and powerful nature of OPA. In fact, we use an even more advanced form of this technique at Cyral to instruct applications how to extract various connection details for database tools and platforms at runtime. We can evolve these policies dynamically by adjusting them or the external data they cross reference to change runtime behavior without any recompilation or updates to our application.

Summary

OPA and Rego provides a readable, data-driven, declarative, and lightweight framework for dynamic policy evaluation and decision-making across the entire technology stack. When compared to other policy engines, OPA offers clear advantages and benefits. In addition to traditional policy use-cases such as API authorization, OPA’s flexible architecture facilitates decision-making for a wide-range of simple to complex systems. Its data-driven approach to policy evaluation allows the state of systems to evolve dynamically to impact the outcome of decisions, while policies themselves can remain unchanged. At Cyral, we found that our unique requirements and use-cases for policy evaluation, spread across a mesh of microservices, required something different from what legacy policy engines could provide. We have found that OPA fits nicely into our stack and solves these problems uniformly and reliably.