# External Data in OPA

## **Introduction to External Data in OPA**

Open Policy Agent (OPA) is powerful in its ability to evaluate policies based on static data inputs. However, many real-world scenarios require policies to consider external data sources, such as APIs, databases, or other services that provide information relevant to the decision-making process. By integrating OPA with external data, you can create dynamic policies that respond to real-time information, enabling more granular and context-aware decisions.

Examples of using external data in OPA include:

* **Checking User Roles**: Querying an external database to determine a user’s role or permissions.
* **Real-Time Threat Intelligence**: Using data from a threat intelligence API to block requests from known malicious IP addresses.
* **Compliance Verification**: Verifying configuration compliance by querying an external system for the latest regulatory standards.

## **Approaches to Integrating External Data with OPA**

There are several ways to integrate external data with OPA:

1. **Bundle API**: Pre-fetching data and bundling it with the policy.
2. **Data Queries in Policies**: Querying external data sources directly from within OPA policies using built-in functions.
3. **External Data Services**: Querying external data services or APIs during policy evaluation.

Each approach has its advantages depending on the use case and the nature of the external data.

### **Using the Bundle API for External Data**

The Bundle API allows you to pre-fetch data from external sources and bundle it with your policies. This data can then be used during policy evaluation. This approach is suitable when the data doesn’t change frequently or when you want to avoid real-time queries during policy evaluation.

**Step 1: Configure OPA to Use a Bundle**

You can configure OPA to load data from a bundle hosted on an external server or object storage like AWS S3.

**Example Configuration:**

```yaml
services:
  my_bundle:
    url: https://my-bundle-server.com

bundles:
  example:
    service: my_bundle
    resource: /bundles/example.tar.gz
```

In this configuration, OPA fetches the bundle containing both policies and external data from the specified URL.

**Step 2: Accessing the Data in Rego**

Once the bundle is loaded, the data can be accessed in Rego just like any other input.

**Example Rego Policy:**

```rego
package example

default allow = false

allow {
    input.user == data.external_data.users[_].name
}
```

In this example, `data.external_data.users` contains user data from the external bundle, which is used in the policy to determine access.

### **Querying External Data in Rego Policies**

OPA supports querying external data directly within Rego policies using built-in functions such as `http.send`. This approach is useful for real-time data queries, where decisions depend on the latest information from external services.

**Step 1: Write a Rego Policy with External Data Query**

You can use the `http.send` function to make HTTP requests to external APIs or services.

**Example: Checking Threat Intelligence**

```rego
package example

default allow = false

allow {
    input.source_ip = ip
    response := http.send({
        "method": "GET",
        "url": sprintf("https://threat-intelligence.com/check?ip=%s", [ip]),
        "headers": {
            "Authorization": "Bearer my-token"
        }
    })
    not response.body.threat_detected
}
```

In this example, the policy sends a request to a threat intelligence API to check if the source IP is associated with any known threats. The policy allows access only if no threats are detected.

**Step 2: Handling Response Data**

The `http.send` function returns the response, which can be used to make decisions based on the external data. You can also handle errors, timeouts, or unexpected responses to ensure robustness.

**Example: Handling Errors**

```rego
package example

default allow = false

allow {
    input.source_ip = ip
    response := http.send({
        "method": "GET",
        "url": sprintf("https://threat-intelligence.com/check?ip=%s", [ip]),
        "headers": {
            "Authorization": "Bearer my-token"
        }
    })
    response.status == 200
    not response.body.threat_detected
}

allow {
    input.source_ip = ip
    response := http.send({
        "method": "GET",
        "url": sprintf("https://backup-intelligence.com/check?ip=%s", [ip]),
        "headers": {
            "Authorization": "Bearer my-backup-token"
        }
    })
    response.status == 200
    not response.body.threat_detected
}
```

This example includes a fallback to a backup threat intelligence service if the primary service is unavailable.

### **Integrating OPA with External Data Services**

OPA can also be integrated with external data services that provide real-time data feeds or updates, such as databases, cloud services, or specialized APIs.

**Step 1: Set Up the External Data Service**

Ensure that your external data service is accessible and configured to provide the necessary data for your policies. This could be a database, an API endpoint, or a cloud service like AWS DynamoDB.

**Step 2: Access the External Data in Rego**

Use the `http.send` function or other mechanisms to query the external service. Alternatively, use custom-built data adapters or services that fetch and transform data before passing it to OPA.

**Example: Accessing Database Records**

```rego
package example

default allow = false

allow {
    input.user_id = user_id
    response := http.send({
        "method": "POST",
        "url": "https://my-database.com/query",
        "body": {
            "query": "SELECT * FROM users WHERE user_id = ?",
            "params": [user_id]
        }
    })
    response.body.role == "admin"
}
```

In this example, the policy queries a database to check if the user has an "admin" role before allowing access.

### **Best Practices for Using External Data in OPA**

When using external data in OPA policies, follow these best practices to ensure efficiency, security, and reliability:

* **Minimize Latency**: External data queries can introduce latency. Use caching, pre-fetching, or asynchronous updates to minimize delays in policy evaluation.
* **Handle Failures Gracefully**: External data sources may be unavailable or slow to respond. Implement error handling and fallback mechanisms in your policies to ensure they remain robust.
* **Secure Data Access**: Ensure that all external data queries are secure, using encryption, authentication, and authorization as needed to protect sensitive information.
* **Use Rate Limiting**: Be mindful of rate limits on external services, especially if you are querying public APIs. Implement rate limiting on your side if necessary to avoid service disruptions.
* **Keep Data Up-to-Date**: If using pre-fetched or bundled data, ensure that it is updated regularly to reflect the latest information and prevent outdated policies from being applied.
* **Test Thoroughly**: Test your policies with real data and in real-world scenarios to ensure they behave as expected under various conditions, including failure scenarios.

### **Summary**

In this lesson, you learned how to extend OPA with external data sources to create dynamic, context-aware policies. You explored various approaches to integrating external data, including using the Bundle API, querying data directly in Rego policies, and integrating with external data services. You also discussed best practices for managing and using external data in OPA policies.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://cthfm-k8s.gitbook.io/kubernetes/security-tools/open-policy-agent-opa/external-data-in-opa.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.