DataDog · dd-mergequeue · May 19, 2025 · Apr 23, 2025 · Apr 23, 2025 · Apr 23, 2025
@@ -6067,6 +6067,12 @@ menu:
       parent: log_troubleshooting
       identifier: log_troubleshooting_live_tail
       weight: 1001
+    # - name: CloudPrem
+    #   pre: cloud
+    #   url: cloudprem/
+    #   parent: log_management_heading 
+    #   identifier: cloud_prem
+    #   weight: 30000
     - name: Security Overview
       url: security/
       pre: security-platform

@@ -0,0 +1,32 @@
+---
+title: CloudPrem
+description: Learn how to deploy and manage Datadog CloudPrem, a self-hosted log management solution for cost-effective log ingestion, processing, indexing, and search capabilities
+private: true 
+---
+
+<div class="alert alert-warning">CloudPrem is in Preview.</div>
+
+## Overview
+
+Datadog CloudPrem is a self-hosted log management solution that enables cost-effective log ingestion, processing, indexing, and search capabilities within your own infrastructure. Built to meet data residency, stringent security, and high-volume requirements, CloudPrem integrates with the Datadog platform to provide log analysis, visualization, and alerting - all while keeping your log data at rest within your infrastructure boundaries.
+
+## Architecture
+
+CloudPrem uses a decoupled architecture that separates compute from storage, enabling independent scaling of indexing, search, and storage. Key components include indexers, searchers, a PostgreSQL-based metastore, and a control plane for task orchestration. For more information, see the [Architecture][4] documentation.
+
+## Get started
+
+{{< whatsnext desc="What's next:" >}}
+   {{< nextlink href="/cloudprem/installation/" >}}Install CloudPrem and Send Logs with the Agent {{< /nextlink >}}
+   {{< nextlink href="/cloudprem/ingress/" >}}CloudPrem Ingress Configuration{{< /nextlink >}}
+   {{< nextlink href="/cloudprem/cluster/" >}}Learn more about Cluster Sizing and Operations{{< /nextlink >}}
+   {{< nextlink href="/cloudprem/aws_config" >}}AWS Configuration{{< /nextlink >}}
+   {{< nextlink href="/cloudprem/processing/" >}}Configure CloudPrem Log Processing{{< /nextlink >}}
+   {{< nextlink href="/cloudprem/troubleshooting/" >}}Troubleshooting{{< /nextlink >}}
+{{< /whatsnext >}}
+
+
+[1]: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/ 
+[2]: /cloudprem/installation/
+[3]: /cloudprem/processing/
+[4]: /cloudprem/architecture/
@@ -0,0 +1,53 @@
+---
+title: Architecture
+further_reading:
+- link: "/cloudprem/"
+  tag: "Documentation"
+  text: "Learn more about CloudPrem"
+- link: "/cloudprem/installation/"
+  tag: "Documentation"
+  text: "CloudPrem Installation"
+- link: "/cloudprem/ingress/"
+  tag: "Documentation"
+  text: "Ingress Configuration"
+- link: "/cloudprem/cluster/"
+  tag: "Documentation"
+  text: "Cluster Sizing and Operations"
+- link: "/cloudprem/aws_config"
+  tag: "Documentation"
+  text: "AWS Configuration"
+- link: "/cloudprem/processing/"
+  tag: "Documentation"
+  text: "CloudPrem Log Processing"
+- link: "/cloudprem/troubleshooting/"
+  tag: "Documentation"
+  text: "Troubleshooting CloudPrem"
+---
+
+<div class="alert alert-warning">CloudPrem is in Preview.</div>
+
+## Overview
+
+{{< img src="/cloudprem/overview_architecture.png" alt="CloudPrem architecture showing Indexers, Searchers, Metastore, and Control Plane components interacting with object storage" style="width:100%;" >}}
+
+CloudPrem uses a decoupled architecture which separates the compute (indexing and searching) and data on an object storage. This allows for independent scaling and optimization of different cluster components based on workload demands.
+
+## Components
+
+The CloudPrem cluster, typically deployed on Kubernetes (EKS), consists of several components:
+
+**Indexers**
+: Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called _splits_ to the object storage (for example, Amazon S3).
+
+**Searchers**
+: Handle search queries from the Datadog UI, reading metadata from Metastore and index data from the object storage.
+
+**Metastore**
+: Stores metadata about the indexes, including splits locations on the object storage. CloudPrem uses PostgreSQL for this purpose.
+
+**Janitor/Control Plane**
+: Responsible for tasks like indexing tasks scheduling and delete tasks.
+
+## Further reading
+
+{{< partial name="whats-next/whats-next.html" >}}
@@ -0,0 +1,103 @@
+---
+title: AWS Configuration
+description: Learn how to configure AWS for CloudPrem
+further_reading:
+- link: "/cloudprem/"
+  tag: "Documentation"
+  text: "Learn more about CloudPrem"
+- link: "/cloudprem/installation/"
+  tag: "Documentation"
+  text: "CloudPrem Installation"
+- link: "/cloudprem/ingress/"
+  tag: "Documentation"
+  text: "Ingress Configuration"
+- link: "/cloudprem/cluster/"
+  tag: "Documentation"
+  text: "Cluster Sizing and Operations"
+- link: "/cloudprem/processing/"
+  tag: "Documentation"
+  text: "CloudPrem Log Processing"
+- link: "/cloudprem/troubleshooting/"
+  tag: "Documentation"
+  text: "Troubleshooting CloudPrem"
+---
+
+## Overview
+
+This guide covers how to configure your AWS account for CloudPrem. For ingress configuration, see [CloudPrem Ingress Configuration][1].
+
+To deploy CloudPrem on AWS, you need to configure:
+- AWS credentials
+- AWS region
+- IAM permissions for S3
+
+## AWS credentials
+
+When starting a node, CloudPrem attempts to find AWS credentials using the credential provider chain implemented by [rusoto\_core::ChainProvider][2] and looks for credentials in this order:
+
+1. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN` (optional).  
+2. Credential profiles file, typically located at `~/.aws/credentials` or otherwise specified by the `AWS_SHARED_CREDENTIALS_FILE` and `AWS_PROFILE` environment variables if set and not empty.  
+3. Amazon ECS container credentials, loaded from the Amazon ECS container if the environment variable `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` is set.  
+4. Instance profile credentials, used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service.
+
+An error is returned if no credentials are found in the chain.
+
+## AWS Region
+
+CloudPrem attempts to find the AWS region from multiple sources, using the following order of precedence:
+
+1. **Environment variables**: Checks `AWS_REGION`, then `AWS_DEFAULT_REGION`.  
+2. **AWS config file**: Typically located at `~/.aws/config`, or at the path specified by the `AWS_CONFIG_FILE` environment variable (if set and not empty).  
+3. **EC2 instance metadata**: Uses the region of the currently running Amazon EC2 instance.  
+4. **Default**: Falls back to `us-east-1` if no other source provides a region.
+
+## IAM permissions for S3
+
+Required authorized actions:
+
+* `ListBucket` (on the bucket directly)  
+* `GetObject`  
+* `PutObject`  
+* `DeleteObject`  
+* `ListMultipartUploadParts`  
+* `AbortMultipartUpload`
+
+Here is an example of a bucket policy:
+
+```
+
+{
+ "Version": "2012-10-17",
+ "Statement": [
+   {
+     "Effect": "Allow",
+     "Action": [
+       "s3:ListBucket"
+     ],
+     "Resource": [
+       "arn:aws:s3:::my-bucket"
+     ]
+   },
+   {
+     "Effect": "Allow",
+     "Action": [
+       "s3:GetObject",
+       "s3:PutObject",
+       "s3:DeleteObject",
+       "s3:ListMultipartUploadParts",
+       "s3:AbortMultipartUpload"
+     ],
+     "Resource": [
+       "arn:aws:s3:::my-bucket/*"
+     ]
+   }
+ ]
+}
+```
+
+## Further reading
+
+{{< partial name="whats-next/whats-next.html" >}} 
+
+[1]: /cloudprem/ingress/
+[2]: https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html
@@ -0,0 +1,74 @@
+---
+title: Cluster Sizing and Operations
+description: Learn about cluster sizing, scaling, monitoring, and maintenance for CloudPrem
+further_reading:
+- link: "/cloudprem/"
+  tag: "Documentation"
+  text: "Learn more about CloudPrem"
+- link: "/cloudprem/architecture/"
+  tag: "Documentation"
+  text: "CloudPrem Architecture"
+- link: "/cloudprem/installation/"
+  tag: "Documentation"
+  text: "Installation"
+- link: "/cloudprem/ingress/"
+  tag: "Documentation"
+  text: "Ingress Configuration"
+- link: "/cloudprem/aws_config"
+  tag: "Documentation"
+  text: "AWS Configuration"
+- link: "/cloudprem/processing/"
+  tag: "Documentation"
+  text: "CloudPrem Log Processing"
+- link: "/cloudprem/troubleshooting/"
+  tag: "Documentation"
+  text: "Troubleshooting CloudPrem"
+---
+
+<div class="alert alert-warning">CloudPrem is in Preview.</div>
+
+## Overview
+
+This document gives recommendations on dimensioning your CloudPrem cluster components, particularly indexers and searchers.
+
+<div class="alert alert-info">
+These are starting recommendations. Monitor your cluster's performance and resource utilization closely and adjust sizing as needed.
+</div>
+
+## Indexers
+
+- **Performance:** To index 5 MB/s of logs, CloudPrem needs approximately 1 vCPU and 2 GB of RAM.
+- **Recommended Pod Sizes:** We recommend deploying indexer pods with either:
+  - 2 vCPUs and 4 GB of RAM
+  - 4 vCPUs and 8 GB of RAM
+  - 8 vCPUs and 16 GB of RAM
+- **Storage:** Indexers require persistent storage (preferably SSDs, but local HDDs or remote EBS volumes can also be used) to store temporary data while constructing the index files.
+  - Minimum: 100 GB per pod
+  - Recommendation (for pods > 4 vCPUs): 200 GB per pod
+- **Example Calculation:** To index 1 TB per day (~11.6 MB/s):
+  - Required vCPUs: `(11.6 MB/s / 5 MB/s/vCPU) ≈ 2.3 vCPUs`
+  - Rounding up, you might start with one indexer pod configured with 3 vCPUs and 6 GB RAM, requiring a 100 GB EBS volume. (Adjust this configuration based on observed performance and redundancy needs.)
+
+## Searchers
+
+- **Performance:** Search performance depends heavily on the workload (query complexity, concurrency, data scanned).
+- **Rule of Thumb:** A general starting point is to provision roughly double the total number of vCPUs allocated to Indexers.
+- **Memory:** We recommend 4 GB of RAM per searcher vCPU. Provision more RAM if you expect many concurrent aggregation requests.
+
+## Other services
+
+The following components are typically lightweight:
+
+- **Control Plane:** 1 vCPU, 2 GB RAM
+- **Metastore:** 1 vCPU, 2 GB RAM
+- **Janitor:** 1 vCPU, 2 GB RAM
+
+## Postgres Metastore backend
+
+- **Instance Size:** For most use cases, a PostgreSQL instance with 1 vCPU and 4 GB of RAM is sufficient
+- **AWS RDS Recommendation:** If using AWS RDS, the `t4g.medium` instance type is a suitable starting point
+- **High Availability:** Enable Multi-AZ deployment with one standby replica for high availability
+
+## Further reading
+
+{{< partial name="whats-next/whats-next.html" >}}