Implement safe hybrid and multicloud log ingestion with Amazon OpenSearch Ingestion

Operating purposes throughout hybrid or multicloud environments creates a standard problem: fragmented logs scattered throughout totally different platforms. This fragmentation complicates monitoring, slows troubleshooting, and reduces operational visibility. To deal with this, many organizations search to implement safe log ingestion from all environments right into a centralized platform.

Amazon OpenSearch Service supplies a unified resolution for real-time search, analytics, and log administration throughout your complete infrastructure. Amazon OpenSearch Ingestion, a completely managed knowledge collector, simplifies knowledge processing with built-in capabilities to filter, remodel, and enrich your logs earlier than evaluation.

Nevertheless, securely sending logs from non-AWS environments presents a problem. Each request to OpenSearch Ingestion requires AWS Signature Model 4 (AWS SigV4) authentication, historically requiring long-term credentials that introduce safety dangers. AWS Identification and Entry Administration Roles Anyplace solves this downside by offering short-term credentials for workloads operating outdoors AWS.

On this publish, we reveal learn how to configure Fluent Bit, a quick and versatile log processor and router supported by varied working methods, to securely ship logs from any surroundings to OpenSearch Ingestion utilizing IAM Roles Anyplace. This strategy alleviates the necessity for long-term credentials whereas offering a complete view of your utility logs throughout all environments—bettering safety, simplifying operations, and enhancing your capacity to rapidly resolve points.

Options overview

The answer on this publish makes use of Fluent Bit to gather logs, retrieve short-term credentials from IAM Roles Anyplace, and signal HTTP log ingestion requests with AWS SigV4 earlier than sending them to the OpenSearch Ingestion pipeline. The next diagram reveals the structure.

This resolution provisions the next key elements:

• Certificates authority – For this publish, we use AWS Non-public Certificates Authority (AWS Non-public CA) because the certificates authority (CA) supply. Alternatively, you possibly can combine with an exterior CA; for extra particulars, see IAM Roles Anyplace with an exterior certificates authority. Certificates issued from public CAs can’t be used as belief anchors for IAM Roles Anyplace.
• X.509 Certificates – We use a pattern non-public certificates saved in AWS Certificates Supervisor (ACM) and issued by AWS Non-public CA.
• IAM Roles Anyplace configuration – This consists of the next:
  • Belief anchor – Establishes belief between IAM Roles Anyplace and the required CA.
  • IAM position – Grants permissions for log ingestion and trusts the IAM Roles Anyplace service principal. At minimal, this position have to be granted permission for the osis:Ingest motion.
  • Profile – Defines which roles IAM Roles Anyplace can assume and the utmost permissions granted with the short-term credentials.
• OpenSearch Service area – For this publish, we use an OpenSearch Service area, which is an AWS provisioned equal of an open supply OpenSearch cluster. We create the area inside a digital non-public cloud (VPC); see VPC versus public domains for extra data. Alternatively, you need to use an Amazon OpenSearch Serverless assortment, which is an OpenSearch cluster that scales compute capability primarily based in your utility’s wants.
• OpenSearch Ingestion – That is configured to obtain logs over HTTP because the pipeline supply and ahead them to the OpenSearch Service area because the pipeline sink.

Connectivity between AWS and your hybrid or multicloud environments

You’ll be able to entry your OpenSearch Ingestion pipelines utilizing an interface VPC endpoint with push-based HTTP supply, which supplies non-public IP tackle connectivity. For manufacturing environments, we suggest utilizing these non-public connections by interface endpoints for enhanced safety.

Establishing this connectivity requires extra configuration, similar to creating an AWS Web site-to-Web site VPN connection together with your hybrid and multicloud community. Though this publish focuses on the log ingestion resolution, you’ll find detailed steerage on community connectivity within the following assets:

How Fluent Bit retrieves short-term credentials utilizing IAM Roles Anyplace

Utilizing the HTTP output plugin, Fluent Bit can ship logs to the OpenSearch Ingestion pipeline. The next diagram is a simplified view of how Fluent Bit retrieves AWS credentials.

On Linux methods, Fluent Bit can use an AWS Command Line Interface (AWS CLI) profile that makes use of the credential_process parameter to set off an exterior course of. This exterior course of is invoked to generate or retrieve credentials in a roundabout way supported by the AWS CLI.

The next are two frequent mechanisms for the exterior course of:

Though each choices are viable, this publish focuses on IAM Roles Anyplace. On this setup, the AWS IAM Roles Anyplace Credential Helper is executed to deal with the signing course of for the CreateSession API. This returns credentials in a JSON format that Fluent Bit can devour.

As of this writing, the Fluent Bit aws_profile configuration is supported solely on Linux. It’s untested on different Unix-based methods (similar to macOS) and isn’t carried out for Home windows.

Conditions

Earlier than you start this walkthrough, be sure you have the next:

• AWS account necessities – This consists of:
  • An AWS account with permissions to deploy AWS CloudFormation templates.
  • Entry to AWS CloudShell for exporting a pattern non-public certificates we are going to create utilizing AWS CloudFormation in a later step.
• Distant (hybrid or multicloud) surroundings – You have to have a distant machine with Linux-based working system. This resolution was examined on Ubuntu 24.04 with the next extra tooling put in:

Deploy AWS assets with AWS CloudFormation

Comply with these steps to deploy AWS assets required for this resolution:

1. Select Launch Stack:
   
   
   
2. Enter a novel title for Stack title. The default worth is osis-with-iamra.
3. Configure the stack parameters. Default values are offered within the following desk.

4. Choose the acknowledgement examine field and select Create Stack.
   Stack deployment takes about half-hour to finish.
5. When stack creation is full, navigate to the Outputs tab on the AWS CloudFormation console and be aware down the values for the assets created.
   The next desk summarizes the output values.

Export a pattern non-public certificates utilizing CloudShell

Comply with these steps to export the pattern non-public certificates created by the CloudFormation stack:

1. Open CloudShell. For extra particulars, see Navigating the AWS CloudShell interface.
2. Export the certificates ARN from the CloudFormation outputs. In case you modified the stack title within the earlier step, use that worth for , in any other case use the default worth osis-with-iamra.

export CERT_ARN=$(aws cloudformation describe-stacks      --stack-name       --query 'Stacks[0].Outputs[?OutputKey==`ACMCertificateArn`].OutputValue'      --output textual content)

3. Extract the certificates and personal key recordsdata:

# Generate and save the passphrase export PASSPHRASE=$(openssl rand -base64 32) # Export certificates utilizing surroundings variables aws acm export-certificate      --certificate-arn $CERT_ARN      --passphrase $(echo -n "$PASSPHRASE" | base64)      > cert_export.json # Extract elements to separate recordsdata jq -r '.Certificates' cert_export.json > certificates.pem jq -r '.PrivateKey' cert_export.json > encrypted_private_key.pem # Decrypt the non-public key openssl rsa -in encrypted_private_key.pem -out private_key.pem -passin cross:"$PASSPHRASE" # Clear surroundings variables unset PASSPHRASE CERT_ARN

4. Obtain the extracted certificates and personal key recordsdata from CloudShell:
   1. /residence/cloudshell-user/certificates.pem
   2. /residence/cloudshell-user/private_key.pem

Configure an AWS CLI profile

Comply with these steps to configure an AWS CLI profile on your log ingestion surroundings:

1. Retailer the downloaded certificates and personal key to your surroundings. For an automatic strategy to generate and rotate certificates, see Arrange AWS Non-public Certificates Authority to situation certificates to be used with IAM Roles Anyplace.
2. Create a brand new profile named osis-pipeline-credentials that invokes the credential course of. Change the placeholders together with your particular values. Discover the values for trusted-anchor-arn, profile-arn, and ingestion-role-arn in your CloudFormation stack outputs.

aws configure set profile.osis-pipeline-credentials.credential_process " credential-process        --certificate         --private-key         --trust-anchor-arn         --profile-arn         --role-arn "

[profile osis-pipeline-credentials] credential_process =  credential-process       --certificate        --private-key        --trust-anchor-arn        --profile-arn        --role-arn 

cat  ~/fluent-bit.conf [INPUT]   title                  tail   path                  /var/log/syslog   read_from_head        true   refresh_interval      5 [OUTPUT]   title 			http   match	   		*   aws_service 		osis   host 			   port 			443   uri 			    format 		json   aws_auth	 	true   aws_region 		      aws_profile 		osis-pipeline-credentials   tls 			On EOF

pip set up awscurl export OPENSEARCH_DOMAIN_ENDPOINT=https:// # Record indices matching logs-%{yyyy.MM.dd} format and get most up-to-date one to question export INDEX=$(awscurl --service es "$OPENSEARCH_DOMAIN_ENDPOINT/_cat/indices?v" | grep -E "logs-[0-9]{4}.[0-9]{2}.[0-9]{2}" | kind -r | head -1 | awk '{print $3}') awscurl --service es $OPENSEARCH_DOMAIN_ENDPOINT/$INDEX/_search          -X GET -H "Content material-Sort: utility/json"          -d '{             "dimension": 10,             "kind": [               {"@timestamp": {"order": "desc"}}             ],             "question": { "match_all": {} }           }' | jq '.hits.hits[]._source'

{   "date": 1732039662.399506,   "log": "2024-11-19T18:07:42.399375+00:00 test-server fluent-bit[9986]: 200 OK",   "@timestamp": "2024-11-19T18:07:42.812Z" } {   "date": 1732039662.399501,   "log": "2024-11-19T18:07:42.399224+00:00 test-server fluent-bit[9986]: [2024/11/19 18:07:42] [ info] [output:http:http.0] test-pipeline-123456789012.us-east-2.osis.amazonaws.com:443, HTTP standing=200",   "@timestamp": "2024-11-19T18:07:42.812Z" } ...