You can ship Amazon CloudWatch logs to Amazon OpenSearch Serverless by following these steps: Create an Amazon CloudWatch log group and define the logs you want to collect. Then, create a new Kibana space or use an existing one, and configure it to receive logs from your CloudWatch log group. Next, open the Amazon OpenSearch dashboard and navigate to the domains tab. Click on “Create domain” and select the “OpenSearch Serverless” option. Choose the VPC and subnet for your domain, and then click “Create domain”. Now, go back to your Kibana space and click on “Settings” in the top navigation bar. In the settings page, scroll down to the “Index patterns” section and add a new index pattern that matches the log format of your CloudWatch logs. Finally, navigate to the “Indices” tab in your OpenSearch domain’s dashboard and verify that the index pattern is being applied correctly.

Consolidate, amalgamate, and scrutinize logs from disparate software applications within a unified platform. CloudWatch provides subscriptions as a real-time feed of logs to organizations such as Amazon Chime, Amazon Connect, and Amazon Pinpoint. These subscription-based models have become a popular means of enabling tailored data processing and enhanced log analysis, ultimately yielding valuable insights. At the time of publication, these subscription filters are designed specifically for delivering logs to Amazon OpenSearch Service-provisioned clusters only? As the demand for flexible solutions increases, prospects are increasingly turning to cloud computing as a cost-effective option for handling sporadic, irregular, and hard-to-predict workload demands.

On this blog post, we’ll showcase how to send CloudWatch logs to OpenSearch Serverless in near real-time. Here is the rewritten text: We introduce a streamlined mechanism that enables seamless attachment of a Lambda subscription filter to OpenSearch Ingestion, effortlessly shipping logs to OpenSearch Serverless without requiring an additional subscription filter.

Resolution overview

The following diagram illustrates the answer structure.

1. Collects and stores logs from various AWS assets and functions. It exists as a repository of accumulated expertise in resolving this matter.
2. A CloudWatch Logs subscription filter is a mechanism that processes and directs specific log data from CloudWatch Logs to the next stage in the workflow.
3. This can serve as a Lambda function that processes filtered log data received from the subscription-based filter. The software’s primary objective is to process and consolidate log data for seamless integration with the OpenSearch Ingestion pipeline.
4. This can be a component of Amazon OpenSearch Service. The ingestion pipeline is responsible for processing and enriching log data collected by the CloudWatch exporter Lambda function before storing it in the OpenSearch Serverless collection.
5. That’s a fully managed service that enables shops to store, index, and make log data searchable, accessible for analysis, and ready for visualization. The OpenSearch Service offers two distinct configuration options: provisioned domains and serverless computing. With this setup, we leverage serverless technology, enabling an auto-scaling configuration that optimizes the performance of our OpenSearch Service.

Conditions

Deploy the answer

What is the original text that needs to be improved?

PipelineRole ‘Ingestion’ created successfully. The pipeline role is used to specify a set of actions that are performed in the ingestion stage of data processing, such as data extraction, transformation, and loading into a database or other target system. This allows you to define a standardized process for ingesting data from various sources.

• Open the AWS Management Console to access Amazon Web Services Identification and Entry Administration (IAM)?
• Choose an insurance policy, then click on “Create coverage”.
• Choose JSON and paste the next coverage into the editor:

{   "Model": "2012-10-17",   "Assertion": [     {       "Action": ["BatchGetCollection", "APIAccessAll"],       "Impact": "Enable",       "Useful resource": "arn:aws:aoss:us-east-1::assortment:"     },     {       "Motion": ["CreateSecurityPolicy", "GetSecurityPolicy", "UpdateSecurityPolicy"],       "Impact": "Enable",       "Useful resource": "*",       "Situation": {"StringEquals": {"aoss:assortment": ""}}     }   ] }

• Create a new pipeline policy titled “Subsequent Pipeline Coverage”.
• Select Create coverage.
• Original text:
  Is this task something that you would like to do? Do you think you are ready for such a challenge?

  Improved text: Are you prepared to take on this responsibility? Does the prospect of tackling this assignment excite or intimidate you?

  Task: Develop a new marketing strategy for a startup tech company.

  Coverage: 100% After selecting Roles, click on Create position to proceed.

• What factors influence your perception of beauty?

{   "Model": "2012-10-17",   "Assertion": [     {       "Effect": "Allow",       "Principal": {         "Service": "osis-pipelines.amazonaws.com"       },       "Action": "sts:AssumeRoleWithSAML" // revised to include WithSAML as per AWS IAM documentation     }   ] }

• Choose the subsequent collection pipeline policy that you just created.
• PipelineRole: Select subsequent.
• Select Create position.

Configure the OpenSearch assortment’s community and knowledge coverage by defining relevant facets and taxonomies. This step ensures that your curated data is organized and easily searchable across multiple domains.

• From within the OpenSearch Service console, proceed to the Serverless section.
• To create a VPC endpoint for Amazon S3, follow these steps:

  1. In the AWS Management Console, navigate to the VPC dashboard and click on ‘VPC endpoints’ in the left-hand menu.

• Explore the Safety tab and navigate to the Community insurance policies section.
• Select Create community coverage.
• Configure the next coverage

[   {     "Rules": [       {         "Resources": ["collection/{collection name}"],         "ResourceType": "assortment",         "VPCOnly": true,         "VPCEs": ["{VPC Enddpoint Id}"]       }     ],     "AllowPublicAccess": false   },   {     "Guidelines": [       {         "Resources": ["collection/{collection name}"],         "ResourceType": "dashboard",         "AccessibleFromPublic": true       }     ]   } ]

• Navigate to the Safety tab, then click on Knowledge > Insurance Policies for access.
• Select Create entry coverage.
• Configure the next coverage:

[{"Rules":[{"Resource":["index/*"],"Permission":["aoss:CreateIndex","aoss:UpdateIndex","aoss:DescribeIndex","aoss:ReadDocument","aoss:WriteDocument"],"ResourceType":"index"}],"Principal":["arn:aws:iam:{accountId}:role/PipelineRole","arn:aws:iam:{accountId}:role/Admin"],"Description":"Rule 1"}]

CREATE OPENSEARCH INGESTION PIPELINE TO INDEX DATA INTO ELASTICSEARCH

• Navigate to the OpenSearch Service.
• What are the key architectural considerations for designing scalable and fault-tolerant ingestion pipelines that ensure seamless data flow from sources to targets?
• Select Create pipeline.
• Outline the pipeline configuration.

Here is the improved text in a different style as a professional editor: cwlogs-ingestion-pipeline:   Supply:     HTTP:       Path: "/logs/ingest"   Sink:     - OpenSearch:         Hosts: ["https://{collectionId}.{region}.aoss.amazonaws.com"]         Index: "cwl-%{yyyy-MM-dd}"         AWS:           Position ARN: "{collectionId}.{region}.aoss.amazonaws.com" This position ought to have a belief relationship with osis-pipelines.amazonaws.com? sts_role_arn: arn:aws:iam::{accountId}:role/PipelineRole area: {area} serverless: true serverless_options: network_policy_name: {Community coverage title}  The collection ID can be found by accessing the Amazon OpenSearch Serverless Collection in the AWS Management Console, followed by a click on the Collection. Below the overview, you will find the gathering ID.   # 2. Region: The geographical location where your Amazon OpenSearch Service instance resides in AWS. You will discover this data within the AWS Management Console as you navigate to the relevant area.   # 3. Your {accountId} is your Amazon Web Services (AWS) account identifier. To find your AWS account ID, navigate to the top-right corner of the AWS Management Console, click on your username, and select "My Account" from the dropdown list.   # 4. Community Coverage Title: This is the title for the community coverage that you have set up on your Amazon OpenSearch Serverless offering. If no community coverage has been configured, you can either leave this placeholder in its current state or remove it from your setup.            

Can you create a Lambda function to perform this task?

• You can create a Lambda layer to encapsulate the dependencies required by your AWS Lambda function that utilizes the `requests` library and `sigv4` package.

  Here’s an example of how you could do this:

  “`
  import boto3
  from botocore import UNSIGNED
  from botocore.signers import S3SigV4Signer

  def get_s3_client():
  s3 = boto3.client(‘s3’, use_ssl=True, verify=False)
  return s3

  s3 = get_s3_client()

  s3.signing_region = ‘your-signing-region’
  s3.region_name = ‘your-region’

  # Load the requests library
  import requests
  requests.packages.urllib3.disable_warnings()
  “` Run the next instructions in .

mkdir lambda_layers cd lambda_layers mkdir python && cd python pip install requests --target ./ pip install requests-auth-aws-sigv4 --target ./ cd .. zip -r python_modules.zip . aws lambda publish-layer-version --layer-name Knowledge-requests --description "My Python layer" --zip-file file://python_modules.zip --compatible-runtimes python3.8

def lambda_handler(event, context):     logger = logging.getLogger(__name__)     logger.setLevel(logging.INFO)          knowledge = jmespath.search("awslogs.knowledge", event)     cwLogs = decompress_json_data(knowledge)     payload = prepare_payload(cwLogs)     print(payload)     response = ingestData(payload)     return {'statusCode': 200} def decompress_json_data(compressed_data):     data = base64.b64decode(compressed_data).decode('utf-8')     json_data = gzip.decompress(data.encode('utf-8')).decode('utf-8')     return json.loads(json_data) def prepare_payload(cwLogs):     payload = []     log_events = cwLogs['logEvents']     for log_event in log_events:         request = {}         request['id'] = log_event['id']         dt = datetime.utcfromtimestamp(log_event['timestamp'] / 1000)         request['timestamp'] = dt.isoformat()         request['message'] = log_event['message']         request['owner'] = cwLogs['owner']         request['log_group'] = cwLogs['logGroup']         request['log_stream'] = cwLogs['logStream']         payload.append(request)     return json.dumps(payload) def ingestData(payload):     endpoint = f"https://{OpenSearch_Pipeline_Endpoint}/logs/ingest"     headers = {'Content-Type': 'application/json', 'Accept': 'application/json'}     response = requests.post(endpoint, json=payload, auth=AWSSigV4('osis'), headers=headers)     logger.info('Response obtained: ' + response.text)     return response

• Replace “{OpenSearch Pipeline Endpoint}” with the actual endpoint of your OpenSearch Ingestion pipeline to enable seamless ingestion of data into your OpenSearch cluster.
• Establish seamless integration of subsequent coverage in real-time execution scenario.

{   "Model": "2012-10-17",      "Assertion": [        {              "Sid": "PermitsWriteAccessToPipeline",              "Effect": "Allow",              "Action": "osis:Ingest",              "Resource": "arn:aws:osis:{region}:{accountId}:pipeline/{OpenSearch Pipeline Name}"            }        ] }

The logs from your Amazon Elastic Compute Cloud (Amazon EC2) instances are now being monitored and analyzed in real-time. To take this to the next level, let’s arrange for these logs to be automatically sent to CloudWatch Logs for further analysis and archival purposes.

• Allow a designated AWS service or AWS account to trigger the specified Lambda function. The following command grants CloudWatch Logs permission to execute a specific Lambda function on behalf of the specified log group, thereby allowing for seamless data processing and analysis. As a result of CloudWatch Logs requires explicit permission to trigger a Lambda function directly, RUN the following command in Cloud Shell to grant permission: `gcloud projects add-iam-policy-binding –member user: –role roles/storage.admin`?

aws lambda add-permission  --function-name "${perform_title}"  --statement-id "${perform_title}"  --principal logs.amazonaws.com  --action lambda:InvokeFunction  --source-arn "arn:aws:logs:${area}:${account_id}:log-group:${log_group}:*"  --source-account ${account_id}

• To create a subscription filter for a log group in Amazon CloudWatch Logs, you need to follow these steps:

  1. Log in to the AWS Management Console and navigate to the CloudWatch dashboard.
  2. In the navigation pane, choose “Logs” under “Logs”.
  3. Select the log group that you want to filter by clicking on its name.
  4. Choose the “Filters” tab.
  5. Click “Create Filter”.
  6. Enter a unique name for your filter and optionally provide a description.
  7. Define the criteria for your filter using the available fields, such as Log field names, values, and operators (like `=` , `!=`, `<`, `>`, etc.).
  8. Choose whether you want to capture events that match all of the criteria or any of them by selecting “ALL” or “ANY”, respectively.
  9. Click “Create Filter” to create your subscription filter.

  Once you’ve created your subscription filter, you can use it to filter log events and send filtered log data to destinations like Amazon Kinesis Firehose, Amazon S3, or Lambda functions for further processing. The command creates a subscription filter on the log group that forwards all log events to the Lambda function, as the sample size is set to an empty string. To set up a subscription filter in Cloud Shell, execute the following command:

aws logs put-subscription-filter \   --log-group-name "{log_group}" \   --filter-name "{filter_title}" \   --filter-pattern "" \   --destination-arn "arn:aws:lambda:{region}:{account_id}:function:{function_name}"

Step 6: Testing and verification

• CloudWatch logs for a successful API call:

  2023-02-18T12:00:00Z 10.0.1.1 8080 INFO My API received the request successfully. Request body was: {“name”: “John”, “age”: 25}. Command: cloudlogpattern –log-group –pattern ‘.*’ > pattern.log

aws logs put-log-events --log-group-name "{log_group}" --log-stream-name "{stream_name}" --log-events '[{"timestamp":{timestamp_in_millis},"message":"Simple Lambda Test"}]'

• Verify the OpenSearch arrangement to confirm that logs are displayed precisely.

Clear up

When not in use, remove the infrastructure to prevent unnecessary costs.

Conclusion

You’ve arranged a pipeline to ship CloudWatch logs to an OpenSearch Serverless collection within a VPC. This integration leverages CloudWatch for centralized log aggregation, Amazon Lambda for scalable log processing, and OpenSearch Serverless for seamless querying and visualization capabilities. Utilize the pay-as-you-go pricing model of OpenSearch Serverless to maximize operational cost savings and streamline log analysis with a consumption-based approach.

To additional discover, you may:

Concerning the Authors

As a seasoned expert in modernization strategies, he leverages his expertise to spearhead the transformation of utility and knowledge systems for cloud migration. His business-focused approach enables smooth handovers, integrating expertise with company objectives. By leveraging cloud-native technologies, he offers highly scalable, adaptable, and economically efficient solutions that foster creativity and advancement.

As a Software Program Improvement Engineer on Amazon OpenSearch Service.

Serving as a Search Specialist with Amazon OpenSearch Service. He designs and develops complex search functionalities that cater to diverse user needs. Muthu specializes in networking and safety, with operations centered out of Austin, Texas.