Amazon Kinesis Data Firehose ships real-time data from Apache Kafka topics to Amazon OpenSearch Service domains through Amazon OpenSearch Ingestation.

This publication showcases a step-by-step guide on how to employ buffering and combining capabilities for processing real-time streaming data in domains and collections via . This innovative approach enables seamless application across diverse scenarios, encompassing real-time log analysis and synchronization with software messaging data for instant querying. We focus on a real-world scenario where a company with stringent compliance requirements needs to consolidate log collection and retention practices to ensure seamless auditing and regulatory adherence.

Kinesis Knowledge Streams is a fully managed, serverless real-time data streaming service that cost-effectively stores and processes large volumes of diverse streaming data at massive scale. In various log analytics scenarios, Amazon Kinesis Data Firehose streamlines log aggregation by separating producer and consumer functionalities, providing a fault-tolerant, highly available buffer that captures and delivers log data efficiently. By decoupling various components, organizations gain significant advantages over traditional system designs. As logging operations grow or shrink in scope, Amazon Kinesis Data Streams enables dynamic scaling of knowledge streams, ensuring persistent buffering of log data. This feature ensures that load adjustments do not affect an OpenSearch Service domain, providing a robust repository of log data for analysis. This feature allows multiple clients to process log data in real-time, providing a persistent repository of up-to-date information that can be consumed by various applications. This setup enables the log analytics pipeline to adhere to best practices for achieving both resilience and cost-effectiveness.

OpenSearch Ingestion offers a seamless, serverless pipeline empowering users with robust tools to extract, transform, and load data efficiently into their designated OpenSearch Service domain. OpenSearch Ingestion seamlessly integrates with numerous AWS services, providing pre-configured blueprints that accelerate the ingestion process for various analytics scenarios within OpenSearch Service domains. By integrating with Kinesis Knowledge Streams, OpenSearch Ingestion enables sophisticated, real-time analysis of data, significantly reducing the burden of building a scalable, real-time search and analytics infrastructure.

Resolution overview

When considering this resolution, we ponder a typical scenario for centralized log aggregation within a corporate environment. Organisations may consider implementing a centralised log aggregation approach for various reasons. Organizations often require compliance and governance protocols that dictate specific logging requirements, including the types of data that must be recorded, retention periods, and searchability for investigative purposes. Organizations seek to integrate software and safety operations by providing widespread access to real-time observability tools and capabilities across their teams.

To meet these demands, it is recommended to collect data from log sources (producers) through a scalable, resilient, and cost-efficient approach that ensures seamless integration with existing systems and infrastructure. Sources of logs may vary depending on software and infrastructure utilization scenarios and configurations, as depicted in the table below.

The accompanying diagram elucidates an instance structure.

You should leverage Kinesis Knowledge Streams to tackle a broad spectrum of scenarios. You can configure logs to send data to Kinesis Data Firehose, which will then stream the information to Amazon Knowledge, using a subscription filter (refer to our documentation). When shipping data with Kinesis Knowledge Streams for analytics purposes, leverage OpenSearch Ingestion to establish a scalable and extensible pipeline that consumes your streaming data and writes it to OpenSearch Service indexes. Amazon Kinesis Data Firehose provides a scalable buffer to help numerous customers, featuring customizable retention periods, and seamless integration with a diverse array of AWS services. For diverse use cases, location data is stored in Amazon S3, whereas an agent writes similar information directly onto OpenSearch Ingestion without an intermediate buffer, leveraging its built-in persistent buffers and automated scaling capabilities.

Standardizing logging approaches minimizes growth and operational overhead for organizations. To ensure consistent monitoring, you could standardize on all function logging to CloudWatch logs whenever feasible, while addressing Amazon S3 logs where CloudWatch logging is unavailable? This simplification enables a centralized team to effectively manage a broader range of scenarios within their log aggregation approach, thereby reducing the complexity associated with log aggregation solutions. To optimize processing for advanced growth teams, consider standardizing the use of FluentBit brokers to stream data directly into OpenSearch Ingestion, thereby reducing costs and minimizing the need to store logs in CloudWatch.

Utilizing CloudWatch logs as an informational supply enables effective log aggregation in this resolution. For the Amazon S3 log use case, consider leveraging AWS Lake Formation to enable scalable and governed data processing. For agent-based options, refer to the respective documentation on integrating with OpenSearch Ingestion, analogous to .

Stipulations

Several crucial components of infrastructure utilized in this resolution necessitate the ingestion of data into OpenSearch Service via OpenSearch Ingestion.

• A Kinesis data stream enables you to combine log information from CloudWatch.
• A dedicated OpenSearch space for storing log data.

When setting up a Kinesis information stream, we recommend starting in On-Demand mode initially. This feature enables automatic scaling of the number of shards required for your log throughput in Kinesis Knowledge Streams. Upon establishing a consistent baseline for our log aggregation use case, we strongly recommend transitioning to Provisioned mode, leveraging the predefined shard configurations identified during the On-Demand phase. By leveraging this information, you can effectively optimize your long-term pricing strategy to ensure a stable and profitable approach in high-throughput usage scenarios.

While we strongly recommend processing a single Kinesis information stream for your log aggregation needs. OpenSearch ingestion enables significant scalability, supporting up to 96 OpenSearch cluster units (OCUs) per pipeline, and accommodating pipeline definitions of up to 24,000 characters in a single file. Each pipeline can process up to 96 shards from a Kinesis data stream, as each OCU handles one shard. By leveraging a single Kinesis information stream, you can streamline the process of aggregating log data in OpenSearch Service, as well as simplify the creation and management of subscription filters for your logging teams.

Considering the scale of your log workloads and the intricacy of your OpenSearch ingestion pipeline’s logic, you may want to explore additional Kinesis data streams tailored to your specific use case. To streamline your manufacturing workflow, consider dedicating a distinct stream to each primary log type. By separating log information into distinct streams based on specific use cases, organizations can significantly reduce operational complexity in managing OpenSearch ingestion pipelines, enabling them to scale and deploy adjustments to each log use case independently as needed.

To create a Kinesis Knowledge Stream, refer to the AWS documentation on setting up and configuring this feature.

To establish a new OpenSearch region, refer to https://opensearch.org/docs/regions/.

Configure log subscription filters

CloudWatch log groups allow for subscription filter implementation at either the account level or log group level. In every situation, we recommend implementing a subscription filter with a randomized distribution method to guarantee that log data is consistently dispersed across Kinesis data stream shards.

Account-level subscription filters enable administrators to apply the same filtering settings to all log teams within an account, streamlining log management and providing a centralized location for monitoring log data. If you intend to store all your log data in Amazon OpenSearch Service using Amazon Kinesis Data Streams, A single account-level subscription filter is restricted to one per account. By leveraging Kinesis Knowledge Streams as the destination, you can facilitate multiple log consumers to process account log data in real-time, enabling efficient and scalable logging operations. To set up an account-level subscription filter, follow these steps.

Log group-level subscription filters are applied to each log group. This method effectively facilitates storing subsets of log data in OpenSearch Service using Kinesis Data Streams, allowing for the processing and retention of multiple log types from various streams. There is a restriction of up to two log group-level subscription filters per log group. When creating a log group-level subscription filter, refer to .

Upon creating your subscription filter, verify that log data is successfully transmitted to your Amazon Kinesis data stream. Navigate to the Kinesis Knowledge Streams console and click on the link corresponding to your stream’s title.

Retrieve a specific shard by setting the database name and selecting the desired schema.

The data must comprise a single column of significant value and a complementary binary column. CloudWatch sends log data in compressed .gzip format to minimize the size of log records.

Configure an OpenSearch Ingestion pipeline

Once you’ve established a Kinesis information stream and configured CloudWatch subscription filters to send data to that stream, you’ll be able to set up an OpenSearch ingestion pipeline to process your log data seamlessly. To begin with, you craft an IAM function that grants learning access to the Kinesis data stream and read/write permissions to the OpenSearch domain. To establish a pipeline, the supervisor’s function responsible for constructing the pipeline necessitates iam:PassRole The necessary permissions have been granted to the pipeline function at this stage.

1. Can you execute a Lambda function with the necessary permissions to read from your Kinesis info stream and write to your OpenSearch domain?

   {
     "Model": "2012-10-17",
     "Assertions": [
       {
         "StatementId": "allowReadFromStream",
         "Effect": "Allow",
         "Actions": [
           "kinesis:DescribeStream",
           "kinesis:DescribeStreamConsumer",
           "kinesis:DescribeStreamSummary",
           "kinesis:GetRecords",
           "kinesis:GetShardIterator",
           "kinesis:ListShards",
           "kinesis:ListStreams",
           "kinesis:ListStreamConsumers",
           "kinesis:RegisterStreamConsumer",
           "kinesis:SubscribeToShard"
         ],
         "Resource": [
           "arn:aws:kinesis:*:*:stream/*"
         ]
       },
       {
         "StatementId": "allowAccessToOS",
         "Effect": "Permit",
         "Actions": [
           "es:DescribeDomain",
           "es:ESHttp*"
         ],
         "Resource": [
           "arn:aws:es:*:*:domain/*",
           "arn:aws:es:*:*:domain/*/index/*"
         ]
       }
     ]
   }

2. What specific requirements must the input meet to ensure successful processing? The function should be designed to accommodate inputs that are comprised of:

   {
     "Model": "2012-10-17",
     "Assertion": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": ["osis-pipelines.amazonaws.com"]
         },
         "Action": "sts:AssumeRole",
         "Condition": {
           "StringEquals": {"aws:SourceAccount": "{account-id}"},
           "ArnLike": {"aws:SourceArn": "arn:aws:osis:{area}:{account-id}:pipeline/*"}
         }
       }
     ]
   }

To write information to an OpenSearch site via a pipeline, the region requires an identity that grants permission for the pipeline function to access it. If your area employs IAM, map the IAM function to a backend function within the OpenSearch Service security plugin, enabling creation and writing privileges to indexes.

3. Within the OpenSearch Service console’s navigation pane, select the option that appears underneath after creating your pipeline function.
4. Select .
   
5. Stream data into Amazon Kinesis.
   
6. Set the pipeline’s reputation to match the number of shards in your Amazon Kinesis data stream.

When using On-Demand mode for your information stream, ensure that the selected capability matches the current number of shards in the stream. As this scenario does not necessitate a persistent buffer, leveraging Kinesis Knowledge Streams’ inherent persistence capabilities for logging data allows OpenSearch Ingestion to efficiently track its position within the Kinesis stream, thereby preventing information loss during restarts.

7. Replace the pipeline supply settings with your Kinesis data stream title and pipeline IAM function’s Amazon Resource Name (ARN), leveraging its capabilities.

All configuration details can be found in . In most cases, it’s recommended to utilize the default settings. By design, the pipeline automatically writes batches of 100 documents every one-second interval, and allows for real-time subscription to the Kinesis information stream starting from the latest point in the stream using, with checkpointing occurring every two minutes. To optimize buyer engagement, you can dynamically adjust checkpoint intervals, initiate the process at varied points in the stream, and leverage polling to reduce costs associated with increased fan-out.

  kinesis-data-streams:
  acknowledgments: true
  codec:
    json:
      key_name: logEvents
      include_keys: [owner, logGroup, logStream]
  streams:
  - stream_name: KINESIS_STREAM_NAME
    initial_position: EARLIEST
    compression: gzip
  aws:
    Function: ARN with KDS access This function should maintain a belief relationship with osis-pipelines.amazonaws.com?
    sts_role_arn: "${PIPELINE_ROLE_ARN}"
    area: ${REGION}

8. Replace the pipeline sink settings with your OpenSearch domain’s endpoint URL and pipeline’s IAM execution role ARN.

The IAM role’s Amazon Resource Name (ARN) must match exactly between each OpenSearch Service sink definition and corresponding Kinesis Data Streams source definition. You can manage which information gets listed in multiple indexes using the index definition within the sink. To better facilitate querying and data retrieval, consider leveraging metadata regarding the Kinesis information stream title to effectively index by information stream.${getMetadata("kinesis_stream_name")You should utilise doc fields to index information depending on the CloudWatch log group or distinct doc details.${path/to/subject/in/doc}). On this occasion, we utilize three document-level fieldsdata_stream.kind, data_stream.dataset, and data_stream.namespaceWithin our pipeline’s processor logic, we need to establish a method for indexing our paperwork, thereby creating corresponding fields.

  

sink:
  - opensearch:
      # Expose an AWS OpenSearch Service domain endpoint
      hosts: ["OPENSEARCH_ENDPOINT"]
      # Route log information to separate destination indexes based on the log context:
      index: "${data_stream.kind}-${data_stream.dataset}-${data_stream.namespace}"
      aws:
        # Provide a Lambda function ARN with access to this domain The following function should have a belief relationship with osis-pipelines.amazonaws.com, mirroring the equivalent functionality used above for Kinesis. 
sts_role_arn: "PIPELINE_ROLE_ARN"
area: REGION
serverless: boolean(false) // Allow the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection

Ultimately, you’ll have the ability to modify the pipeline configuration to integrate custom definitions and refactor your log data before sending it to the OpenSearch domain for processing. This use case leverages the OpenSearch Ingestion pipeline to define a customized schema for SS4O. This functionality includes integrating frequent fields to affiliate metadata with the corresponding paperwork, as well as parsing log data to enhance information searchability. This use case leverages the log group title to create distinct log types as datasets, utilizing this data to generate reports that are indexed according to their respective use cases.

9. Renamed CloudWatch occasion timestamp to mark the noticed timestamp when the log was generated using, ” @timestamp” and added the current timestamp as the processed timestamp when OpenSearch Ingestion handled the file using “:@processed_timestamp”.

     Processor logic transforms OpenSearch log information parsing. 
   
   processor:
     - rename_keys:
         entries:
         - 
           from_key: "timestamp"
           to_key: "observed_timestamp"
       - date:
         from_time_received: true
         destination: "processed_timestamp"

10. The document processing pipeline successfully integrated metadata into the processed document, leveraging log group and log stream identifiers from Amazon CloudWatch Logs, account ID and AWS Region details, and rich dataset metadata from the Kinesis information stream.

        - add_entries:
        entries:
          # Assist SS4O frequent log fields
          - key: cloud/supplier, worth: aws
          - key: cloud/account/id, format: ${proprietor}
          - key: cloud/area, worth: us-west-2
          - key: aws/cloudwatch/log_group, format: ${logGroup}
          - key: aws/cloudwatch/log_stream, format: ${logStream}
          # Embody default values for the data_stream:
          - key: data_stream/namespace, worth: default
          - key: data_stream/kind, worth: logs
          - key: data_stream/dataset, worth: common
          # Embody metadata in regards to the supply Kinesis message that contained this log occasion:
          - key: aws/kinesis/stream_name, value_expression: getMetadata(stream_name)
          - key: aws/kinesis/partition_key, value_expression: getMetadata(partition_key)
          - key: aws/kinesis/sequence_number, value_expression: getMetadata(sequence_number)
          - key: aws/kinesis/sub_sequence_number, value_expression: getMetadata(sub_sequence_number)

11. Use to replace the data_stream.dataset Fields reliant on the log supply are used to regulate which index the document is written to, and utilised to delete the uniquely-named CloudWatch document fields that have been re-designated.

        - add_entries:
        entries:
          - key: data_stream.dataset
            value: cloudtrail
            when: accommodates(/logGroup, "cloudtrail") or accommodates(/logGroup, "CloudTrail")
            overwrite_if_exists: true
          - key: data_stream.dataset
            value: lambda
            when: accommodates(/logGroup, "/aws/lambda/")
            overwrite_if_exists: true

12. The log message fields are parsed to enable structured and JSON information for enhanced searchability within OpenSearch indexes via the use of.

Grok processors utilize sample matching techniques to extract relevant information from predefined textual data structures. Here are some examples of built-in Grok patterns: see and .

    # Use Grok parser to parse non-JSON apache logs
  - grok:
    grok_when: "/data_stream/dataset == 'apache'"
    match: ['%{COMMONAPACHELOG_DATATYPED}']
    target_key: "http"

# Parse log information as JSON for field-level searches within the OpenSearch index
- parse_json:
  supply: "message"
  target_key: "logs"
  parse_when: "/data_stream/dataset in ['cloudtrail', 'lambda', 'common']"
  tags_on_failure: ["json_parse_fail"]

When everything is combined, your pipeline configuration will resemble this code:

kinesis-pipeline:
  supply:
    kinesis-data-streams:
      acknowledgments: true
      codec:
        json:
          key_name: "logEvents"
          include_keys: ["owner", "logGroup", "logStream"]
      streams:
        - stream_name: ?KINESIS_STREAM_NAME?
          initial_position: LATEST
          compression: gzip
      aws:
        function_arn: ?FUNCTION_ARN? sts_role_arn: "${PIPELINE_ROLE_ARN}"
area: "${REGION}"

processor:
  - rename_keys:
      entries:
        - from_key: "timestamp"
          to_key: "observed_timestamp"
    - date:
        from_time_received: true
        destination: "processed_timestamp"
    - add_entries:
        entries:
          - key: "cloud/supplier"
            value: "aws"
          - key: "cloud/account/id"
            format: "${proprietor}"
          - key: "cloud/area"
            value: "us-west-2"
          - key: "aws/cloudwatch/log_group"
            format: "${logGroup}"
          - key: "aws/cloudwatch/log_stream"
            format: "${logStream}"
          - key: "data_stream/namespace"
            value: "default"
          - key: "data_stream/kind"
            value: "logs"
          - key: "data_stream/dataset"
            value: "common"
          - key: "aws/kinesis/stream_name"
            value_expression: "getMetadata('stream_name')"
          - key: "aws/kinesis/partition_key"
            value_expression: "getMetadata('partition_key')"
          - key: "aws/kinesis/sequence_number"
            value_expression: "getMetadata('sequence_number')"
          - key: "aws/kinesis/sub_sequence_number"
            value_expression: "getMetadata('sub_sequence_number')"

  - add_entries:
      entries:
        - key: "data_stream/dataset"
          value: "cloudtrail"
          add_when: "matches('/logGroup', 'cloudtrail') or matches('/logGroup', 'CloudTrail')"
          overwrite_if_key_exists: true
        - key: "data_stream/dataset"
          value: "lambda"
          add_when: "matches('/logGroup', '/aws/lambda/')"
          overwrite_if_key_exists: true
        - key: "data_stream/dataset"
          value: "apache"
          add_when: "matches('/logGroup', '/apache/')"
          overwrite_if_key_exists: true

  - delete_entries:
    - with_keys:
      - "logGroup"
      - "logStream"
      - "proprietor"

  - grok:
    - grok_when: "/data_stream/dataset == 'apache'"
    - match:
      - message: ['%{COMMONAPACHELOG_DATATYPED}']
    - target_key: "http"

  - parse_json:
    - supply: "message"
    - destination: "aws/cloudtrail"
    - parse_when: "/data_stream/dataset == 'cloudtrail'"
    - tags_on_failure: ["json_parse_fail"]

  - parse_json:
    - supply: "message"
    - destination: "aws/lambda"
    - parse_when: "/data_stream/dataset == 'lambda'"
    - tags_on_failure: ["json_parse_fail"]

  - parse_json:
    - supply: "message"
    - destination: "body"
    - parse_when: "/data_stream/dataset == 'common'"
    - tags_on_failure: ["json_parse_fail"]

sink:
  - opensearch:
    - hosts: [ "${OPENSEARCH_ENDPOINT}" ]
    - index: "ss4o_${data_stream/kind}-${data_stream/dataset}-${data_stream/namespace}"
    - aws:
      - role_arn: "${ROLE_ARN}" This function should maintain a trust relationship with osis-pipelines.amazonaws.com, identical to that used for Kinesis; sts_role_arn defaults to "PIPELINE_ROLE_ARN". The area parameter specifies the region. Optional 'serverless' flag enables Amazon OpenSearch Serverless if set to true; default is false.

13. When your configuration is at maximum capacity, please select to check your pipeline syntax for any potential mistakes?
    
14. Here is the rewritten text:

    To initiate your pipeline execution, you may choose to append a suffix to the service function, thereby creating a singular entry point for starting your workflow.
    

15. What is your original text that you would like me to improve?

When setting up a Kinesis Knowledge Streams supply, avoid selecting a digital personal cloud (VPC), subnets, or security groups. OpenSearch Ingestion exclusively relies on the following attributes for HTTP information sources situated within a Virtual Private Cloud (VPC). Kinesis Knowledge Streams leverage OpenSearch Ingestion to tap into the power of machine learning on data stored in Kinesis Knowledge Streams, then writing the results to OpenSearch domains or serverless collections.

16. Enable optional CloudWatch logging to your pipeline for enhanced monitoring and analytics capabilities.
17. What’s the best approach for creating a tailored pipeline?

When using account-level subscription filters for CloudWatch logs within the account where OpenSearch Ingestion is running, ensure that the log group is excluded from the account-level subscription. It’s possible that OpenSearch Ingestion pipeline logs triggering a recursive loop with the subscription filter can lead to an explosion of log data, potentially causing unanticipated costs due to increased ingestion volume?

18. The team’s enthusiasm was palpable as they discussed their project.
    

As your pipeline transitions into its designated state, a flurry of logs will begin to accumulate within your OpenSearch domain or serverless storage arrangement.

Monitor the answer

To maintain optimal functioning and ensure the overall wellness of the log ingestion pipeline, several critical aspects require close monitoring:

• Monitor these key performance indicators:
  • The CloudWatch subscription filter fails to write data to the designated Kinesis information stream, indicating an underlying issue requiring investigation and resolution. Reach out to AWS Support if this metric remains at a non-zero level for an extended period.
  • Kinesis information streams occasionally require additional shards to handle the growing volume of logs emanating from CloudWatch, ensuring seamless data processing and analytics.
  • When your Kinesis information stream experiences unexpectedly high customer demand, exceeding the allocated shard limits, consider adopting a more advanced fan-out client approach to optimize performance.
  • Your Kinesis information stream requires additional shards to handle the volume of logs generated by CloudWatch, as well as any irregularities in log distribution among existing shards. To ensure the subscription filter’s randomness, consider implementing a truly randomized approach, rather than relying solely on a pseudo-random method. This can be achieved by utilizing a cryptographically secure pseudo-random number generator (CSPRNG) to generate a unique identifier for each shard. Additionally, consider enabling enhanced shard-level monitoring on the information stream to proactively identify and mitigate any issues with scorching shards.
  • The configuration of your client may be incorrect for streaming purposes, potentially triggering excessive subscription requests due to an issue with your OpenSearch Ingestion pipeline. What drives Kinesis client adoption and retention strategies will be examined to identify opportunities for improvement.
  • The improved fan-out client fails to keep pace with the demand in the data flow. Ensure adequate OpenSearch Control Unit (OCU) allocation in the ingestion pipeline by verifying that the number of OCUs matches or exceeds the number of Kinesis information stream shards.
  • Fails to sustain pace within data flow, indicating a lagging polling client. Examine the OpenSearch Ingestion pipeline OCU configuration carefully, verifying that sufficient OCUs exist to support the anticipated Kinesis information stream shards; concurrently, scrutinize the polling mechanism employed by the buyer to guarantee seamless data ingestion.
• The key performance indicators to track are:
  • CloudWatch subscription filters occasionally struggle to convey data to Kinesis information streams, indicating a potential issue. Examine information stream metrics.
  • A warning sign signifies an inadequate capability within the Kinesis information stream indicating a need for further development or refinement to ensure seamless integration and optimal performance. Examine information stream metrics.
• Benefit from effective monitoring of OpenSearch Ingestion by consulting.
• To effectively monitor your OpenSearch Service, please refer to.

Clear up

Eliminate any unwanted AWS resources created during the process to prevent additional billing for these assets? Streamline your AWS account by following these key steps:

1. .
2. .
3. Remove the account-level CloudWatch subscription filter from your AWS setup.
4. Remove your existing log group-level CloudWatch subscription filter to prevent unwanted log deliveries and ensure a cleaner monitoring experience.
   1. Select the desired log group from the CloudWatch console interface.
   2. On the menu, selectand.
5. .

Conclusion

In this publication, you learned how to construct a serverless ingestion pipeline that streams CloudWatch logs in real-time to an OpenSearch domain or serverless collection using OpenSearch Ingestion. By leveraging this approach, you can effectively handle a broad spectrum of real-time data ingestion scenarios and seamlessly integrate it with existing workflows that rely on Kinesis Data Streams for real-time analytics.

When selecting between OpenSearch Ingestion and Kinesis Knowledge Streams for varying usage scenarios, consider that

To optimize log analysis capabilities in OpenSearch, consider leveraging pre-configured dashboards readily available.

In regards to the authors

Throughout their distinguished career, the individual has consistently excelled in distributed programs engineering, holding various roles including Software Program Engineer, Architect, and Knowledge Engineer. M has successfully developed and deployed solutions capable of handling massive amounts of streaming data at extremely low latencies, powered complex enterprise machine learning pipelines, and designed intuitive programs for effortless information sharing across teams using diverse data tools and software ecosystems. At AWS, they’re a Sr. Professional Options Trader supporting prominent US Federal monetary clients with tailored strategies and in-depth market analysis for informed investment decisions.

As a Product Supervisor with Amazon OpenSearch Service. He specializes in scalable solutions for ingesting diverse data sources into Amazon OpenSearch Service, enabling seamless information integration at large scales. Arjun contemplates large-scale distributed programs and cloud-based innovations from his base in Seattle, Washington.

As a seasoned Search Specialist at Amazon OpenSearch Service, I leverage my expertise to optimize search queries and deliver precise results for our customers. He develops comprehensive search functionalities with a wide range of customizable options. Founded in Austin, Texas, Muthu specializes in networking and safety expertise.