Amazon OpenSearch Ingestion is a robust information ingestion pipeline that AWS prospects use for a lot of totally different functions, corresponding to observability, analytics, and zero-ETL search. Many purchasers at present push logs, traces, and metrics from their functions to OpenSearch Ingestion to retailer and analyze this information.
In the present day, we’re completely satisfied to announce that OpenSearch Ingestion pipelines now help cross-account ingestion for push-based sources corresponding to HTTP and OpenTelemetry (OTel). Organizations can now use this function to effortlessly share information throughout groups. For instance, many organizations have central observability groups—now these groups can create OpenSearch Ingestion pipelines and share them with different groups of their group. You too can use this function to ingest information into Amazon OpenSearch Service domains or Amazon OpenSearch Serverless collections in different accounts.
Beforehand, sharing OpenSearch Ingestion pipelines throughout accounts required groups to make use of digital personal cloud (VPC) options to share entry. For instance, groups might use VPC peering, which isn’t all the time possible, or AWS Transit Gateway. The brand new cross-account ingestion options in OpenSearch Ingestion can simplify your deployment and cut back value for sharing pipelines.
Answer overview
Let’s have a look at how one can share a pipeline from a central logging account with two different improvement accounts (A and B). The central logging account can create an OpenSearch Ingestion pipeline utilizing a push-based supply, for instance, HTTP. After creating the pipeline, a member of the central logging workforce can grant entry to the opposite groups. They will use a useful resource coverage that provides permissions to the 2 different workforce accounts to create pipeline endpoints. After making this variation, the OpenSearch Ingestion pipeline is obtainable to be used by the opposite groups.
The next diagram illustrates this configuration.
Within the following sections, we reveal how one can implement this resolution.
Conditions
First, the central logging account will need to have a VPC with two choices enabled.
- enableDnsSupport have to be set to true
- enableDnsHostnames have to be set to true
The central logging account should additionally create a push-based OpenSearch Ingestion pipeline within the VPC. This is usually a pipeline receiving logs from FluentBit or OpenTelemetry telemetry.
The event accounts which can be going to hook up with the pipeline additionally will need to have VPCs in the identical area with the identical DNS choices enabled.
- enableDnsSupport have to be set to true
- enableDnsHostnames have to be set to true
Create useful resource coverage
Because the proprietor of the pipeline, you may create a useful resource coverage that permits the 2 improvement accounts to create pipeline endpoints in opposition to your pipeline.
The next is an instance useful resource coverage for this state of affairs:
The OpenSearch Ingestion console makes it easy to create these insurance policies, as proven within the following screenshot.
Create pipeline endpoint
Now that the central logging account has shared permissions on their pipeline, the event accounts can create pipeline endpoints. A pipeline endpoint is a connection from one VPC to an OpenSearch Ingestion pipeline.
The event accounts are accountable for creating the pipeline endpoints within the VPCs they wish to join from. They create this within the subnets they want and supply a safety group. The safety group ought to have an inbound rule permitting entry port HTTPS over port 443 from any supply that the event accounts have to ingest logs.
Growth workforce A can create a pipeline endpoint utilizing a command much like the next:
Growth workforce A can even use the OpenSearch Ingestion console to create the pipeline endpoint.
After performing this variation, the VPC for improvement workforce A can have a pipeline endpoint. This pipeline endpoint now permits for ingesting information into the central logging pipeline. Now, Amazon Elastic Compute Cloud (Amazon EC2) situations, Amazon Elastic Container Service (Amazon ECS) duties, Kubernetes pods, and different compute operating within the VPC can ingest their log information into the pipeline utilizing instruments corresponding to FluentBit.
On the identical time or at a later time, improvement workforce B can create a pipeline endpoint as properly. This workforce will create it for their very own VPC.
After this, the pipeline will now have two pipeline endpoints, so each groups can ingest their log information into the central logging VPC.
Clear up
After a pipeline endpoint is created, both account can take away it. The event groups in our state of affairs can use the DeletePipelineEndpoint API to delete it from their accounts. Moreover, if the central logging account must take away a pipeline endpoint from a pipeline, it might probably use the RevokePipelineEndpointConnections API. Each choices can be found on the OpenSearch Ingestion console as properly.
After the pipeline endpoints are eliminated, the central logging workforce can even take away the pipeline in the event that they now not want it.
Conclusion
The brand new pipeline endpoint function for OpenSearch Ingestion simplifies how one can share pipelines for cross-account ingestion. This will help groups use the highly effective options of OpenSearch Ingestion and open up new potentialities for groups or organizations utilizing a number of accounts and VPCs. The brand new pipeline endpoint function is obtainable at present in AWS Areas the place OpenSearch Ingestion is obtainable.
To get began with cross-account ingestion in OpenSearch Ingestion, consult with OpenSearch Ingestion documentation or strive creating your first cross-account pipeline on the OpenSearch Ingestion console.
In regards to the authors
