Utilize infrastructure-as-code tools and AWS CDK to streamline Amazon OpenSearch Service cluster administration tasks.

Accessing accurate and timely data is crucial for informed business decision-making. Firms are increasingly relying on AI-powered solutions to energize their search and analytics capabilities. The OpenSearch Service simplifies deployment, operation, and scaling of search capabilities within the cloud, empowering scenarios such as log analysis, software monitoring, and website search functionality.

Effective management of OpenSearch Service indexes and cluster resources yields significant enhancements to efficiency, scalability, and reliability, thereby positively impacting a company’s bottom line directly. Despite having a solid foundation, the business currently lacks intuitive and thoroughly documented features that enable seamless automation of its core operational tasks.

Implementing steady integration and deployment processes (CI/CD) effectively manages OpenSearch index assets. By storing index configurations in a centralized supply repository, you enable enhanced monitoring, streamlined collaboration, and seamless rollbacks. By leveraging infrastructure as code (IaC) tools, organisations can streamline the deployment of resources, ensuring uniformity and minimising manual effort. By leveraging a CI/CD pipeline, organizations can seamlessly automate deployments, significantly streamlining their development workflows while ensuring the reliability and efficiency of software releases.

We explore two alternatives to achieve this: the Python package and the NumPy library. The choice that best suits your needs hinges on the tools at your disposal, your preferred programming language, and your current workflow.

Answer overview

Let’s stroll through a simple implementation. Using AWS Cloud Development Kit (CDK), we provision the required infrastructure in accordance with the predefined structure diagram, which enables us to execute Evolution scripts and leverage Terraform data files seamlessly. The solution to your complete resolution can be found on GitHub.

Conditions

To comply with the requirements of this publication, you must have the necessary.

• Familiarity with and
• Proficiency in harnessing the power of AWS CDK, Terraform, and the command line enables you to effectively orchestrate cloud-based infrastructure.
• The upcoming software program updates installed on your machine include Python 3.12, Node.js 20, and AWS CDK version 2.17.0 or later.
• An instance, with an IAM position configured with relevant permissions.

Construct the answer

To streamline OpenSearch Service cluster management, follow this straightforward procedure:

1. aws cloudformation deploy –stack-name MyStack –template-body file://cloudformation-template.yaml –capabilities CAPABILITY_IAM –parameter-overrides lambdaFunctionName=MyLambdaLayerLambda,searchIndexName=MySearchIndex,buildEnvironment=NODE_14,zipFilePattern=*.zip –region us-west-2

git clone https://github.com/aws-samples/opensearch-automated-cluster-management
cd opensearch-automated-cluster-management
cd app/openSearchMigration
mvn bundle
cd ../../lambda_layer
chmox a+x create_layer.sh
./create_layer.sh
cd ../infra
npm set up
npx cdk bootstrap
aws iam create-service-linked-role --aws-service-name es.amazonaws.com
npx cdk deploy --require-approval by no means

2. After a 15-20 minute deployment of the infrastructure, verify the OpenSearch area’s availability and functionality by checking the corresponding screenshots, which should demonstrate the successful creation of the Lambda function and CodeBuild project.

Earlier than using automated tools to generate index templates, you should first verify whether any exist already by utilising the OpenSearchQuery Lambda perform.

3. On the AWS Lambda console, navigate to the relevant function.
4. Choose from the menu.

Terraform informs that no index patterns have been created, stating: No index patterns created by Terraform or Evolution.

Apply Terraform recordsdata

You leverage Terraform within a CodeBuild project. Here are a couple of vital items of configuration to consider:

1. What ambient conditions prevail around me?

variable "OpenSearchDomainEndpoint" {
  type        = string
  description = "The URL of the OpenSearch domain"
}

variable "IAMRoleARN" {
  type        = string
  description = "The Amazon Resource Name (ARN) of the IAM role to use with OpenSearch"

2. Outline and configure the supplier

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.3.1"
    }
  }
}

provider "opensearch" {
  endpoint = var.OpenSearchDomainEndpoint
  assume_role_arn = var.IAMRoleARN
}

As of the publication date of this publication, a known issue exists in the Terraform OpenSearch provider that can trigger upon launching your CodeBuild project, potentially preventing successful execution? Until it’s mounted, please use the next model.

terraform {
  required_providers {
    opensearch = {
      source  = "gnuletik/opensearch"
      version = "2.7.0"
    }
  }
}

3. Create an index template

useful resource "opensearch_index_template" "template_1" {
  identify = "cicd_template_terraform"
  physique = <<EOF
{
  "index_patterns": ["terraform_index_*"],
  "template": {
    "settings": {
      "number_of_shards": "1"
    },
    "mappings": {
        "_source": {
            "enabled": false
        },
        "properties": {
            "host_name": {
                "kind": "key phrase"
            },
            "created_at": {
                "kind": "date",
                "format": "EEE MMM dd HH:mm:ss Z YYYY"
            }
        }
    }
  }
}
EOF
}

Are you truly ready for this?

4. On the CodeBuild console, navigate to the relevant challenge and select it.

The construction should operate efficiently; optimal performance is ensured by reviewing subsequent lines in the logs.

The OpenSearch index template 'template_1' has been successfully created, and the creation process took zero seconds. The ID of this template is 'cicd_template_terraform'. The application process has also been completed without any issues. Records updated? 1 source added, 0 modified, and 0 destroyed.

You will be able to verify that the index template has been properly generated by using the same Lambda function, which will yield the following results.

Run Evolution scripts

Subsequently, the Evolution library is leveraged within the next step.

Let’s examine the code in more detail, focusing on a few key elements of the coding structure and configuration settings.

1. To initiate development, first ensure you have included the latest Evolution core library and Amazon Web Services (AWS) Software Development Kit (SDK) in your project’s Maven dependencies. Here is the rewritten text:

<dependency>
    <groupId>com.senacor.elasticsearch.evolution</groupId>
    <artifactId>elasticsearch-evolution-core</artifactId>
    <model>0.6.0</model><!--check the most recent version-->
</dependency>
<dependency>
   <groupId>software program.amazon.awssdk</groupId>
   <artifactId>auth</artifactId>
</dependency>

2. Creating EvolutionBean and an AWS Interceptor which implements SignatureVersionVerifier: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/awssdk/core.interceptors/SignatureVersionVerifierInterceptor.html HttpRequestInterceptor).

Interceptors provide an open-ended mechanism whereby your written code is executed by the SDK at key points in the request-response lifecycle, enabling the injection of custom logic or behavior. The performance of the AWS interceptor involves intercepting the execution of API requests and generating AWS signed requests authenticated with the appropriate IAM roles? You must utilize the OpenSearch client to send all requests to OpenSearch instances within AWS.

3. What’s needed is a robust and configurable OpenSearch consumer that efficiently manages the lifecycle of indices, mappings, templates, and aliases. This should be achieved through a comprehensive API that seamlessly integrates with existing infrastructure.

   “`java
   public class OpenSearchConsumer {
   private final static Logger logger = LoggerFactory.getLogger(OpenSearchConsumer.class);

   public void createIndex(String indexName) {
   try (var client = OpenSearchClient.builder()
   .setEndpoint(“localhost:9200”)
   .build()) {

   var createIndexRequest = new CreateIndexRequest(indexName);
   var response = client.indices().create(createIndexRequest, RequestOptions.DEFAULT);

   if (!response.isAcknowledged()) {
   logger.warn(“Failed to create index ‘{}'”, indexName);
   } else {
   logger.info(“Successfully created index ‘{}'”, indexName);
   }
   } catch (Exception e) {
   logger.error(“Error creating index ‘{}’: {}”, indexName, e.getMessage());
   }
   }

   public void createMapping(String indexName, String mapping) {
   try (var client = OpenSearchClient.builder()
   .setEndpoint(“localhost:9200”)
   .build()) {

   var createIndexRequest = new CreateIndexRequest(indexName);
   var response = client.indices().putMapping(createIndexRequest, RequestOptions.DEFAULT, mapping);

   if (!response.isAcknowledged()) {
   logger.warn(“Failed to create mapping for index ‘{}'”, indexName);
   } else {
   logger.info(“Successfully created mapping for index ‘{}'”, indexName);
   }
   } catch (Exception e) {
   logger.error(“Error creating mapping for index ‘{}’: {}”, indexName, e.getMessage());
   }
   }

   public void createTemplate(String templateName, String template) {
   try (var client = OpenSearchClient.builder()
   .setEndpoint(“localhost:9200”)
   .build()) {

   var createIndexRequest = new CreateIndexRequest(templateName);
   var response = client.indices().putTemplate(createIndexRequest, RequestOptions.DEFAULT, template);

   if (!response.isAcknowledged()) {
   logger.warn(“Failed to create template ‘{}'”, templateName);
   } else {
   logger.info(“Successfully created template ‘{}'”, templateName);
   }
   } catch (Exception e) {
   logger.error(“Error creating template ‘{}’: {}”, templateName, e.getMessage());
   }
   }

   public void createAlias(String indexName, String aliasName) {
   try (var client = OpenSearchClient.builder()
   .setEndpoint(“localhost:9200”)
   .build()) {

   var createIndexRequest = new CreateIndexRequest(indexName);
   var response = client.indices().putAlias(createIndexRequest, RequestOptions.DEFAULT, aliasName);

   if (!response.isAcknowledged()) {
   logger.warn(“Failed to create alias ‘{}’ for index ‘{}'”, aliasName, indexName);
   } else {
   logger.info(“Successfully created alias ‘{}’ for index ‘{}'”, aliasName, indexName);
   }
   } catch (Exception e) {
   logger.error(“Error creating alias ‘{}’ for index ‘{}’: {}”, aliasName, indexName, e.getMessage());
   }
   }
   }
   “`

The default Elasticsearch consumer that comes bundled with the Maven dependency cannot be utilized to process requests without further configuration. PUT calls to the OpenSearch cluster. Subsequently, you must bypass the default REST consumer occurrence, and add an explicit handler to configure the necessary settings. CallBack to the AwsRequestSigningInterceptor.

What type of sequence are you trying to implement?

non-public RestClient getOpenSearchEvolutionRestClient() {
    return RestClient.builder(getHttpHost())
        .setHttpClientConfigCallback(hacb -> 
            hacb.addInterceptorLast(getAwsRequestSigningInterceptor()))
        .construct();
}

4. Utilizing a meticulous approach, the Evolution Bean methodology initiates the seamless migration of scripts, seamlessly integrating new features and functionalities. classpath or filepath:

public void executeOpensearchScripts() {
    ElasticsearchEvolution opensearchEvolution = configureElasticsearchEvolution()
        .setEnabled(true)
        .setScriptLocations(Arrays.asList("classpath:opensearch_migration/base", "classpath:opensearch_migration/dev"))
        .setHistoryIndex("opensearch_changelog")
        .setValidateOnMigrate(false)
        .setOutOfOrder(true)
        .addPlaceholder("env", "dev")
        .load(getElasticsearchEvolutionRestClient());
    opensearchEvolution.migrate();
}

5. An Evolution Migration script represents a RESTful interface to the OpenSearch API, for instance. PUT /_index_template/cicd_template_evolutionThe configuration file, where you define index patterns, settings, and mappings in a structured JSON format. Evolutionary algorithms interpret and manage script versions, providing structured execution pathways. See the next instance:

{
  "index_patterns": ["evolution_index_*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    "mappings": {
      "_source": {"enabled": false},
      "properties": {
        "host_name": {"type": "text", "fields": {"keyword": {"type": "keyword"}}},
        "created_at": {"type": "date", "format": "EEE MMM dd HH:mm:ss Z YYYY"}
      }
    }
  }
}

The primary two strains . Evolution Additionally, they help remark strains in their migration scripts? Each line beginning with # or // This comment will likely be ignored. The remark strains that are intended for OpenSearch are not being sent. As substitutes, they are filtered through Evolution.

The migration script filename convention should adhere to a standardised format.

• Begin with esMigrationPrefix which is by default V Based on the configuration options, or the worth that has been configured esMigrationPrefix
• Defined as a numerical value, which is composed of separate model components separated by intervals..)
• Adopted by the versionDescriptionSeparator: __ (the double underscore image)
• The adopted content by an outline, which may be any textual content your filesystem helps, seamlessly integrates with a hierarchical structure, streamlining navigation and facilitating easy access to relevant information.
• Finish with esMigrationSuffixes which is by default .http The search is flexible and adaptable, with capabilities for customization and sensitivity to letter cases.

The automation system has been activated successfully and is now ready to perform its first task. A previously created instance of a migration script is available to guide your progress, as introduced in an earlier section. The following is an index template named: cicd_template_evolution.

5. On the AWS Lambda console, navigate to your function.
6. On the tab, click on “.

After a few seconds, the performance should efficiently complete. You’ll be able to evaluate the log output within the relevant section, as demonstrated in the following screenshots.

The index template is now available for testing, ensuring its configuration aligns with the script as evident from this screenshot.

Clear up

To consolidate and manage the assets generated through this publication, follow these steps: infra folder):

Conclusion

This publication demonstrates how to automate OpenSearch index templates using continuous integration/continuous deployment (CI/CD) best practices and tools such as Terraform or the Evolution library.

To gain a deeper understanding of OpenSearch Service, consult the. To further explore the Evolution library, consult the . To delve deeper into the Terraform OpenSearch provider, consult its documentation.

We’re confident that this comprehensive guide and its corresponding code will facilitate a seamless onboarding experience. The idea is being explored? What specific concepts or initiatives are you hoping to explore? Can you provide more context about what you mean by “ideas”? Are there any specific areas of improvement or innovation that you’d like to focus on?

Concerning the Authors

Is a senior options architect with Amazon Web Services (AWS), based in Hong Kong. He collaborates with major financial institutions to craft and develop secure, scalable, and highly accessible solutions in the cloud. Outside of work, Camille delights in a variety of gaming experiences, encompassing both classic board games and the latest online titles.

Serves as a senior options architect with Amazon Web Services (AWS), primarily located in Bengaluru, India. His primary objective is assisting large-scale enterprise clients in streamlining their operations and adopting cloud-based strategies to achieve their organizational goals effectively. With a strong background in knowledge and analytics.