This can be a visitor submit by Leeneksh Dubey, Cloud Engineer at Trellix, in partnership with AWS.
Trellix, a world chief in cybersecurity options, emerged in 2022 from the merger of McAfee Enterprise and FireEye. Serving over 40,000 enterprise clients worldwide, Trellix delivers the trade’s most complete, open, and native AI-powered safety platform. Their resolution helps organizations construct operational resilience in opposition to superior threats by way of automated detection, investigation, and response capabilities.
In the present day safety groups face an more and more advanced panorama of cybersecurity threats, whereas the amount of safety and utility logs grows exponentially. With restricted assets and personnel, groups wrestle to analyze all safety occasions, doubtlessly lacking rising threats. Trellix addresses these challenges by unifying safety instruments throughout endpoints, networks, cloud, and electronic mail right into a single, AI-powered platform. By automating risk detection, investigation, and response, it permits safety groups to determine and neutralize threats quicker whereas lowering operational complexity.
To handle exponential log progress throughout their multi-tenant, multi-Area infrastructure, Trellix used Amazon OpenSearch Service, Amazon OpenSearch Ingestion, and Amazon Easy Storage Service (Amazon S3) to modernize their log infrastructure. Dealing with challenges with self-managed Elasticsearch clusters on Amazon Elastic Compute Cloud (Amazon EC2), Trellix’s migration to managed OpenSearch Service considerably optimized their operations. This strategic implementation enabled them to course of terabytes of day by day safety knowledge throughout a number of AWS Areas whereas reaching a 35% discount in storage prices as of Q3 2024. The shift to managed providers saved as much as 10 hours of infrastructure upkeep time weekly, serving to builders focus extra on value-added duties.
On this submit, we share how, by adopting these AWS options, Trellix enhanced their system’s efficiency, availability, and scalability whereas lowering operational overhead.
Resolution overview
Trellix’s modern log administration resolution, constructed on AWS providers, addresses the challenges of processing giant volumes of safety knowledge throughout a number of Areas. This enterprise-grade structure demonstrates how organizations can successfully handle safety logs at scale whereas optimizing prices. The answer addresses three essential enterprise challenges: environment friendly administration of long-term log storage, scalable distribution of analytics and alerting features, and optimization of storage prices throughout their multi-regional infrastructure. The structure is illustrated within the following diagram, demonstrating how Trellix managed the safety logs at scale whereas optimizing prices.
The Trellix safety log administration resolution on AWS implements a complete knowledge pipeline that seamlessly handles log ingestion, processing, storage, and evaluation. Within the following sections, we discover the six steps of the workflow in additional element.
Step 1: Load knowledge to Amazon S3
The answer begins with a knowledge ingestion course of utilizing the Amazon S3 globally distributed and extremely scalable infrastructure. Uncooked safety and utility logs are captured from a number of Regional deployments, serving to Trellix preserve each knowledge sovereignty and low latency entry throughout numerous jurisdictions. These logs are then processed by the Trellix inside engine, which enriches them utilizing proprietary safety logic. This enriched dataset is subsequently saved again in Amazon S3, establishing a safe, scalable basis for additional safety analytics and downstream processing.
Step 2: Amazon SNS notification triggered by S3 Occasions
After the enriched knowledge is efficiently saved in Amazon S3, the system initiates an event-driven automation sequence. Amazon S3 is configured to emit occasion notifications to an Amazon Easy Notification Service (Amazon SNS) matter each time new knowledge is uploaded. Amazon SNS acts as a notification hub, effectively broadcasting these occasions to subscribed providers or endpoints. This method helps the structure stay responsive and decoupled, as a result of it permits numerous shoppers to be alerted in actual time as new knowledge turns into obtainable within the system.
Step 3: Message queuing in Amazon SQS
As the following step within the workflow, the SNS notifications are routed to Amazon Easy Queue Service (Amazon SQS), which serves as a sturdy and scalable queuing layer between producers and shoppers. This queue acts as a buffer, facilitating dependable and asynchronous supply of occasion metadata to downstream processing parts. The usage of Amazon SQS gives message persistence and fault tolerance, significantly underneath high-throughput or failure situations, permitting OpenSearch Ingestion to course of incoming knowledge in a managed and resilient method.
Step 4: Automated knowledge processing with OpenSearch Ingestion
OpenSearch Ingestion constantly polls the SQS queue for brand spanking new messages indicating the provision of information in Amazon S3. Upon receiving these messages, it makes use of its built-in integration capabilities to fetch the corresponding knowledge straight from Amazon S3. After the info is retrieved, the ingestion pipeline performs the required transformations earlier than forwarding it to the OpenSearch Service area. To facilitate optimum cost-efficiency and efficiency, Trellix chosen OR1 situations sorts for his or her OpenSearch deployment. These situations supply a excessive memory-to-vCPU ratio and are particularly optimized for intensive indexing and search workloads, making them supreme for dealing with large-scale log analytics operations.
Step 5: Log lifecycle setup utilizing Index State Administration
To optimize storage utilization and handle knowledge retention, Trellix has carried out Index State Administration (ISM) insurance policies throughout the OpenSearch Service. These insurance policies automate the lifecycle of ingested log knowledge by transitioning it by way of outlined levels primarily based on age and entry patterns. Initially, logs reside within the sizzling tier for as much as 24 hours, enabling fast entry for real-time safety evaluation. As logs age past this threshold, they’re mechanically transitioned to the UltraWarm storage, which provides a more cost effective storage possibility whereas holding the info queryable. Lastly, after the predefined retention interval expires, the ISM coverage deletes the info from the system. This absolutely automated lifecycle administration method balances efficiency, compliance, and cost-efficiency.
Step 6: Complete monitoring and visualization
Utilizing the intensive monitoring capabilities of Amazon CloudWatch, complemented by Trellix’s in-house automations utilizing OpenSearch public APIs for customized monitoring, the answer gives end-to-end visibility by way of built-in visualization instruments. OpenSearch Dashboards gives safety groups with highly effective log evaluation and search capabilities, to allow them to dive deep into safety occasions and determine potential threats. Moreover, the answer makes use of Amazon Managed Grafana to create personalized dashboards that monitor each the info pipeline well being and OpenSearch cluster efficiency.
This twin visualization method delivers a number of advantages: real-time safety occasion monitoring and evaluation, complete efficiency metrics throughout the infrastructure, automated alerting for fast risk response, customized dashboard views for various safety operations wants, and unified visibility throughout the a number of Regional deployments. The mixed energy of those instruments creates a sturdy monitoring framework that helps Trellix preserve a powerful safety posture whereas facilitating optimum efficiency throughout their international infrastructure.This six-step implementation demonstrates how AWS providers could be mixed to create a scalable, cost-efficient safety log administration resolution that processes terabytes of day by day safety knowledge whereas sustaining excessive efficiency and operational effectivity.
Key advantages
Trellix’s implementation of OpenSearch Service as their logging resolution delivered three vital benefits that remodeled their safety operations.
Simplified log administration structure
Trellix streamlined their safety operations by implementing a cohesive log administration structure that avoids the complexity of managing a number of disparate instruments. Through the use of OpenSearch Ingestion, a totally managed serverless knowledge pipeline, Trellix simplified their knowledge pipeline for processing real-time safety knowledge. The mixing with Managed Grafana gives a unified visualization layer, enabling safety groups to concentrate on risk detection reasonably than infrastructure administration.
Scalability and resilience
The implementation of OpenSearch Service permits Trellix to realize unprecedented scalability and resilience of their safety operations. Trellix’s structure makes use of an OpenSearch Ingestion pipeline to supply easy dealing with of sudden log quantity spikes throughout a number of Regional deployments. OpenSearch Ingestion permits dynamic scaling with automated useful resource optimization, facilitating seamless capability administration as knowledge volumes develop. This functionality helps Trellix preserve constant efficiency even in periods of elevated safety occasion logging. The answer additionally implements a sturdy Multi-AZ deployment technique to take care of most resilience and steady service availability. Throughout self-healing testing, the structure demonstrated spectacular restoration instances underneath 9 minutes when a node was rebooted, showcasing its potential to take care of enterprise continuity even in case of node failure. The automated failover capabilities facilitate minimal disruption to safety operations, so Trellix can preserve fixed vigilance over their clients’ safety posture. Lastly, the answer makes use of automated Amazon S3 backups mixed with hourly snapshots for complete point-in-time restoration capabilities. Every Area maintains further buyer knowledge replicas, making a multi-layered knowledge safety technique that maintains the integrity and availability of essential safety info.
Easy scalability with optimized value
Trellix’s exponential progress in safety knowledge processing demanded an answer that might scale dynamically whereas sustaining cost-efficiency. The strategic implementation of Amazon S3 and OpenSearch Service with UltraWarm storage supplied the muse for this scalable structure. UltraWarm, a totally managed heat storage tier for OpenSearch Service, revolutionized how Trellix manages their intensive safety knowledge throughout a number of Areas. The answer makes use of UltraWarm’s modern structure, which makes use of Amazon S3 for sturdy storage whereas sustaining quick question efficiency for safety evaluation. A key benefit of UltraWarm’s Amazon S3 backed structure is the elimination of index replicas, considerably lowering cluster measurement and related prices whereas sustaining knowledge sturdiness.The clever log prioritization framework kinds the spine of Trellix’s knowledge administration technique, categorizing incoming knowledge primarily based on safety significance. This systematic method permits environment friendly routing of P2 and P3 log sources, optimized processing paths for various safety priorities, decreased load on major SIEM infrastructure, and customised dealing with primarily based on buyer necessities. The implementation has confirmed significantly invaluable for safety log analytics, the place historic knowledge evaluation is essential for risk detection and compliance necessities.The implementation delivered substantial operational and monetary advantages for Trellix. By combining priority-based routing and tiered storage administration, the group achieved a 35% discount in storage and compute prices whereas sustaining high-performance safety operations. The answer permits environment friendly storage and evaluation of in depth historic knowledge, supporting Trellix’s dedication to complete safety monitoring whereas optimizing operational prices. This implementation demonstrates how AWS providers might help organizations optimize prices with out compromising safety capabilities or operational effectivity.
What’s subsequent
The profitable implementation of this resolution has positioned Trellix to discover further AWS capabilities and rising applied sciences to reinforce their safety operations:
- Integration of AWS ML/AI providers to investigate petabytes of safety log knowledge
- Implementation of ML-based anomaly detection inside OpenSearch Service
- Utilizing safety analytics plugins for superior risk detection
- Customized configurations and pre-built safety guidelines implementation
Abstract
Trellix efficiently modernized its log administration infrastructure by way of collaboration with AWS, implementing a complicated structure that addresses the challenges of processing terabytes of day by day safety knowledge throughout a number of Areas. Through the use of OpenSearch Service with UltraWarm nodes and integrating Amazon S3, the answer delivered vital efficiency enhancements, together with quicker log ingestion and streamlined operational administration. The structure’s modern tiered storage method, mixed with optimized retention insurance policies, resulted in a 35% discount in storage prices whereas sustaining compliance necessities.This transformation has positioned Trellix to effectively deal with rising knowledge volumes and evolving safety challenges, demonstrating how strategic use of cloud providers can concurrently enhance efficiency, cut back prices, and improve operational effectivity.
Concerning the authors
