We’re excited to announce AWS Glue Knowledge Catalog utilization metrics. The utilization metrics is a brand new characteristic that gives native integration with Amazon CloudWatch. This characteristic gives you with rapid visibility into your AWS Glue Knowledge Catalog API utilization patterns and traits.
AWS Glue Knowledge Catalog is a centralized repository that shops metadata about your group’s datasets. With its unified interface that acts as an index, you’ll be able to retailer and question details about your knowledge sources, together with their location, codecs, schemas, and runtime metrics.
As you scale your lakehouse structure on Amazon Net Providers (AWS) and keep dependable knowledge operations, observability and monitoring turns into crucial to understanding and optimizing Knowledge Catalog API usages.
With Knowledge Catalog utilization metrics in CloudWatch, you’ll be able to obtain the next:
- Monitor API name patterns at 1-minute intervals
- Proactively request service quota enhance for API fee limits
- Allow the CloudWatch pre-built anomaly detection characteristic to establish abnormalities in your API utilization
- Perceive lakehouse utilization throughout greater than 50 APIs
On this put up, we display how one can entry these metrics, present a step-by-step walkthrough, and arrange significant alarms.
Entry Knowledge Catalog utilization metrics in Amazon CloudWatch console
To entry Knowledge Catalog utilization metrics, full the next steps:
- Open Amazon CloudWatch console
- Underneath Metrics, select All metrics
- Within the search bar, enter
Glueand select Enter - Select Utilization > By AWS Useful resource, as proven within the following screenshot
- The Metrics part opens and shows completely different catalog utilization metrics you can choose from to create dashboards and alarms, as proven within the following screenshot
Monitor CallCount metrics
Every Amazon CloudWatch metric for Knowledge Catalog is of a sort API and set as CallCount. Because of this for every API name on that particular useful resource (for instance, GetConnection API) shall be logged as one rely. These metrics can seamlessly combine into your present CloudWatch dashboards, or you need to use them to create new ones. For proactive monitoring, you’ll be able to configure customized alarms that set off mechanically when this API utilization exceeds your outlined thresholds, serving to you adjust to service limits.
Underneath the Graphed metrics tab, you’ll be able to present further customizations to match your monitoring wants. Within the Particulars column, you’ll be able to create alarms and allow anomaly detection to establish uncommon patterns.
To assist with efficient API monitoring, CallCount metrics particularly deal with profitable API calls. This manner, you might have extra exact monitoring and may troubleshoot several types of API behaviors. The next screenshot reveals the AWS Glue utilization metrics view for GetTables API.
Within the Statistics column, you’ll be able to view your API utilization past the default Sum, Min, and Max metrics. Now you can choose all kinds of statistical strategies to research your utilization patterns, as proven within the following screenshot.
Metrics and dimensions for Knowledge Catalog utilization metrics
Knowledge Catalog utilization metrics use the AWS/Utilization namespace and supply CallCount metrics. These metrics are printed with the size Service, Useful resource, Kind and Class.
The CallCount metric doesn’t have a specified unit. Probably the most helpful statistic for the metric is SUM, which represents the entire operation rely for the 1-minute interval. An essential word is that the metric worth is emitted at 1-minute intervals. Decreasing the interval additional (for instance, to 1 second) gained’t change the emittance interval.
Metrics
| Metric | Description |
CallCount | The variety of specified operations carried out in your account. |
Dimensions
| Dimension key | Dimension worth | Description |
| Service | AWS Glue | The identify of the AWS service containing the useful resource. For Knowledge Catalog utilization metrics, the worth for this dimension is AWS Glue. |
| Kind | API | The kind of useful resource being tracked. At the moment, when the Service dimension is AWS Glue, the one legitimate worth for Kind is API. |
| Useful resource | The identify of the API operation. Legitimate values embody the next: GetCatalogs, GetCatalog, GetDatabases, GetDatabase, GetTables, GetTable, GetTableVersion, GetTableVersions, SearchTables, GetPartitionIndexes, GetColumnStatisticsForTable, GetPartition, GetPartitions, BatchGetPartition, GetColumnStatisticsForPartition, GetConnection, GetConnections, GetUserDefinedFunction, GetUserDefinedFunctions, GetCatalogImportStatus, GetTableOptimizer, BatchGetTableOptimizer, ListTableOptimizerRuns, CreateCatalog, CreateDatabase, CreateTable, CreatePartitionIndex, CreatePartition, BatchCreatePartition, CreateConnection, CreateUserDefinedFunction, CreateTableOptimizer, UpdateCatalog, UpdateDatabase, UpdateTable, UpdateColumnStatisticsForTable, UpdatePartition, BatchUpdatePartition, UpdateColumnStatisticsForPartition, UpdateConnection, UpdateUserDefinedFunction, UpdateTableOptimizer, DeleteCatalog, DeleteDatabase, DeleteTable, BatchDeleteTable, DeleteTableVersion, DeletePartitionIndex, DeleteColumnStatisticsForTable, DeletePartition, BatchDeletePartition, DeleteColumnStatisticsForPartition, DeleteConnection, BatchDeleteConnection, DeleteUserDefinedFunction, DeleteTableOptimizer, TestConnection, ImportCatalogToGlue | |
| Class | None | The category of useful resource being tracked. Knowledge Catalog utilization metrics use this dimension with a price of None. |
Arrange CloudWatch alarms for Knowledge Catalog utilization metrics
Knowledge Catalog has outlined guidelines to handle atypical utilization patterns that restrict the shopper name fee on the granularity of requests per second. You possibly can generate CloudWatch alarms utilizing the CallCount metric in order that restrict will increase might be completed proactively. To configure a CloudWatch alarm with this threshold, full the next steps:
- On the CloudWatch metrics console, choose one of many out there metrics, as proven within the following screenshot. On this instance, we choose the useful resource
GetTables. You possibly can choose a number of metrics to suit your use case.
- Select Graphed metrics.
- Select Sum as the first statistic.
- Set interval to 1 minute.
- Select Particulars and Create Alarm.
- For Threshold kind, select Anomaly Detection. You may as well choose Static based mostly in your necessities and after you’ve decided a particular threshold worth.
- Set the Anomaly detection threshold to 2 (default). The brink worth is used to find out the conventional vary of values for the metric. A better worth produces a thicker band of regular values. For extra info on how CloudWatch anomaly detection works, seek advice from How CloudWatch anomaly detection works.
- Select Subsequent.
- For Ship a notification to the next SNS matter, select Create new matter.
- For Create a brand new matter, enter your Amazon Easy Notification Service (Amazon SNS) matter identify.
- For E-mail endpoints that can obtain the notification, enter your e mail handle. On this instance, we’re going to create a brand new SNS matter. Nonetheless, you need to use your present SNS subjects or use different choices similar to AWS Lambda or auto scaling motion.
- Select Create matter.
- Scroll down and select Subsequent.
- Enter an alarm identify and an outline and select Subsequent.
- Overview all the main points you’ve entered and select Create alarm, as proven within the following screenshot.
By following these steps, you’ve efficiently configured a CloudWatch alarm utilizing anomaly detection that screens your Knowledge Catalog utilization with the edge that you just set. The alarm will set off when the CallCount metric exceeds the calculated threshold, sending notifications to your specified SNS matter and e mail endpoints.
This proactive monitoring strategy prevents API fee restrict points and gives a clean operation of your Knowledge Catalog utilization. For extra info on utilizing CloudWatch alarms, seek advice from Utilizing Amazon CloudWatch alarms.
Conclusion
AWS Glue Knowledge Catalog utilization metrics is an efficient enhancement to your knowledge infrastructure monitoring capabilities. It addresses the rising want for detailed observability via Amazon CloudWatch in trendy knowledge architectures constructed on prime of Knowledge Catalog. You now have entry to extra granular statistics, shifting past easy most and common request metrics to complete efficiency indicators together with p99 percentiles. These metrics are emitted in 1-minute intervals, offering visibility into your knowledge catalog operations. Organizations can now proactively establish bottlenecks earlier than they have an effect on operations and effectively conduct capability planning via detailed utilization patterns.
From constructing monitoring dashboards to establishing alerts, the native assist with CloudWatch anomaly detection and versatile alarm configurations makes it simple to proactively monitor your lakehouse deployment and forestall abnormalities in your lakehouse utilization. For extra info, seek advice from Monitoring Knowledge Catalog utilization metrics in Amazon CloudWatch within the AWS Glue documentation. We advocate testing and utilizing these metrics as a part of your trendy monitoring and observability technique. We encourage you to share your suggestions with us.
Particular because of everybody who contributed to this launch: Vineet Sunkavalli, Shubham Bansal, Mike Kloss, Zarius Dubash.
In regards to the authors
David Zhang is an Analytics Options Architect specializing in designing and implementing large-scale knowledge infrastructure, ETL processes, and in depth knowledge administration techniques. He helps prospects modernize knowledge platforms on Amazon Net Providers (AWS). David can be an lively speaker at AWS occasions and contributor to technical content material and open supply initiatives. He enjoys enjoying volleyball, tennis, and basketball throughout his free time.
Noritaka Sekiyama is a Principal Massive Knowledge Architect with Amazon Net Providers (AWS) Analytics companies. He’s chargeable for constructing software program artifacts to assist prospects. In his spare time, he enjoys biking on his highway bike.
Sandeep Adwankar is a Senior Product Supervisor at AWS. Based mostly within the California Bay Space, he works with prospects across the globe to translate enterprise and technical necessities into merchandise that allow prospects to enhance how they handle, safe, and entry knowledge.
Abhay Joshi is a Software program Growth Engineer at AWS Glue and AWS Lake Formation. He’s obsessed with constructing fault tolerant and dependable distributed techniques at scale.
