Amazon EMR on EC2 pricing will be attributed to your end-users at a rate of $0.005 per hour for instance hours and $0.0005 per hour for storage hours, respectively, based on the number of instances and storage used during each hour. Your users’ total cost will be calculated by multiplying these rates by the number of hours and storage used, then adding any additional costs such as data transfer fees.

On Amazon Elastic Compute Cloud (EC2), a managed service allows for seamless operation of massive data processing and analytics workloads on the AWS cloud platform. By streamlining the installation and management of popular open-source platforms such as Apache Spark and TensorFlow, you can focus on uncovering valuable insights from large data sets rather than the underlying infrastructure itself? Using Amazon EMR, you’ll leverage the power of these massive computing resources to process, analyze, and derive valuable business insights from vast amounts of data.

Value optimization is undoubtedly a cornerstone of successful businesses. The strategy prioritizes efficient budgeting by eschewing unnecessary expenditures, selecting the most relevant resource types, scrutinizing spending patterns over time, and dynamically scaling up or down to meet business needs without overextending resources. An optimized workload effectively leverages a comprehensive array of resources to deliver desired outcomes at the lowest possible cost, while aligning with intended goals.

The present exhibit displays the estimated price of the cluster. Are you looking for more in-depth information regarding our pricing? These views provide a general insight into your Amazon EMR costs. Notwithstanding this, you would need to allocate specific pricing to individual Spark job stages. To determine the effective utilization rate of Amazon EMR for the finance business unit, one might require knowledge of its actual usage patterns and costs. For chargeback functions, you might need to aggregate the cost of Spark components by functional area. After assigning prices to specific individuals for Spark jobs, this insight can assist in making informed decisions to optimise pricing strategies accordingly. By repurposing your goals, you can maximize results with limited resources. Alternatively, you might consider exploring various pricing models such as tiered pricing, value-based pricing, or freemium pricing.

We present a chargeback model that enables tracking and allocation of costs for Spark workloads running on Amazon EMR within EC2 clusters, facilitating more accurate financial planning and optimization. We detail a methodology for attributing Amazon Elastic MapReduce (EMR) costs to distinct business activities, such as job submissions, user groups, or workflow stages. To efficiently allocate prices across a range of business products and services. By leveraging this capability, you’ll gain a better understanding of the return on investment for your Spark-based projects, enabling informed decisions about future resource allocation.

Answer overview

The solution helps you track the cost of your Spark applications running on Amazon Elastic MapReduce (EMR) instances launched on Amazon Elastic Compute Cloud (EC2). By streamlining your EMR cluster pricing strategies, you can significantly improve cost-effectiveness.

A daily scheduled performance is utilized in accordance with this proposed resolution. The solution performs data capture for utilization and pricing metrics, which are then stored in Amazon RDS tables. The information stored in RDS tables is subsequently queried to determine chargeback metrics and develop reporting patterns leveraging. The use of these AWS services comes with additional costs associated with implementing this solution. You’ll have the flexibility to devise a strategy leveraging your existing EMR cluster, allowing you to avoid additional AWS services and associated costs when implementing your chargeback solution. The script stores related metrics in an Amazon S3 bucket and leverages Python Jupyter notebooks to generate chargeback numbers by processing data stored in Amazon S3 using tables.

The diagram below illustrates the current resolution framework.

The workflow comprises the following sequential steps:

1. A Lambda function will retrieve the next set of parameters based on its functionality.

   {
     "yarnUrl": "http://dummy.compute-1.amazonaws.com:8088/ws/v1/cluster/apps",
     "tblApplicationLogsLz": "public.emr_applications_execution_log_lz",
     "tblApplicationLogs": "public.emr_applications_execution_log",
     "tblEmrCost": "public.emr_cluster_usage_cost",
     "tblEmrInstanceUsage": "public.emr_cluster_instances_usage",
     "emrClusterId": "j-xxxxxxxxxx",
     "emrClusterName": "EMR Cost Measure",
     "emrClusterRole": "dt-dna-shared",
     "emrClusterLinkedAccount": "xxxxxxxxxxx",
     "postgresRds": {
       "host": "xxxxxxxxx.amazonaws.com",
       "databaseName": "postgres",
       "username": "postgresadmin",
       "secretId": "postgressecretid"
     }
   }

2. The Lambda function extracts Spark utility run logs from an EMR cluster using the Resource Supervisor API effectively. Metrics extracted as part of the methodology include vCore seconds, Reminiscence in megabytes per second, and Storage in gigabytes per second.
3. The Lambda function captures the daily price of EMR clusters from Value Explorer.

4. The Lambda function also extracts Amazon EMR On-Demand and Spot Occurrence usage data using the Amazon EC2 Boto3 APIs.

5. Lambda performs mass imports of these datasets into an Amazon Relational Database Service (RDS) instance.
6. The cost of executing a Spark utility is determined by the number of CPU cores it consumes relative to the total CPU utilization of all Spark tasks. This data is utilized to allocate the general price among distinct groups, business lines, or Electronic Medical Record (EMR) queues.

The daily extraction process runs each morning, consolidating yesterday’s knowledge and storing it securely on the server. The historical knowledge stored on the desk needs to be thoroughly cleansed, with decisions made according to specific use cases.

The solution is openly available and accessible online.

Use AWS CDK to orchestrate the deployment of a Lambda function, an RDS for PostgreSQL database schema defining entity relationships, and a QuickSight dashboard visualizing EMR cluster costs at the job, group, or organizational level.

The schemas presented illustrate the tables utilized within the resolution that are queried by QuickSight to populate the dashboard seamlessly.

• Storage for daily run metrics of all jobs executed on the Amazon Elastic MapReduce (EMR) cluster.
  • – Log assortment date
  • – Spark job run ID
  • – Run identify
  • The EMR cluster that executed the job.
  • – Job working state
  • Job closure report.Succeeded or Failed)
  • – Job begin time
  • – Job finish time
  • – Runtime in seconds
  • Consumed vCPU usage over time, measured in seconds.
  • – Reminiscence consumed
  • – Containers used
  • – EMR cluster ID
• Captures daily price consumption data for both Amazon Elastic Map Reduce (EMR) and Amazon Elastic Compute Cloud (EC2) using Amazon’s Value Explorer tool, and subsequently aggregates this information in a relational database management system (RDS) table.
  • – Value assortment date
  • – Value begin date
  • – Value finish date
  • – EMR cluster related tag
  • The daily average spot price of unblended crude oil.
  • Daily settlement price of the unblended crude oil contract.
  • – Each day price
  • AWS service that incurs an associated fee?
  • – EMR cluster ID
  • – EMR cluster identify
  • – Desk load date/time
• Monitors and aggregates valuable resource utilization metrics (vCores) per EMR cluster node, enabling accurate identification of idle time within the cluster.
  • – Occasion utilization acquire date
  • What energizing moments spark joy in your daily life? – EMR occasion energetic seconds within the day
  • – EMR cluster AWS Area
  • – EMR cluster ID
  • – EMR cluster identify
  • – EMR cluster fleet kind
  • – Occasion node kind
  • Marketplace (on-demand or provisioned)
  • – Occasion dimension
  • – Corresponding EC2 occasion ID
  • – Operating standing
  • – Allotted vCPU
  • – EC2 occasion reminiscence
  • – EC2 occasion creation date/time
  • – EC2 occasion finish date/time
  • – EC2 occasion prepared date/time
  • – Desk load date/time

Stipulations

Prior to executing any solution, it is crucial to establish the following fundamental conditions.

• An EMR on EC2 cluster.
• The EMR (Elastic MapReduce) cluster requires a singularly defined tag to outline its specifications clearly. You can assign the tag instantly on the Amazon EMR console or using AWS CLI. The truly powerful metadata hidden gem is cost-center Integrated into a single, high-value entity within your Enterprise Master Repository (EMR) cluster. After creating and applying user-defined tags, it typically takes up to 24 hours for the tag keys to be reflected on the price allocation tags webpage and become active.
• in AWS Billing. The activation process typically requires around 24 hours to complete, assuming timely initiation. To successfully activate the tag, follow these straightforward instructions:
  • In the AWS Billing and Cost Exploration console, navigate to the desired location via the left-hand sidebar.
  • I’ll get this done quickly, let’s go with ‘Editor Mode’ activated!
  • Select .
• The Spark utility’s identity should conform to the standardized naming convention. The dataset comprises seven distinct components, discreetly separated by underscores. <business_unit>_<program>_<utility>_<supply>_<job_name>_<frequency>_<job_type>. These components serve to condense valuable insights on resource utilization and pricing trends in the final report. For instance: HR_PAYROLL_PS_PSPROD_TAXDUDUCTION_DLY_LD, FIN_CASHRECEIPT_GL_GLDB_MAIN_DLY_LD, or MKT_CAMPAIGN_CRM_CRMDB_TOPRATEDCAMPAIGN_DLY_LD. The applicant should be informed that the applying identify spark submit command utilizing the --name The parameter with standardized naming conventions. If any of those elements do not have a price, default to the following values:
  • frequency
  • job_type
  • Business_unit
• The Lambda function should have the capability to integrate with Value Explorer, establish a connection with the EMR cluster via Resource Manager APIs, and load data into the RDS for PostgreSQL database. To successfully execute this process, you must first configure your Lambda function by setting up the necessary parameters in the following manner:
  • The Lambda function should have the capability to access the EMR cluster, Value Explorer, and Parameter Store. If an entry does not already exist, you can accomplish this by creating a digital private cloud (VPC), featuring the EMR cluster, as well as Parameters and Secrets Manager, and linking them to the VPC securely. Due to the lack of a VPC endpoint for Value Explorer, establishing direct connectivity between Lambda and Value Explorer necessitates creating a non-public subnet and a route table to redirect VPC traffic to a public NAT gateway. When operating an EMR cluster in a public subnet, consider creating a non-public subnet paired with a custom route table and a public NAT gateway. This setup enables communication between Value Explorer and your VPC’s private subnet. Configure the setup directions and securely attach the privately crafted subnet to the Lambda function with explicit clarity.
  • The Lambda execution role must possess an IAM function with the following permissions: AmazonEC2ReadOnlyAccess, AWSCostExplorerFullAccess, and AmazonRDSDataFullAccess. This routine AWS CDK stack deployment function is often established automatically, eliminating the need for manual setup.
• The AWS CDK should be installed with the most popular or an alternative improvement setting, similar to VSCode or PyCharm. To obtain additional information, consult with.
• The RDS for PostgreSQL database (version 10 or higher) requires its credentials to be stored securely in the Secrets Manager. To gather additional information, consider consulting with.

Create RDS tables

The following SQL script can be used to create the information schema views:

“`sql
CREATE VIEW v_column_privileges AS
SELECT s.table_catalog, t.table_name, c.column_name, d.privilege AS privilege_type,
privilege AS grantor,
privilege AS is_granted
FROM information_schema.tables t
JOIN information_schema.columns c ON t.table_schema = c.table_schema AND t.table_name = c.table_name
JOIN information_schema.table_privileges p ON t.table_catalog = p.table_catalog AND t.table_schema = p.table_schema AND t.table_name = p.table_name
JOIN information_schema.column_privileges d ON c.table_schema = d.table_schema AND c.table_name = d.table_name AND c.column_name = d.column_name;
“`

SKIP postgres rds Manually integrate into the general public schema.

Verify database connectivity using a tool like DBeaver or another suitable SQL client, ensuring that the RDS instance is successfully connected and the expected tables are present.

Deploy AWS CDK stacks

Deploy the next stages using the AWS Cloud Development Kit (CDK):

• To facilitate seamless integrations among retailers, we’ve established a set of essential parameter values that must be provided by each participating retailer. These parameters are crucial for ensuring accurate data exchange and minimizing potential issues.

  1. **API Key**: A unique identifier assigned by the retailer’s API management system, used to authenticate and authorize API requests.
  2. **Store Code**: A unique code representing a specific store or location, essential for routing orders and tracking inventory.
  3. **Order ID**: A unique identifier assigned to each order, used to track order status and facilitate returns processing.
  4. **Customer Email**: The email address of the customer placing an order, necessary for communication and order tracking.
  5. **Product ID**: A unique identifier representing a specific product or variant, used for inventory management and order fulfillment.

  By providing these mandatory parameter values, retailers can streamline their integrations, reduce errors, and improve overall efficiency.

• The IAM function for Lambda enables seamless connectivity with various services, including Amazon EMR, underpinning EC2 instances, as well as Value Explorer, CloudWatch, and Parameter Retailer.

• Lambda perform

1. Clone the GitHub repo:

   git clone git@github.com:aws-samples/attribute-amazon-emr-costs-to-your-end-users.git

2. Replace the next the setting parameters in cdk.context.json This document will be included in the main directory.
   1. To gain insight into job run logs and metrics? To enable access within a VPC where a Lambda function is hosted, the provided URL must be reachable from within that VPC.
   2. Retailer EMR utility run logs transferred to RDS temporary desk.
   3. Retailers’ EMR Utility Run Logs are available on the RDS Desk.
   4. – Regularly Review Daily Dashboard metrics to optimize EMR cluster resource allocation and cost efficiency.
   5. RDS desk delivers real-time utilization data for retailer EMR clusters to facilitate strategic decision-making.
   6. – EMR cluster occasion ID.
   7. – EMR cluster identify.
   8. Tag name assigned to EMR cluster.
   9. What’s unique value does an EMR cluster tag bring to the table?
   10. Service function for Amazon Elastic MapReduce (EMR).
   11. The account ID under which the Amazon EMR cluster operates.
   12. RDS for PostgreSQL Connection Particulars:
   13. The VPC ID through which an EMR cluster is configured, as well as the associated fee metering Lambda function that can be deployed to it.
   14. Non-public subnets with IDs, separated by commas, that are related to a Virtual Private Cloud (VPC).
   15. – EMR safety group ID.

The next is a pattern cdk.context.json The file, once fully stocked with the necessary specifications.

{
  "yarnUrl": "http://dummy.compute-1.amazonaws.com:8088/ws/v1/cluster/apps",
  "tblApplicationLogsLz": "public.emr_applications_execution_log_lz",
  "tblApplicationLogs": "public.emr_applications_execution_log",
  "tblEmrcost": "public.emr_cluster_usage_cost",
  "tblEmrinstanceUsage": "public.emr_cluster_instances_usage",
  "emrClusterId": "j-xxxxxxxxxx",
  "emrClusterName": "EMR Cluster Name",
  "emrClusterTag": "EMR Cluster Tag",
  "emrClusterTagValue": "EMR Cluster Unique Tag Value",
  "emrClusterRole": "EMR Cluster Service Role",
  "emrClusterLinkedAccount": "xxxxxxxxxxx",
  "postgresRds": {
    "host": "xxxxxxxxx.amazonaws.com",
    "dbname": "database name",
    "username": "user name",
    "secretId": "Database User Secret ID"
  },
  "vpcId": "xxxxxxxxx",
  "vpcSubnets": "subnet-xxxxxxxxxxx",
  "sgId": "xxxxxxxxxx"

You may choose to deploy the AWS CDK stack using AWS Cloud9 or another improvement setting based on your preferences. For setting up and utilizing AWS Cloud9, consult with experienced professionals at awscloud9.com.

3. Navigate to AWS Cloud9 and select the Venture folder for addition.
4. Creates a new AWS Cloud Development Kit (CDK) project and deploys the specified stack.

   cd attributes/amazon-emr-costs-to-your-end-users/; pip install -r requirements.txt; cdk deploy --all

The deployed Lambda function performs effectively with the inclusion of two essential external libraries: psycopg2 and requests. The necessary correspondence should be established and allocated to the Lambda function. To deploy a Lambda layer for your application, follow these steps:

1. Create a new directory for your layer and navigate into it in the terminal or command prompt. For example, you could name this directory `my-layer`. requests module, seek advice from .

Creation of the psycopg2 The package deal and layer are intricately linked to the Python runtime model used by Lambda functions.

Suppose the Lambda function uses the Python 3.9 runtime; complete the next steps to create the corresponding layer package that includes the required dependencies and libraries. peycopog2:

1. Obtain psycopg2_binary-2.9.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from .
2. Transfer the contents to a directory named python:

   zip ‘python’ listing

3. Create a Lambda layer for psycopg2 utilizing the zip file.
4. Select the Layer tab for the Lambda function and choose the desired layer from the list of available layers under the Deployed Perform Properties section.
5. Validate the AWS CDK deployment.

Your Lambda function’s particulars should resemble the following screenshot:

On the Programs Supervisor console, thoroughly verify the Parameter Retailer content for exact and accurate values.

The IAM function details should resemble the following code, which enables a Lambda function to access Amazon EMR and its underlying EC2 instances, Value Explorer, CloudWatch, Secret Manager, and Parameter Store:

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Actions": ["ce:GetCostAndUsage", "ce:ListCostAllocationTags", "ec2:AttachNetworkInterface", "ec2:CreateNetworkInterface", "ec2:DeleteNetworkInterface", "ec2:DescribeInstanceTypes", "ec2:DescribeInstances", "ec2:DescribeNetworkInterfaces", "elasticmapreduce:Describe*", "elasticmapreduce:List*", "ssm:Describe*", "ssm:Get*", "ssm:List*"],
      "Useful resource": "*",
      "Impact": "Enable"
    },
    {
      "Actions": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents"],
      "Useful resource": "arn:aws:logs:*:*:*",
      "Impact": "Enable"
    },
    {
      "Action": "secretsmanager:GetSecretValue",
      "Useful resource": "arn:aws:secretsmanager:*:*:*",
      "Impact": "Enable"
    }
  ]
}

Check the answer

Within an EMR cluster, you’ll be able to execute a Spark job that combines various data sets by defining distinct steps within the cluster. Consult with experts to obtain detailed instructions on how to integrate roles as step-by-step procedures within an Electronic Medical Record (EMR) cluster.

1. Submit the Spark job using the following command: spark-submit –class –master local[] target/SparkJob.jaremr_union_job.py).
   It takes in three arguments:
   1. The Amazon S3 location where the information file is learned by the Spark job. The trail shouldn’t be modified. The input_full_path is s3://aws-blogs-artifacts-public/artifacts/BDB-2997/sample-data/enter/part-00000-a0885743-e0cb-48b1-bc2b-05eb748ab898-c000.snappy.parquet
   2. The directory where outcome files are saved, typically referred to as an S3 bucket.
   3. By modifying the enter to the Spark job, you can ensure the job runs for varying durations and also adjust the number of Spot nodes utilized.

spark-submit --deploy-mode cluster --name HR_PAYROLL_PS_PSPROD_TAXDUDUCTION_DLY_LD s3://aws-blogs-artifacts-public/artifacts/BDB-2997/scripts/emr_union_job.py s3://aws-blogs-artifacts-public/artifacts/BDB-2997/sample-data/enter/part-00000-a0885743-e0cb-48b1-bc2b-05eb748ab898-c000.snappy.parquet s3://<output_bucket>/<output_path>/ 6

spark-submit --deploy-mode cluster --name FIN_CASHRECEIPT_GL_GLDB_MAIN_DLY_LD s3://aws-blogs-artifacts-public/artifacts/BDB-2997/scripts/emr_union_job.py s3://aws-blogs-artifacts-public/artifacts/BDB-2997/sample-data/enter/part-00000-a0885743-e0cb-48b1-bc2b-05eb748ab898-c000.snappy.parquet s3://<output_bucket>/<output_path>/ 12

The following screenshot displays a log detailing the sequence of actions executed within the Amazon Elastic MapReduce (EMR) interface.

2. Deploy the Lambda function from the Lambda console and then execute it. This script aggregates daily utility logs, EMR dollar utilization, and EMR event utilization details into their corresponding Amazon Relational Database Service (RDS) tables.

The following screenshot of the Amazon RDS question editor displays the results for public.emr_applications_execution_log.

The following screenshot displays the results for public.emr_cluster_usage_cost.

The following screenshot displays the results for public.emr_cluster_instances_usage.

Values will be calculated using the preceding three tables, as specified by your requirements. Based on relative utilization of all applications within a day, the associated fee is calculated. You initially quantify the total vCore-seconds CPU utilization over a 24-hour period, subsequently calculating the proportion of usage attributed to a specific utility. This determines the associated fee primarily based on the prevailing market price of a specific cryptocurrency cluster at any given time.

Considering the current situation, when 10 tasks were executed on the cluster for a specific day? You would utilise the following sequential steps to determine the chargeback price:

1. What is the percentage consumption of each application compared to the total CPU usage?
2. Now you may have the relative useful consumption of every utility, distributing cluster price fairly among each one. The total cost of the EMR cluster for that specific day is actually $400.

A pattern-based chargeback price calculation query is currently being implemented at the .

To generate actionable insights from the SQL query results, consider developing a comprehensive report dashboard that incorporates various visualizations, such as bar graphs, line charts, or scatter plots, to effectively communicate key findings and trends? Two examples of visualizations created using Amazon QuickSight’s powerful analytics capabilities are showcased below.

Each day’s bar chart.

The next exhibit shows total dollars consumed.

Answer price

Here is the rewritten text in a professional tone:

Calculations are performed daily for an environment that executes 1,000 tasks.

• One invocation per day for 30 days is required to meet the monthly Lambda perform quota.
• The vast array of data across public.emr_applications_execution_log A desk with approximately 5.72 MB of storage capacity can accommodate up to 30,000 pieces of information in a 30-day month, equivalent to roughly five records per second. Considering opposing factors of smaller tables and storage overhead, a rough estimate for the overall monthly storage demand would be approximately 12 megabytes.

The annual answer price, at $34.20, is remarkably affordable.

Clear up

To avoid ongoing expenses for the resources you’ve created, follow these next steps:

• Delete the AWS CDK stacks:
• Remove the QuickSight report and dashboard, if they were generated.
• DROP TABLE IF EXISTS orders;
  DROP TABLE IF EXISTS customers;
  DROP TABLE IF EXISTS products;

  DROP TABLE IF EXISTS public.emr_applications_execution_log_lz;
  DROP TABLE IF EXISTS public.emr_applications_execution_log;
  DROP TABLE IF EXISTS public.emr_cluster_usage_cost;
  DROP TABLE IF EXISTS public.emr_cluster_instances_usage;

Conclusion

By achieving this resolution, you’ll have the capability to deploy a chargeback model that assigns costs to users and teams using the EMR cluster’s infrastructure resources. You may also establish choices for optimizing, scaling, and separating workloads to different clusters based on utilization and growth requirements.

By accumulating metrics over an extended duration, you can effectively analyze trends in Amazon EMR source usage, leveraging this insight for informed forecasting purposes.

When you have any thoughts or concerns, leave them in the comments section.

In regards to the Authors

Is Amazon Web Services (AWS), as a lead marketing consultant, primarily focused on Knowledge Analytics offerings with an Indian-based presence? He prioritizes developing and refining advanced analytical tools. With a background in knowledge warehousing and lake administration, he excels at structuring, enhancing, and managing these complex systems. With more than 14 years of experience in the fields of knowledge and analysis.

 As a senior knowledge architect with WWCO ProServe at Amazon Web Services (AWS), As a subject matter expert, he collaborates closely with AWS customers to design, implement, and seamlessly transition their knowledge repositories and data hubs onto the cloud-based infrastructure of Amazon Web Services (AWS). When he’s not on the job, Ramesh loves exploring new places, cherishing moments with loved ones, and practicing yoga for mental clarity.

As a seasoned Senior Knowledge Architect with AWS-skilled companies, they excel at modernizing massive knowledge systems for clients, harnessing the power of the cloud to transform information architecture and enhance data accessibility. With a keen focus on crafting innovative analytics solutions, he’s driven to deliver timely insights that inform critical business decisions. Outside of work, he enjoys spending time with his family and has a penchant for watching movies and sporting events.

 As a lead marketing consultant with Amazon Net Services, primarily based out of India, he advises global clients on building highly secure, scalable, reliable, and cost-effective applications in the cloud. With a deep understanding of software development, structure, and analytics, he leverages his extensive experience from various sectors, including finance, telecommunications, retail, and healthcare.