The Amazon EMR runtime for Apache Spark is a performance-optimized runtime that’s 100% API suitable with open supply Apache Spark. It affords quicker out-of-the-box efficiency than Apache Spark by improved question plans, quicker queries, and tuned defaults. Amazon EMR on EC2, Amazon EMR Serverless, Amazon EMR on Amazon EKS, and Amazon EMR on AWS Outposts all use this optimized runtime, which is 4.5 instances quicker than Apache Spark 3.5.1 and has 2.8 instances higher price-performance based mostly on an trade normal benchmark derived from TPC-DS at 3 TB scale (notice that our TPC-DS derived benchmark outcomes should not instantly comparable with official TPC-DS benchmark outcomes).
We added 35 optimizations for the reason that EOY 2022 launch, EMR 6.9, which are included in each EMR 7.0 and EMR 7.1. These enhancements are turned on by default and are 100% API suitable with Apache Spark. A number of the enhancements since our earlier put up, Amazon EMR on EKS widens the efficiency hole, embrace:
- Spark bodily plan operator enhancements – We proceed to enhance Spark runtime efficiency by altering the operator algorithms:
- Optimized information constructions utilized in hash joins for efficiency and reminiscence necessities, permitting using extra performant be part of algorithm for extra circumstances
- Optimized sorting for partial window
- Optimized rollup operations
- Improved type algorithm for shuffle partitioning
- Optimized hash combination operator
- Extra environment friendly decimal arithmetic operations
- Aggregates based mostly on Parquet statistics
- Spark question planning enhancements – We launched new guidelines within the Spark’s Catalyst optimizer to enhance effectivity:
- Adaptively decrease redundant joins
- Adaptively establish and disable unhelpful optimizations at runtime
- Infer extra superior Bloom filters and dynamic partition pruning filters from complicated question plans to scale back quantity of knowledge shuffled and browse from Amazon Easy Storage Service (Amazon S3)
- Fewer requests to Amazon S3 – We diminished requests despatched to Amazon S3 when studying Parquet recordsdata by minimizing pointless requests and introducing a cache for Parquet footers.
- Java 17 as default Java runtime utilized in Amazon EMR 7.0 – Java 17 was extensively examined and tuned for optimum efficiency, permitting us to make it the default Java runtime for Amazon EMR 7.0.
For extra particulars on EMR Spark efficiency optimizations, consult with Optimize Spark efficiency.
On this put up, we share the testing methodology and benchmark outcomes evaluating the newest Amazon EMR variations (7.0 and seven.1) with the EOY 2022 launch (model 6.9) and Apache Spark 3.5.1 to display the newest value enhancements Amazon EMR has achieved.
Benchmark outcomes for Amazon EMR 7.1 vs. Apache Spark 3.5.1
To guage the Spark engine efficiency, we ran benchmark checks with the three TB TPC-DS dataset. We used EMR Spark clusters for benchmark checks on Amazon EMR and put in Apache Spark 3.5.1 on Amazon Elastic Compute Cloud (Amazon EC2) clusters designated for open supply Spark (OSS) benchmark runs. We ran checks on separate EC2 clusters comprised of 9 r5d.4xlarge situations for every of Apache Spark 3.5.1, Amazon EMR 6.9.0, and Amazon EMR 7.1. The first node has 16 vCPU and 128 GB reminiscence and eight employee nodes have a complete of 128 vCPU and 1024 GB reminiscence. We examined with Amazon EMR defaults to focus on the out-of-the-box expertise and tuned Apache Spark with the minimal settings wanted to offer a good comparability.
For the supply information, we selected the three TB scale issue, which incorporates 17.7 billion data, roughly 924 GB of compressed information in Parquet file format. The setup directions and technical particulars might be discovered within the GitHub repository. We used Spark’s in-memory information catalog to retailer metadata for TPC-DS databases and tables. spark.sql.catalogImplementation is about to the default worth in-memory. The actual fact tables are partitioned by the date column, which consists of partitions starting from 200–2,100. No statistics have been pre-calculated for these tables.
A complete of 104 SparkSQL queries have been run in three iterations sequentially and a mean of every question’s runtime in these three iterations was used for comparability. The typical of the three iterations’ runtime on Amazon EMR 7.1 was 0.51 hours, which is 1.9 instances quicker than Amazon EMR 6.9 and 4.5 instances quicker than Apache Spark 3.5.1. The next determine illustrates the overall runtimes in seconds.
The per-query speedup on Amazon EMR 7.1 when in comparison with Apache Spark 3.5.1 is illustrated within the following chart. Though Amazon EMR is quicker than Apache Spark on all TPC-DS queries, the speedup is way larger on some queries than on others. The horizontal axis represents queries within the TPC-DS 3 TB benchmark ordered by the Amazon EMR speedup descending and the vertical axis reveals the speedup of queries because of the Amazon EMR runtime.
Value comparability
Our benchmark outputs the overall runtime and geometric imply figures to measure the Spark runtime efficiency by simulating a real-world complicated resolution help use case. The associated fee metric can present us with further insights. Value estimates are computed utilizing the next formulation. They consider Amazon EC2, Amazon Elastic Block Retailer (Amazon EBS), and Amazon EMR prices, however don’t embrace Amazon S3 GET and PUT prices.
- Amazon EC2 value (embrace SSD value) = variety of situations * r5d.4xlarge hourly fee * job runtime in hours
- 4xlarge hourly fee = $1.152 per hour
- Root Amazon EBS value = variety of situations * Amazon EBS per GB-hourly fee * root EBS quantity dimension * job runtime in hours
- Amazon EMR value = variety of situations * r5d.4xlarge Amazon EMR value * job runtime in hours
- 4xlarge Amazon EMR value = $0.27 per hour
- Whole value = Amazon EC2 value + root Amazon EBS value + Amazon EMR value
Based mostly on the calculation, the Amazon EMR 7.1 benchmark end result demonstrates a 2.8 instances enchancment in job value in comparison with Apache Spark 3.5.1 and a 1.7 instances enchancment when in comparison with Amazon EMR 6.9.
| Metric | Amazon EMR 7.1 | Amazon EMR 6.9 | Apache Spark 3.5.1 |
| Runtime in hours | 0.51 | 0.87 | 1.76 |
| Variety of EC2 situations | 9 | 9 | 9 |
| Amazon EBS Dimension | 20gb | 20gb | 20gb |
| Amazon EC2 value | $5.29 | $9.02 | $18.25 |
| Amazon EBS value | $0.01 | $0.02 | $0.04 |
| Amazon EMR value | $1.24 | $2.11 | $0.00 |
| Whole value | $6.54 | $11.15 | $18.29 |
| Value Financial savings | Baseline | Amazon EMR 7.1 is 1.7 instances higher | Amazon EMR 7.1 is 2.8 instances higher |
Run OSS Spark benchmarking
For operating Apache Spark 3.5.1, we used the next configurations to arrange an EC2 cluster. We used one main node and eight employee nodes of sort r5d.4xlarge.
| EC2 Occasion | vCPU | Reminiscence (GiB) | Occasion Storage (GB) | EBS Root Quantity (GB) |
| r5d.4xlarge | 16 | 128 | 2 x 300 NVMe SSD | 20GB |
Stipulations
The next stipulations are required to run the benchmarking:
- Utilizing the directions within the emr-spark-benchmark GitHub repo, arrange the TPC-DS supply information in your S3 bucket and your native laptop.
- Construct the benchmark software following the steps offered in Steps to construct spark-benchmark-assembly software and duplicate the benchmark software to your S3 bucket. Alternatively, copy spark-benchmark-assembly-3.5.1.jar to your S3 bucket.
This benchmark software is constructed from department tpcds-v2.13. Should you’re constructing a brand new benchmark software, swap to the right department after downloading the supply code from the GitHub repo.
Create and configure a YARN cluster on Amazon EC2
Observe the directions within the emr-spark-benchmark GitHub repo to create an OSS Spark cluster on Amazon EC2 utilizing Flintrock.
Based mostly on the cluster choice for this check, the next are the configurations used:
Run the TPC-DS benchmark for Apache Spark 3.5.1
Full the next steps to run the TPC-DS benchmark for Apache Spark 3.5.1:
- Log in to the OSS cluster main utilizing
flintrock login $CLUSTER_NAME. - Submit your Spark job:
- The TPC-DS supply information is at
s3a://<YOUR_S3_BUCKET>/BLOG_TPCDS-TEST-3T-partitioned. Verify the stipulations on how you can arrange the supply information. - The outcomes are created in
s3a://<YOUR_S3_BUCKET>/benchmark_run. - You possibly can monitor progress in
/media/ephemeral0/spark_run.log.
- The TPC-DS supply information is at
Summarize the outcomes
When the Spark job is full, obtain the check end result file from the output S3 bucket s3a://<YOUR_S3_BUCKET>/benchmark_run/timestamp=xxxx/abstract.csv/xxx.csv. You need to use the Amazon S3 console and navigate to the output bucket location or use the Amazon Command Line Interface (AWS CLI).
The Spark benchmark software creates a timestamp folder and writes a abstract file inside a abstract.csv prefix. Your timestamp and file title will likely be totally different from the one proven within the previous instance.
The output CSV recordsdata have 4 columns with out header names:
- Question title
- Median time
- Minimal time
- Most time
As a result of now we have three runs, we are able to then compute the common and geometric imply of the runtimes.
Run the TPC-DS benchmark utilizing Amazon EMR Spark
For detailed directions, see Steps to run Spark Benchmarking.
Stipulations
Full the next prerequisite steps:
- Run aws configure to configure your AWS CLI shell to level to the benchmarking account. Check with Configure the AWS CLI for directions.
- Add the benchmark software to Amazon S3.
Deploy the EMR cluster and run the benchmark job
Full the next steps to run the benchmark job:
- Use the AWS CLI command as proven in Deploy EMR Cluster and run benchmark job to spin up an EMR on EC2 cluster. Replace the offered script with the right Amazon EMR model and root quantity dimension, and supply the values required. Check with create-cluster for an in depth description of the AWS CLI choices.
- Retailer the cluster ID from the response. You want this within the subsequent step.
- Submit the benchmark job in Amazon EMR utilizing add-steps within the AWS CLI:
- Exchange <cluster ID> with the cluster ID from the create cluster response.
- The benchmark software is at
s3://<YOUR_S3_BUCKET>/spark-benchmark-assembly-3.5.1.jar. - The TPC-DS supply information is at
s3://<YOUR_S3_BUCKET>/BLOG_TPCDS-TEST-3T-partitioned. - The outcomes are created in
s3://<YOUR_S3_BUCKET>/benchmark_run.
Summarize the outcomes
After the job is full, retrieve the abstract outcomes from s3://<YOUR_S3_BUCKET>/benchmark_run in the identical means because the OSS benchmark runs and compute the common and geomean for Amazon EMR runs.
Clear up
To keep away from incurring future fees, delete the assets you created utilizing the directions within the Cleanup part of the GitHub repo.
Abstract
Amazon EMR continues to enhance the EMR runtime for Apache Spark, resulting in a efficiency enchancment of 1.9x year-over-year and 4.5x quicker efficiency than OSS Spark 3.5.1. We advocate that you simply keep updated with the newest Amazon EMR launch to make the most of the newest efficiency advantages.
To maintain updated, subscribe to the Massive Information Weblog’s RSS feed to study extra concerning the EMR runtime for Apache Spark, configuration finest practices, and tuning recommendation.
In regards to the creator
Ashok Chintalapati is a software program growth engineer for Amazon EMR at Amazon Net Companies.
Steve Koonce is an Engineering Supervisor for EMR at Amazon Net Companies.
