Since its launch in 2013, the platform has undergone significant evolution, empowering clients to expand their capacity for data warehousing and advanced SQL analytics capabilities. Currently, Amazon Redshift serves a broad clientele across various industries, leveraging its capabilities for diverse use cases, including data warehouse migration and modernization, near real-time analytics, self-service analytics, data lake analytics, machine learning, and data monetization.
Amazon Redshift marked significant advancements in 2024, unveiling more than a hundred innovative features and upgrades. These innovations significantly improved value-for-money ratios, bridged the gap between data lakes and data warehouses through seamless integration, streamlined data ingestion, and expedited real-time analysis, while also integrating generative AI capabilities to create natural language-based tools and amplify user productivity.
Figure 1: A concise summary of key options and enhancements planned for 2024.
Let’s take a stroll through some of the latest key launches and exciting news from AWS re:Invent 2024.
Business-leading price-performance
Amazon Redshift boasts a price-performance advantage of up to three times that of other cloud data warehousing solutions. Amazon Redshift’s scalability enables it to handle a growing number of customers and data volumes in a linear fashion, making it an ideal solution for scaling businesses and enterprises. In Redshift user environments, dashboarding functions are a prevalent example of query scenarios that necessitate fast, low-latency responses to handle excessive concurrency effectively. When pitting Amazon Redshift against other cloud data warehouses, the service delivers up to seven times more throughput per dollar spent, underscoring its unique value proposition and transparent pricing model.
Efficiency enhancements
In recent months, we’ve successfully rolled out multiple efficiency improvements to Amazon Redshift. First-instance responses for dashboard queries have significantly accelerated by streamlining code execution and minimizing compilation latency. By streamlining metadata processing, we’ve achieved a significant acceleration in information sharing, allowing producers’ data updates to be executed approximately 4 times faster from initial request. We have refined our autonomous algorithm capabilities to produce and integrate more intelligent and expedient optimal information architecture recommendations for dissemination and key types, further enhancing operational efficiency. We’ve introduced a compact, high-performance RA3 node format, offering greater flexibility in terms of price-to-performance ratio and providing an affordable migration path for customers currently using DC2 massive cases. With the launch of AWS Graviton in Serverless, we’ve achieved up to 30% better price-performance, while also introducing expanded concurrency scaling to support a broader range of write queries, ultimately allowing for significantly increased capacity to maintain consistent performance at scale. Amazon Redshift’s latest upgrades further solidify its position as the top-tier cloud data warehousing solution, delivering unparalleled speed and value to customers.
Availability of Multi-Data Warehouse Writes: A Growing Concern?
Traditional data warehousing has long been a cornerstone of business intelligence. However, the proliferation of big data and the Internet of Things (IoT) has created an explosion in data volume and complexity. This has led to the need for multiple data warehouses, each catering to specific needs and use cases.
Amazon Redshift enables scalable and flexible data warehousing through its support for multiple cluster deployments. With the advent of managed storage in 2019, customers gained the flexibility to scale and pay for compute and storage resources separately. Launched in 2020, Redshift pioneered real-time data sharing, allowing for seamless collaboration across accounts and regions without physically relocating information, thereby preserving transactional integrity. This feature enables clients to seamlessly scale their learn analytics workloads, thereby ensuring the preservation of stringent Service Level Agreements (SLAs) for mission-critical processes. At AWS re:Invent 2024, the highly anticipated milestone was reached with the general availability of Amazon Redshift’s RA3 nodes and Serverless capabilities. Now, with just a few clicks, you can seamlessly connect to and write data to multiple Amazon Redshift databases across your organization’s information warehouses. The data is disseminated rapidly to every information hub, owing to its singular focus. This enables groups to dynamically scale write workloads similar to extract, transform, and load (ETL) processes and data processing by allocating compute resources of varying types and sizes based on individual workload price-performance requirements, as well as securely collaborating with other teams on live data for use cases such as customer 360.
Accessibility of AI-powered scalability solutions has revolutionized the way businesses operate.
The introduction of pay-per-use in 2021 brought about a significant transformation, obviating the need for cluster management while offering a cost-effective solution that aligns with actual usage? By leveraging Redshift Serverless, organizations empowered their information-sharing-enabled clients to seamlessly deploy scalable, distributed multi-cluster architectures for analytics workloads. In 2024, Amazon Web Services expanded the availability of its Serverless offering to an additional 10 regions, enhancing performance and introducing support for configurations of up to 1024 Reserved Processing Units (RPUs), enabling customers to process larger workloads on Redshift with greater ease. Redshift Serverless has been enhanced to demonstrate greater intelligence and adaptability through its innovative features. When purchasing, you determine whether to prioritize optimizing workloads for value, efficiency, or maintaining a balance between the two, and your decision is final. Redshift Serverless seamlessly scales compute resources up or down, deploying optimisations that maintain efficiency ranges, even as workloads dynamically shift and evolve. During internal examinations, the integration of AI-powered scaling and optimization techniques yielded remarkable improvements in price-performance ratios, with a significant boost of up to 10 times for diverse workload scenarios.
Seamless Lakehouse architectures
By combining the scalability and data fluidity of information lakes with the streamlined processing and analytical power of information warehouses, Lakehouse empowers seamless data integration and insights. Lakehouse empowers users to seamlessly integrate their preferred analytics engines and AI models, ensuring seamless governance and control across all data assets. At AWS re:Invent 2024, we introduced the , a comprehensive platform for data, insights, and artificial intelligence. This launch integrates widely used AWS machine learning and analytics features, providing a seamless blend of analytics and AI capabilities through a reengineered lakehouse and robust governance framework.
Amazon SageMaker Lakehouse is currently available in the following regions: us-east-1, us-west-2, eu-west-1, ap-northeast-1, and ap-southeast-2?
By integrating your data across Amazon S3 data lakes and Redshift data warehouses, you can seamlessly create sophisticated analytics and AI/ML models based on a unified dataset. SageMaker Lakehouse enables flexible access and querying of data using Apache Iceberg open tables, allowing for seamless integration with popular AWS, open-source, or third-party Iceberg-compliant engines and tools. SageMaker Lakehouse offers intrinsic entry controls and granular permissions that remain consistently applied across all analytics engines, AI models, and tools. By leveraging SageMaker Lakehouse, you can seamlessly make Redshift information warehouses accessible through a straightforward publishing process, unlocking the full potential of your data warehouse insights with effortless access to Iceberg’s REST API. Can you easily create new data lake tables utilizing Redshift Managed Storage (RMS), offering a local storage option? Introduced at AWS re:Invent 2024?
What’s the future of data science? Discover the power of unified collaboration with Amazon SageMaker Unified Studio – a revolutionary platform that streamlines your workflow from data prep to deployment. With seamless integration across Jupyter notebooks, Apache Spark, and AWS services like Glue, LakeFormation, and Rekognition, you’ll be able to tackle complex projects with ease.
Say goodbye to tedious setup and hello to productive workflows as you work alongside colleagues in a centralized, cloud-based environment. Visualize your data insights with ease using the intuitive Jupyter notebook interface or leverage Apache Spark for large-scale analytics. Plus, get instant access to AWS services like SageMaker Autopilot for automated machine learning and SageMaker Ground Truth for high-quality training data.
Take collaboration to new heights by sharing notebooks, datasets, and results seamlessly across teams and departments. With Amazon SageMaker Unified Studio, you can accelerate innovation, reduce costs, and improve decision-making with data-driven insights that drive business success.
An integrated setting for information and AI growth, this feature enables seamless collaboration among teams to accelerate the development of informative products. Amazon SageMaker Unified Studio harmoniously consolidates performance and tools from a diverse range of standalone studios, question editors, and visual instruments currently available within Amazon EMR, AWS Glue, Amazon Redshift, Amazon Bedrock, and existing Amazon SageMaker Studio, to deliver a cohesive experience. With SageMaker Unified Studio, diverse groups – comprising builders, analysts, data scientists, and business stakeholders – can effortlessly collaborate, share resources, execute analytics, and develop and refine models, thereby facilitating a streamlined and efficient analytics and AI experience.
Amazon Redshift enables scalable, fast querying of data stored in Amazon S3 tables, utilizing ANSI-standard SQL.
At re:Invent 2024, Amazon S3 introduced Amazon S3 Tables, a novel bucket designed specifically for storing large volumes of tabular data at scale, featuring seamless integration with Iceberg capabilities. By using desk buckets, you will quickly be able to create tables and configure table-level permissions to manage access to your data lake. Amazon Redshift has expanded its support for querying data stored in Amazon S3 tables, building on the functionality introduced for Iceberg information in information lakes last year? S3 tables created by clients are also available within the Lakehouse, enabling consumption by various AWS and third-party engines.
Knowledge lake question efficiency
Amazon Redshift now seamlessly integrates with SageMaker Lakehouse to deliver high-performance SQL capabilities, supporting data from both Redshift warehouses and open formats, regardless of their origin. We significantly enhanced support for querying Apache Iceberg metadata, resulting in a threefold improvement in query efficiency year-on-year. Optimizations of various sorts significantly boost efficiency, as well as the seamless integration with AWS Glue’s knowledge catalog statistics, enhanced metadata and information filtering capabilities, real-time dynamic partition elimination, expedited parallel processing of Iceberg manifests, and advanced scanner upgrades. Amazon Redshift now enables incremental refresh capabilities for materialized views on information lake tables, eliminating the need to recompute the materialized view when new data arrives, thereby streamlining the process of building interactive analytics on S3-based data lakes.
Streamlined data intake and near-instant insights
We facilitate seamless data intake and provide near-instant analytics, empowering you to derive timely insights from up-to-date information.
Real-time data integration is seamless when Zero-ETL harmonizes with AWS databases and third-party enterprise functions. By eliminating the need for Extract, Transform, Load (ETL) processes, organizations can accelerate their analytics capabilities and reduce operational complexity.
Amazon Redshift has introduced seamless zero-ETL integration with Aurora, empowering organizations to gain insights into massive datasets and analyze petabytes of transactional data in near-real time. This functionality has evolved to support a broader range of use cases, including data exploration, data integration, and data warehousing, and comprises additional features such as table extraction using common expressions, incremental and auto-refreshed materialized views on replicated data, and configurable change data capture (CDC) refresh rates.
Building on this innovation, at re:Invent 2024, Salesforce, alongside esteemed partners such as Zendesk, ServiceNow, SAP, Facebook Ads, Instagram Ads, Pardot, and Zoho CRM, will further develop this groundbreaking concept. With this innovative feature, you’ll effortlessly extract and integrate valuable insights from your customer support, relationship management, and Enterprise Resource Planning (ERP) modules directly into your Redshift data warehouse for analysis. With this seamless integration, the need for complex, tailored ingestion processes is eliminated, significantly speeding up the journey to actionable intelligence.
Common availability of auto-copy
Enables seamless integration of data from Amazon S3 into Amazon Redshift for streamlined information consumption. With this innovative feature, effortlessly orchestrate continuous file ingestion from your designated Amazon S3 prefix, seamlessly loading fresh data into your Redshift data warehouse without requiring additional tools or bespoke configurations.
Streaming data ingestion from Confluent Managed Cloud and self-managed Apache Kafka clusters enables seamless integration of various data sources into a unified analytics pipeline.
Amazon Redshift has expanded its scope by seamlessly integrating with Amazon EC2 instances, building upon the already robust offerings of Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK), further empowering users. By integrating with a broader spectrum of streaming sources, you can seamlessly ingest information into your Redshift data warehouses, enabling near-real-time analytics capabilities that support applications such as fraud detection, logistics monitoring, and clickstream analysis.
Generative AI capabilities
We highlight advancements in our generative artificial intelligence capabilities.
We’ve incorporated a defining feature into the Redshift Question Editor. Amazon’s Quest generative SQL enhances productivity by allowing users to formulate queries using natural language and receive suggested SQL code based on their intention, query patterns, and schema information. The conversational interface enables customers to gain valuable insights more quickly by providing instant access to summarized database schema information. Utilizing generative artificial intelligence, the system explores consumer insights, inquires about historical background, and considers tailored contexts such as desk or column descriptions and pattern queries to provide more pertinent and accurate SQL recommendations. This characteristic significantly accelerates the question-authoring process, thereby reducing the time needed to extract valuable insights.
What’s the best way to get started with Amazon Redshift integration using AWS Lake Formation? The first step is to create an Amazon Lake Formation lake. To do this, navigate to the Lake Formation console and click on “Create lake.” Fill out the required details such as lake name, description, and storage location. Once your lake has been created, you can start building your data warehouse by creating tables, loading data, and querying with SQL.
SKIP
We’ve introduced a seamless integration, empowering users to effortlessly deploy massive language models (LLMs) via simple SQL queries within their Amazon Redshift data. With this novel characteristic, you can seamlessly execute generative AI tasks akin to language translation, text generation, summarization, customer classification, and sentiment analysis within your Redshift data utilizing widely-established foundation models (FMs) such as Anthropic’s Claude, Amazon Titan, Meta’s Llama 2, and Mistral AI. By leveraging familiar SQL syntax, you can seamlessly integrate generative AI capabilities with your data analytics workflows, making the process easier than ever before?
Amazon Redshift as a database on Amazon Bedrock? That doesn’t quite add up.
Amazon Redshift is a fully-managed cloud data warehouse service offered by AWS (Amazon Web Services), not Amazon Bedrock, which doesn’t seem to be a real entity.
Extracting insights from data stored in Amazon Redshift information warehouses? Through advanced natural language processing, Amazon Bedrock Data Bases seamlessly transforms pure language queries into optimized SQL queries, enabling rapid and direct access to relevant data without requiring manual reformatting or preprocessing. Retail analysts can effortlessly pose queries like “Which were my top five selling products last month?” to be immediately answered by Amazon’s Bedrock Data Bases, which seamlessly interprets natural language into SQL, executes the query against Redshift, and delivers the results alongside a concise narrative summary. Amazon Bedrock databases utilize a database schema, combined with historical query patterns and various contextual data provided about the data sources to generate accurate SQL queries.
Launch abstract
Announcing Key Updates and Initiatives: Explore Our Latest News and Insights.
• Click here to learn more about our latest breakthroughs in [Topic 1] [Link 1]
• Discover how we’re driving innovation in [Industry/Field] with these thought-provoking articles [Blog 1, Blog 2]
• Stay ahead of the curve with our expert analysis on [Trend 1], [Trend 2] [Reference 1, Reference 2]
• Get instant access to exclusive updates and behind-the-scenes stories from our team [Newsletter Link]
Business-leading price-performance:
Reference Blogs:
Seamless Lakehouse architectures:
Reference Blogs:
Streamlined data consumption with instantaneous insights.
Reference Blogs:
Generative AI:
Reference Blogs:
Conclusion
As we continuously push forward in innovating and evolving Amazon Redshift, we aim to meet the growing demands of your organization’s ever-changing information analytics needs. Discover the latest innovations and enhancements by exploring our cutting-edge offerings today! Attend the re:Invent 2024 session to gain further insights. If you need assistance with anything, feel free to reach out to us. We’re pleased to provide comprehensive architectural and design guidance, as well as support for concept development and execution. It’s Day 1!
In regards to the Writer
Is Director of Product Administration for AWS Analytics, primarily focused on Amazon Redshift and Amazon SageMaker Lakehouses? With a storied career spanning more than two decades, Neeraja stands as an accomplished leader, boasting a wealth of experience in guiding product vision, strategy, and management for innovative data products and platforms. With a proven track record in delivering cutting-edge solutions in analytics, databases, information integration, utility integration, AI/ML, and large-scale distributed methodologies across on-premises and cloud environments, she has had the privilege of serving several Fortune 500 companies as part of strategic ventures with esteemed partners such as MapR (acquired by HPE), Microsoft SQL Server, Oracle, Informatica, and Expedia.com.
