Right this moment, Amazon Net Providers (AWS) introduced the final availability of Apache Airflow 3 on Amazon Managed Workflows for Apache Airflow (Amazon MWAA). This launch transforms how organizations use Apache Airflow to orchestrate knowledge pipelines and enterprise processes within the cloud, bringing enhanced safety, improved efficiency, and trendy workflow orchestration capabilities to Amazon MWAA prospects.
Amazon MWAA introduces Airflow 3 options that modernize workflow administration for AWS prospects. Following the April 2025 launch of Airflow 3 by the Apache group, AWS has included these capabilities into Amazon MWAA. Airflow now encompasses a fully redesigned, intuitive UI that simplifies workflow orchestration for customers throughout expertise ranges. With the Process Execution Interface (Process API), duties can run each inside Airflow and as standalone Python scripts, bettering code portability and testing. Scheduler-managed Backfill strikes operations from the CLI to the scheduler, offering centralized management and visibility by means of the Airflow UI. CLI safety enhancements change direct database entry with API calls, sustaining constant safety throughout interfaces. Airflow now helps event-driven workflows, enabling triggers from AWS companies and exterior sources. Amazon MWAA additionally provides help for Python 3.12, bringing the newest language capabilities to workflow growth.
This put up explores the options of Airflow 3 on Amazon MWAA and descriptions enhancements that enhance your workflow orchestration capabilities. The service maintains the Amazon MWAA pay-as-you-go pricing mannequin with no upfront commitments. You may start instantly by visiting the Amazon MWAA console, launching new Apache Airflow environments by means of the AWS Administration Console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, or AWS SDK inside minutes.
Architectural developments in Airflow 3 on Amazon MWAA
Airflow 3 on Amazon MWAA introduces vital architectural enhancements that improve safety, efficiency, and adaptability. These developments create a extra strong basis for workflow orchestration whereas sustaining backward compatibility with current workflows.
Enhanced safety
Amazon MWAA with Airflow 3 modifications the safety mannequin by making part isolation an ordinary apply relatively than non-compulsory. In Airflow 2, the DAG processor (the part that parses and processes DAG recordsdata) runs inside the scheduler course of by default, however can optionally be separated into its personal course of for higher scalability and safety isolation. Airflow 3 makes this separation normal, sustaining constant safety practices throughout deployments.
API server and Process API
Constructing on this safety basis, a brand new API server part is launched in Amazon MWAA with Airflow 3, which serves as an middleman between process situations and the Airflow metadata database. This transformation improves your workflows’ safety posture by minimizing direct entry to the Airflow metadata database from duties. Duties now function with least privilege database entry, lowering the danger of 1 process affecting others and bettering general system stability by means of fewer direct database connections.
The standardized communication by means of well-defined API endpoints creates a basis for safer, scalable, and versatile workflow orchestration. The Process Execution Interface (Process API) helps duties run each inside Airflow and as standalone Python scripts, bettering code portability and testing capabilities.
From data-aware to event-driven scheduling
Airflow’s evolution towards event-driven scheduling started with the introduction of data-aware scheduling in Airflow 2.4, so DAGs might be triggered based mostly on knowledge availability relatively than time schedules alone. Amazon MWAA with Airflow 3 builds on this basis by means of a transition that features the renaming of datasets to property and introduces superior capabilities, together with asset partitions, exterior occasion integration, and asset-centric workflow design.
The transition from datasets to property represents greater than a easy rename. A knowledge asset is a set of logically associated knowledge that may characterize various knowledge merchandise, together with database tables, persevered ML fashions, embedded dashboards, or directories containing recordsdata.
Amazon MWAA with Airflow 3 introduces a brand new asset-centric syntax that represents an essential shift in how workflows will be designed. The @asset decorator helps builders put knowledge property on the middle of their workflow design, creating extra intuitive asset-driven pipelines.
The next code is an instance of asset-aware DAG scheduling:
The next code reveals an asset-centric strategy with the @asset decorator:
The @asset decorator mechanically creates an asset with the operate title, a DAG with the identical identifier, and a process that produces the asset. This reduces code complexity and facilitates computerized DAG creation, the place every asset turns into a self-contained workflow unit.
Exterior event-driven scheduling with Asset Watchers
A big development in Amazon MWAA with Airflow 3 is the introduction of Asset Watchers, which assist Airflow react to occasions occurring exterior of the Airflow system itself. Whereas earlier variations supported inner cross-DAG dependencies, Asset Watchers prolong this functionality to exterior knowledge methods and message queues by means of the AssetWatcher class.
Amazon MWAA with Airflow 3 contains help for Amazon Easy Queue Service (Amazon SQS) by means of Asset Watchers. This enables your workflows to be triggered by exterior messages and facilitates extra event-driven scheduling. Airflow now helps event-driven workflows, enabling triggers from AWS companies and exterior sources. Asset Watchers monitor exterior methods asynchronously and set off workflow execution when particular occasions happen, enabling workflows to answer enterprise occasions, knowledge updates, or system notifications with out the overhead of conventional sensor-based polling mechanisms.
Fashionable React-based UI
Amazon MWAA with Airflow 3 encompasses a fully redesigned, intuitive UI constructed with React and FastAPI that simplifies workflow orchestration for customers throughout expertise ranges. The brand new interface supplies extra intuitive navigation and workflow visualization, with an enhanced grid view that gives higher visibility into process standing and historical past. Customers will respect the addition of darkish mode help, which reduces eye pressure throughout prolonged use, and the general quicker efficiency that’s particularly noticeable when working with massive DAGs.
The brand new UI maintains acquainted workflows whereas offering a extra trendy and environment friendly expertise for DAG administration and monitoring, making day by day operations extra productive for each builders and operators. The legacy UI has been fully eliminated, providing a cleaner, extra constant expertise throughout the system. The muse for the brand new UI is constructed on REST APIs and a set of inner APIs for UI operations, each of which are actually based mostly on FastAPI, making a extra cohesive and safe structure for each programmatic entry and UI operations.
Scheduler optimizations
Amazon MWAA with Airflow 3’s enhanced scheduler delivers efficiency enhancements for process execution and workflow administration. The redesigned scheduling engine processes duties extra effectively, lowering the time between process submissions and executions. This optimization advantages knowledge pipeline operations that require fast process processing and well timed workflow completion.
The scheduler now manages computing sources extra successfully, enabling steady efficiency whilst workloads scale. When working a number of DAGs concurrently, the improved useful resource allocation system helps stop bottlenecks and maintains constant execution speeds. This development is especially helpful for organizations working advanced workflows with various useful resource necessities. The brand new scheduler additionally handles concurrent operations with elevated precision, so groups can run a number of DAG situations concurrently whereas sustaining system stability and predictable efficiency.
Enhanced scheduler backfill operations
Scheduler-managed backfill (the method of working DAGs for historic dates) strikes operations from the CLI to the scheduler, offering centralized management and visibility by means of the Airflow UI. Amazon MWAA with Airflow 3 delivers essential upgrades to the scheduler’s backfill capabilities, serving to knowledge groups course of historic knowledge extra effectively. The backfill course of has been optimized for higher efficiency, lowering the database load throughout these operations and ensuring backfills will be accomplished extra shortly, minimizing the influence on close to real-time workflow execution.
Amazon MWAA with Airflow 3 additionally improves the administration of backfill operations, with the scheduler offering higher isolation between backfill jobs and supporting extra environment friendly processing of historic datasets. Operators now have higher monitoring instruments to trace the progress and standing of their backfill jobs, leading to more practical administration of those important knowledge processing duties.
Developer-focused enhancements
Airflow 3 on Amazon MWAA delivers a number of enhancements designed to enhance the developer expertise, from simplified process definition to higher workflow administration capabilities.
Process SDK
The Process SDK supplies a extra intuitive technique to outline duties and DAGs:
This strategy affords extra intuitive knowledge stream between duties, higher built-in growth setting (IDE) help with improved kind hinting, and extra easy unit testing of process logic. The result’s cleaner, extra maintainable code that higher represents the precise knowledge stream of your pipelines. Groups adopting this sample usually discover their DAGs change into extra readable and less complicated to keep up over time, particularly as workflows develop in complexity.
DAG versioning
Amazon MWAA with Airflow 3 contains primary DAG versioning capabilities that come by default with Airflow 3. Every time a DAG is modified and deployed, Airflow serializes and shops the DAG definition to protect historical past. This computerized model monitoring minimizes the necessity for guide record-keeping and ensures each modification is documented.
By way of the Airflow UI, groups can entry and evaluate the historical past of their DAGs. This visible illustration reveals model numbers (v1, v2, v3, and so forth.) and helps groups perceive how their workflows have advanced over time.
The DAG versioning supported in Amazon MWAA supplies the potential to see totally different DAG variations that have been run within the Airflow UI, providing improved workflow visibility and enhanced collaboration for knowledge engineering groups managing advanced, evolving knowledge pipelines.
Python 3.12 help
Amazon MWAA provides help for Python 3.12, bringing the newest language capabilities to workflow growth. This improve supplies entry to the newest Python language enhancements, efficiency enhancements, and library updates, holding your knowledge pipelines trendy and environment friendly.
Options not at the moment supported in Amazon MWAA
Though we’re launching many of the Airflow 3 options on Amazon MWAA on this launch, some options usually are not supported at the moment:
- DAG versioning (AIP-63) – Superior versioning options past primary model monitoring
- Change Flask AppBuilder (AIP-79) – Full substitute capabilities
- Edge Executor and process isolations (AIP-69) – Distant execution capabilities
- Multi-language help (AIP-72) – Help for languages apart from Python
We plan to help these options in subsequent variations of Airflow on Amazon MWAA.
Conclusion
Airflow 3 on Amazon MWAA delivers enhanced workflow automation capabilities. The architectural enhancements, enhanced safety mannequin, and developer-friendly options present a strong basis for constructing extra dependable and maintainable knowledge pipelines.The introduction of Asset Watchers modifications how workflows can reply to exterior occasions, enabling really event-driven scheduling. This functionality, mixed with the brand new asset-centric workflow design, makes Airflow 3 a extra highly effective and versatile orchestration service.
The scheduler optimizations ship efficiency enhancements for process execution and workflow administration, and the improved backfill capabilities make historic knowledge processing extra environment friendly. The DAG versioning system improves workflow stability and collaboration, and Python 3.12 help retains your knowledge pipelines trendy and environment friendly.
Organizations can now make the most of these new options and enhancements in Airflow 3 on Amazon MWAA to boost their workflow orchestration capabilities. To get began, go to the Amazon MWAA product web page.
Concerning the authors
Anurag Srivastava works as a Senior Huge Knowledge Cloud Engineer at Amazon Net Providers (AWS), specializing in Amazon MWAA. He’s captivated with serving to prospects construct scalable knowledge pipelines and workflow automation options on AWS.
Kamen Sharlandjiev is a Sr. Huge Knowledge and ETL Options Architect, Amazon MWAA and AWS Glue ETL professional. He’s on a mission to make life simpler for patrons who’re going through advanced knowledge integration and orchestration challenges. His secret weapon? Totally managed AWS companies that may get the job carried out with minimal effort. Comply with Kamen on LinkedIn to maintain updated with the newest Amazon MWAA and AWS Glue options and information!
Ankit Sahu brings over 18 years of experience in constructing modern digital services. His various expertise spans product technique, go-to-market execution, and digital transformation initiatives. At the moment, Ankit serves as Senior Product Supervisor at Amazon Net Providers (AWS), the place he leads the Amazon MWAA service.
Mohammad Sabeel works as a Senior Cloud Help Engineer at Amazon Net Providers (AWS), specializing in AWS Analytics companies together with AWS Glue, Amazon MWAA, and Amazon Athena. With over 14 years of IT expertise, he’s captivated with serving to prospects construct scalable knowledge processing pipelines and optimize their analytics options on AWS.
Satya Chikkala is a Options Architect at Amazon Net Providers. Based mostly in Melbourne, Australia, he works intently with enterprise prospects to speed up their cloud journey. Past work, he’s very captivated with nature and pictures.
Sriharsh Adari is a Senior Options Architect at Amazon Net Providers (AWS), the place he helps prospects work backward from enterprise outcomes to develop modern options on AWS. Through the years, he has helped a number of prospects on knowledge system transformations throughout trade verticals. His core space of experience embody expertise technique, knowledge analytics, and knowledge science. In his spare time, he enjoys enjoying sports activities, binge-watching TV reveals, and enjoying Tabla.
