Apache Airflow 3.x on Amazon MWAA introduces architectural enhancements similar to API-based activity execution that gives enhanced safety and isolation. Different main updates embody a redesigned UI for higher consumer expertise, scheduler-based backfills for improved efficiency, and assist for Python 3.12. In contrast to in-place minor Airflow model upgrades in Amazon MWAA, upgrading to Airflow 3 from Airflow 2 requires cautious planning and execution by means of a migration strategy on account of basic breaking modifications.
This migration presents a possibility to embrace next-generation workflow orchestration capabilities whereas offering enterprise continuity. Nonetheless, it’s greater than a easy improve. Organizations migrating to Airflow 3.x on Amazon MWAA should perceive key breaking modifications, together with the elimination of direct metadata database entry from staff, deprecation of SubDAGs, modifications to default scheduling habits, and library dependency updates. This publish gives greatest practices and a streamlined strategy to efficiently navigate this vital migration, offering minimal disruption to your mission-critical information pipelines whereas maximizing the improved capabilities of Airflow 3.
Understanding the migration course of
The journey from Airflow 2.x to three.x on Amazon MWAA introduces a number of basic modifications that organizations should perceive earlier than starting their migration. These modifications have an effect on core workflow operations and require cautious planning to realize a easy transition.
You ought to be conscious of the next breaking modifications:
- Removing of direct database entry – A vital change in Airflow 3 is the elimination of direct metadata database entry from employee nodes. Duties and customized operators should now talk by means of the REST API as a substitute of direct database connections. This architectural change impacts code that beforehand accessed the metadata database immediately by means of SQLAlchemy connections, requiring refactoring of current DAGs and customized operators.
- SubDAG deprecation – Airflow 3 removes the SubDAG assemble in favor of TaskGroups, Belongings, and Information Conscious Scheduling. Organizations should refactor current SubDAGs to one of many beforehand talked about constructs.
- Scheduling habits modifications – Two notable modifications to default scheduling choices require an impression evaluation:
- The default values for catchup_by_default and create_cron_data_intervals modified to False. This transformation impacts DAGs that don’t explicitly set these choices.
- Airflow 3 removes a number of context variables, similar to execution_date, tomorrow_ds, yesterday_ds, prev_ds, and next_ds. You should change these variables with presently supported context variables.
- Library and dependency modifications – A major variety of libraries change in Airflow 3.x, requiring DAG code refactoring. Many beforehand included supplier packages may want express addition to the
necessities.txtfile. - REST API modifications – The REST API path modifications from /api/v1 to /api/v2, affecting exterior integrations. For extra details about utilizing the Airflow REST API, see Creating an online server session token and calling the Apache Airflow REST API.
- Authentication system – Though Airflow 3.0.1 and later variations default to SimpleAuthManager as a substitute of Flask-AppBuilder, Amazon MWAA will proceed utilizing Flask-AppBuilder for Airflow 3.x. This implies prospects on Amazon MWAA is not going to see any authentication modifications.
The migration requires creating a brand new surroundings reasonably than performing an in-place improve. Though this strategy calls for extra planning and sources, it gives the benefit of sustaining your current surroundings as a fallback possibility throughout the transition, facilitating enterprise continuity all through the migration course of.
Pre-migration planning and evaluation
Profitable migration relies on thorough planning and evaluation of your present surroundings. This part establishes the inspiration for a easy transition by figuring out dependencies, configurations, and potential compatibility points. Consider your surroundings and code in opposition to the beforehand talked about breaking modifications to have a profitable migration.
Surroundings evaluation
Start by conducting an entire stock of your present Amazon MWAA surroundings. Doc all DAGs, customized operators, plugins, and dependencies, together with their particular variations and configurations. Be sure that your present surroundings is on model 2.10.x, as a result of this gives the very best compatibility path for upgrading to Amazon MWAA with Airflow 3.x.
Establish the construction of the Amazon Easy Storage Service (Amazon S3) bucket containing your DAG code, necessities file, startup script, and plugins. You’ll replicate this construction in a brand new bucket for the brand new surroundings. Creating separate buckets for every surroundings avoids conflicts and permits continued improvement with out affecting present pipelines.
Configuration documentation
Doc all customized Amazon MWAA surroundings variables, Airflow connections, and surroundings configurations. Evaluation AWS Identification and Entry Administration (IAM) sources, as a result of your new surroundings’s execution function will want similar insurance policies. IAM customers or roles accessing the Airflow UI require the CreateWebLoginToken permission for the brand new surroundings.
Pipeline dependencies
Understanding pipeline dependencies is vital for a profitable phased migration. Establish interdependencies by means of Datasets (now Belongings), SubDAGs, TriggerDagRun operators, or exterior API interactions. Develop your migration plan round these dependencies so associated DAGs can migrate on the similar time.
Think about DAG scheduling frequency when planning migration waves. DAGs with longer intervals between runs present bigger migration home windows and decrease danger of duplicate execution in contrast with often operating DAGs.
Testing technique
Create your testing technique by defining a scientific strategy to figuring out compatibility points. Use the ruff linter with the AIR30 ruleset to routinely determine code requiring updates:
Then, assessment and replace your surroundings’s necessities.txt file to verify package deal variations adjust to the up to date constraints file. Moreover, generally used Operators beforehand included within the airflow-core package deal now reside in a separate package deal and must be added to your necessities file.
Check your DAGs utilizing the Amazon MWAA Docker photographs for Airflow 3.x. These photographs make it potential to create and take a look at your necessities file, and make sure the Scheduler efficiently parses your DAGs.
Migration technique and greatest practices
A methodical migration strategy minimizes danger whereas offering clear validation checkpoints. The beneficial technique employs a phased blue/inexperienced deployment mannequin that gives dependable migrations and speedy rollback capabilities.
Phased migration strategy
The next migration phases can help you in defining your migration plan:
- Section 1: Discovery, evaluation, and planning – On this part, full your surroundings stock, dependency mapping, and breaking change evaluation. With the gathered data, develop the detailed migration plan. This plan will embody steps for updating code, updating your necessities file, making a take a look at surroundings, testing, creating the blue/inexperienced surroundings (mentioned later on this publish), and the migration steps. Planning should additionally embody the coaching, monitoring technique, rollback situations, and the rollback plan.
- Section 2: Pilot migration – The pilot migration part serves to validate your detailed migration plan in a managed surroundings with a small vary of impression. Focus the pilot on two or three non-critical DAGs with numerous traits, similar to completely different schedules and dependencies. Migrate the chosen DAGs utilizing the migration plan outlined within the earlier part. Use this part to validate your plan and monitoring instruments, and modify each based mostly on precise outcomes. Throughout the pilot, set up baseline migration metrics to assist predict the efficiency of the complete migration.
- Section 3: Wave-based manufacturing migration – After a profitable pilot, you’re prepared to start the complete wave-based migration for the remaining DAGs. Group remaining DAGs into logical waves based mostly on enterprise criticality (least vital first), technical complexity, interdependencies (migrate dependent DAGs collectively), and scheduling frequency (much less frequent DAGs present bigger migration home windows). After you outline the waves, work with stakeholders to develop the wave schedule. Embody adequate validation durations between waves to verify the wave is profitable earlier than beginning the subsequent wave. This time additionally reduces the vary of impression within the occasion of a migration subject, and gives adequate time to carry out a rollback.
- Section 4: Publish-migration assessment and decommissioning – In spite of everything waves are full, conduct a post-migration assessment to determine classes realized, optimization alternatives, and every other unresolved objects. That is additionally a great time to supply an approval on system stability. The ultimate step is decommissioning the unique Airflow 2.x surroundings. After stability is decided, based mostly on enterprise necessities and enter, decommission the unique (blue) surroundings.
Blue/inexperienced deployment technique
Implement a blue/inexperienced deployment technique for secure, reversible migration. With this technique, you’ll have two Amazon MWAA environments working throughout the migration and handle which DAGs function during which surroundings.
The blue surroundings (present Airflow 2.x) maintains manufacturing workloads throughout transition. You may implement a freeze window for DAG modifications earlier than migration to keep away from last-minute code conflicts. This surroundings serves because the speedy rollback surroundings if a problem is recognized within the new (inexperienced) surroundings.
The inexperienced surroundings (new Airflow 3.x) receives migrated DAGs in managed waves. It mirrors the networking, IAM roles, and safety configurations from the blue surroundings. Configure this surroundings with the identical choices because the blue surroundings, and create similar monitoring mechanisms so each environments might be monitored concurrently. To keep away from duplicate DAG runs, be sure that a DAG solely runs in a single surroundings. This entails pausing the DAG within the blue surroundings earlier than activating the DAG within the inexperienced surroundings.Keep the blue surroundings in heat standby mode throughout the complete migration. Doc particular rollback steps for every migration wave, and take a look at your rollback process for no less than one non-critical DAG. Moreover, outline clear standards for triggering the rollback (similar to particular failure charges or SLA violations).
Step-by-step migration course of
This part gives detailed steps for conducting the migration.
Pre-migration evaluation and preparation
Earlier than initiating the migration course of, conduct a radical evaluation of your present surroundings and develop the migration plan:
- Be sure that your present Amazon MWAA surroundings is on model 2.10.x
- Create an in depth stock of your DAGs, customized operators, and plugins together with their dependencies and variations
- Evaluation your present
necessities.txtfile to grasp package deal necessities - Doc all surroundings variables, connections, and configuration settings
- Evaluation the Apache Airflow 3.x launch notes to grasp breaking modifications
- Decide your migration success standards, rollback situations, and rollback plan
- Establish a small variety of DAGs appropriate for the pilot migration
- Develop a plan to coach, or familiarize, Amazon MWAA customers on Airflow 3
Compatibility checks
Figuring out compatibility points is vital to a profitable migration. This step helps builders concentrate on particular code that’s incompatible with Airflow 3.
Use the ruff linter with the AIR30 ruleset to routinely determine code requiring updates:
Moreover, assessment your code for cases of direct metadatabase entry.
DAG code updates
Primarily based in your findings throughout compatibility testing, replace the affected DAG code for Airflow 3.x. The ruff DAG verify utility can routinely repair frequent modifications. Use the next command to run the utility in replace mode:
Widespread modifications embody:
- Exchange direct metadata database entry with API calls:
- Exchange deprecated context variables with their trendy equivalents:
Subsequent, consider the utilization of the 2 scheduling-related default modifications. catchup_by_default is now False, that means lacking DAG runs will now not routinely backfill. If backfill is required, replace the DAG definition with catchup=True. In case your DAGs require backfill, you should think about the impression of this migration and backfilling. Since you’re migrating a DAG to a clear surroundings with no historical past, enabling backfilling will create DAG runs for all runs starting with the required start_date. Think about updating the start_date to keep away from pointless runs.
create_cron_data_intervals can be now False. With this transformation, cron expressions are evaluated as a CronTriggerTimetable assemble.
Lastly, consider the utilization of deprecated context variables for manually and Asset-triggered DAGs, then replace your code with appropriate replacements.
Updating necessities and testing
Along with potential package deal model modifications, a number of core Airflow operators beforehand included within the airflow-core package deal moved to the apache-airflow-providers-standard package deal. These modifications should be included into your necessities.txt file. Specifying, or pinning, package deal variations in your necessities file is a greatest apply and beneficial for this migration.To replace your necessities file, full the next steps:
- Obtain and configure the Amazon MWAA Docker photographs. For extra particulars, check with the GitHub repo.
- Copy the present surroundings’s
necessities.txtfile to a brand new file. - If wanted, add the apache-airflow-providers-standard package deal to the brand new necessities file.
- Obtain the suitable Airflow constraints file on your goal Airflow model to your working director. A constraints file is out there for every Airflow model and Python model mixture. The URL takes the next kind:
https://uncooked.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt - Create your versioned necessities file utilizing your un-versioned file and the constraints file. For steering on making a necessities file, see Making a
necessities.txtfile. Be sure that there are not any dependency conflicts earlier than transferring ahead. - Confirm your necessities file utilizing the Docker picture. Run the next command contained in the operating container:
Handle any set up errors by updating package deal variations.
As a greatest apply, we suggest packaging your packages right into a ZIP file for deployment in Amazon MWAA. This makes certain the identical actual packages are put in on all Airflow nodes. Discuss with Putting in Python dependencies utilizing PyPi.org Necessities File Format for detailed details about packaging dependencies.
Creating a brand new Amazon MWAA 3.x surroundings
As a result of Amazon MWAA requires a migration strategy for main model upgrades, you should create a brand new surroundings on your blue/inexperienced deployment. This publish makes use of the AWS Command Line Interface (AWS CLI) for example, you may as well use infrastructure as code (IaC).
- Create a brand new S3 bucket utilizing the identical construction as the present S3 bucket.
- Add the up to date necessities file and any plugin packages to the brand new S3 bucket.
- Generate a template on your new surroundings configuration:
- Modify the generated JSON file:
- Copy configurations out of your current surroundings.
- Replace the surroundings identify.
- Set the AirflowVersion parameter to the goal 3.x model.
- Replace the S3 bucket properties with the brand new S3 bucket identify.
- Evaluation and replace different configuration parameters as wanted.
Configure the brand new surroundings with the identical networking settings, safety teams, and IAM roles as your current surroundings. Discuss with the Amazon MWAA Person Information for these configurations.
- Create your new surroundings:
Metadata migration
Your new surroundings requires the identical variables, connections, roles, and pool configurations. Use this part as a information for migrating this data. Should you’re utilizing AWS Secrets and techniques Supervisor as your secrets and techniques backend, you don’t have to migrate any connections. Relying your surroundings’s measurement, you may migrate this metadata utilizing the Airflow UI or the Apache Airflow REST API.
- Replace any customized pool data within the new surroundings utilizing the Airflow UI.
- For environments utilizing the metadatabase as a secrets and techniques backend, migrate all connections to the brand new surroundings.
- Migrate all variables to the brand new surroundings.
- Migrate any customized Airflow roles to the brand new surroundings.
Migration execution and validation
Plan and execute the transition out of your previous surroundings to the brand new one:
- Schedule the migration throughout a interval of low workflow exercise to reduce disruption.
- Implement a freeze window for DAG modifications earlier than and throughout the migration.
- Execute the migration in phases:
- Pause DAGs within the previous surroundings. For a small variety of DAGs, you need to use the Airflow UI. For bigger teams, think about using the REST API.
- Confirm all operating duties have accomplished within the Airflow UI.
- Redirect DAG triggers and exterior integrations to the brand new surroundings.
- Copy the up to date DAGs to the brand new surroundings’s S3 bucket.
- Allow DAGs within the new surroundings. For a small variety of DAGs, you need to use the Airflow UI. For bigger teams, think about using the REST API.
- Monitor the brand new surroundings carefully throughout the preliminary operation interval:
- Look ahead to failed duties or scheduling points.
- Examine for lacking variables or connections.
- Confirm exterior system integrations are functioning appropriately.
- Monitor Amazon CloudWatch metrics to verify the surroundings is performing as anticipated.
Publish-migration validation
After the migration, completely validate the brand new surroundings:
- Confirm that every one DAGs are being scheduled appropriately based on their outlined schedules
- Examine that activity historical past and logs are accessible and full
- Check vital workflows end-to-end to verify they execute efficiently
- Validate connections to exterior programs are functioning correctly
- Monitor CloudWatch metrics for efficiency validation
Cleanup and documentation
When the migration is full and the brand new surroundings is steady, full the next steps:
- Doc the modifications made throughout the migration course of.
- Replace runbooks and operational procedures to mirror the brand new surroundings.
- After a adequate stability interval, outlined by stakeholders, decommission the previous surroundings:
- Archive backup information based on your group’s retention insurance policies.
Conclusion
The journey from Airflow 2.x to three.x on Amazon MWAA is a chance to embrace next-generation workflow orchestration capabilities whereas sustaining the reliability of your workflow operations. By following these greatest practices and sustaining a methodical strategy, you may efficiently navigate this transition whereas minimizing dangers and disruptions to your online business operations.
A profitable migration requires thorough preparation, systematic testing, and sustaining clear documentation all through the method. Though the migration strategy requires extra preliminary effort, it gives the protection and management wanted for such a major improve.
In regards to the authors
