Big Data

Breaking down information silos: Volkswagen’s method with Amazon DataZone

October 9, 2025

Over time, organizations have invested in constructing purpose-built cloud-based information warehouses which are siloed from each other. One of many main challenges these organizations encounter at present is enabling cross-organization discovery and entry to information throughout these siloed information warehouses constructed utilizing totally different know-how stacks. The information mesh sample addresses these points, based in 4 rules: domain-oriented decentralized information possession and structure, treating information as a product, offering self-serve information infrastructure as a platform, and implementing federated governance. The info mesh sample helps organizations mimic their organizational construction into information domains and makes it attainable to share the information throughout the group and past to enhance their enterprise fashions.

In 2019, Volkswagen AG and Amazon Net Providers (AWS) began their collaboration to co-develop the Digital Manufacturing Platform (DPP), with the aim of enhancing manufacturing and logistics effectivity by 30% whereas lowering manufacturing prices by the identical margin. The DPP was developed to streamline entry to information from store flooring gadgets and manufacturing programs by dealing with integrations and offering a variety of standardized interfaces. Nonetheless, as functions and use circumstances advanced on the platform, a big problem emerged: the power to share information throughout functions saved in remoted information warehouses (inside Amazon Redshift in remoted AWS accounts designated for particular use circumstances), with out the necessity to consolidate information right into a central information warehouse. One other problem was discovering all of the out there information saved throughout a number of information warehouses and facilitating a workflow to request entry to information throughout enterprise domains inside every plant. The frequent methodology used was largely guide, counting on emails and basic communication (via tickets and emails). The guide method not solely elevated the overhead but additionally diversified from one use case to a different by way of information governance.

On this publish, we introduce Amazon DataZone and discover how Volkswagen used Amazon DataZone to construct their information mesh, deal with the challenges encountered, and break the information silos. A key side of the answer was enabling information suppliers to robotically publish their information merchandise to Amazon DataZone, serving as a central information mesh for enhanced information discoverability. Moreover, we offer code to information you thru the deployment and implementation course of.

Introduction to Amazon DataZone

Amazon DataZone is a knowledge administration service that makes it sooner and simple to catalog, uncover, share, and govern information saved throughout AWS, on-premises, and third-party sources. Key options of Amazon DataZone embrace the enterprise information catalog, with which customers can seek for revealed information, request entry, and begin engaged on information in days as an alternative of weeks. As well as, the service facilitates collaboration throughout groups and helps them handle and monitor information belongings throughout totally different organizational items. The service additionally contains the Amazon DataZone portal, which affords a personalised analytics expertise for information belongings via a web-based utility or API. Lastly, Amazon DataZone affords ruled information sharing, which makes certain the appropriate information is accessed by the appropriate person for the appropriate function with a ruled workflow.

Resolution overview

The next structure diagram represents a high-level design that’s constructed on prime of the information mesh sample. It separates supply programs, information area producers (information publishers), information area subscribers (information customers), and central governance to focus on the important thing points. This information mesh structure is specifically tailor-made for cross-AWS account utilization. The target of this method is to create a basis for constructing information governance on a scale, supporting the aims of information producers and customers with sturdy and constant governance.

This structure permits for the mixing of a number of information warehouses right into a centralized governance account that shops all of the metadata from every setting.

An information area producer makes use of Amazon Redshift as their analytical information warehouse to retailer, course of, and handle structured and semi-structured information. The info area producers load information into their respective Amazon Redshift clusters via extract, rework, and cargo (ETL) pipelines they handle, personal, and function. The producers preserve management over their information via Amazon Redshift safety features, together with column-level entry controls and dynamic information masking, supporting information governance on the supply. An information area producer makes use of Amazon Redshift ETL and Amazon Redshift Spectrum to course of and rework uncooked information into consumable information merchandise. The info merchandise might be Amazon Redshift tables, views, or materialized views.

Knowledge area producers expose datasets to the remainder of the group by registering them to Amazon DataZone service, which acts as a central information catalog. They’ll select what information belongings to share, for a way lengthy, and the way customers can work together with these. They’re additionally liable for sustaining the information and ensuring it’s correct and present.

The info belongings from the producers are then revealed utilizing the information supply run to Amazon DataZone within the central governance account. This course of populates the technical metadata into the enterprise information catalog for every information asset. The enterprise metadata might be added by enterprise customers (information analysts) to offer enterprise context, tags, and information classification for the datasets. This method gives the required options to permit producers to create catalog entries with Amazon Redshift from all their information warehouses inbuilt with Redshift clusters. As well as, the central information governance account is used to share datasets securely between producers and customers. It’s essential to notice that sharing is finished via metadata linking alone. No information (besides logs) exists within the governance account. The info isn’t copied to the central account; only a reference to the information is used, in order that the information possession stays with the producer.

Amazon DataZone gives a streamlined solution to seek for information. The Amazon DataZone information portal gives a personalised view for customers to find and search information belongings. An Amazon DataZone person (shopper) with permissions to entry the information portal can seek for belongings and submit requests for subscription of information belongings utilizing a web-based utility. An approver can then approve or reject the subscription request.

When a knowledge area shopper has entry to an asset within the catalog, they will devour it (question and analyze) utilizing the Amazon Redshift question editor. Every shopper runs their very own workload primarily based on their use case. On this manner, the staff can select the instruments for the job to carry out analytics and machine studying actions in its AWS shopper setting.

Publishing and registering information belongings to Amazon DataZone

To publish a knowledge asset from the producer account, every asset have to be registered in Amazon DataZone for shopper subscription. For extra data, confer with Create and run an Amazon DataZone information supply for Amazon Redshift. Within the absence of an automatic registration course of, required duties have to be accomplished manually for every information asset.

Utilizing the automated registration workflow, the guide steps might be automated for the Amazon Redshift information asset (Redshift desk or view) that must be revealed in an Amazon DataZone area or when there’s a schema change in an already revealed information asset.

The next structure diagram represents how information belongings from Amazon Redshift information warehouses have been robotically revealed to the information mesh created with Amazon DataZone.

The method consists of the next steps:

Within the producer account (Account B), the information to be shared resides in a Redshift cluster.
The producer account (Account B) makes use of a mechanism to set off the dataset registration AWS Lambda perform with a selected payload containing the knowledge and title of the database, schema, desk, or view that has a change in metadata.
The Lambda perform performs the steps to robotically register and publish the dataset in Amazon DataZone:
1. Get the Amazon Redshift clusterName, dbName, schemas, and tables from the JSON payload, which is used because the occasion to set off the Lambda perform.
2. Get the Amazon DataZone information warehouse blueprint ID.
3. Allow the blueprint within the information producer account.
4. Establish the Amazon DataZone Area ID and venture ID for the producer through assuming function in Amazon DataZone account (Account A).
5. Examine if an setting already exists within the venture. If not, create an setting.
6. Create a brand new Redshift information supply by offering the proper Redshift database data within the newly created setting.
7. Provoke a knowledge supply run request within the information supply to make the Redshift tables or views out there in Amazon DataZone.
8. Publish the tables or views within the Amazon DataZone catalog.

Stipulations

The next stipulations are required earlier than beginning:

Two AWS accounts to implement the answer have been described on this publish. Nonetheless, you too can use Amazon DataZone to publish information inside a single account or throughout a number of accounts.
- Amazon DataZone account (Account A) – That is the central information governance account, which may have the Amazon DataZone area and venture.
- Knowledge area producer account (Account B) – This account acts as the information area producer. It has been added as an related account to Account A.

Stipulations in information area producer account (Account B)

As a part of this publish, we wish to publish belongings and subscribe to belongings from a Redshift cluster that already exists. Full the next prerequisite steps to arrange Account B:

Arrange the Redshift cluster, together with database, schema, tables, and views (elective). The node sort have to be from the RA3 household. For extra data, see Amazon Redshift provisioned clusters.
Create a superuser in Amazon Redshift for Amazon DataZone. For the Redshift cluster, the database person you present in AWS Secrets and techniques Supervisor will need to have superuser permissions. For reference please see the word part on this QuickStart information with pattern Amazon Redshift information
Retailer the person’s credentials in Secrets and techniques Supervisor. Choose the credential sort, enter the credential values, and select the AWS Key Administration Service (AWS KMS) key with which to encrypt the key.
Add the tags to the Secret Supervisor secret to permit Amazon DataZone to search out this secret and restrict the entry to a selected Amazon DataZone area and Amazon DataZone venture. The Redshift cluster Amazon Useful resource Identify (ARN) have to be added as a tag so it may be utilized by Amazon Redshift as a sound credential. For reference please see the word part on this QuickStart information with pattern Amazon Redshift information

Add an Amazon DataZone provisioning IAM function and Amazon Redshift handle entry IAM function within the secret’s useful resource coverage. The AWS Id and Entry Administration (IAM) roles are created as a part of the AWS Cloud Improvement Equipment (AWS CDK) deployment (mentioned later on this publish). The next code reveals an instance of the Secrets and techniques Supervisor secret’s useful resource coverage. Retailer the key ARN in an AWS Methods Supervisor parameter.

{   "Model" : "2012-10-17",   "Assertion" : [ {     "Effect" : "Allow",     "Principal" : "*",     "Action" : "secretsmanager:GetSecretValue",     "Resource" : "*",     "Condition" : {       "ArnEquals" : {         "aws:PrincipalArn" : [            "arn:aws:iam::>:role/DzRedshiftAccess->->",           "arn:aws:iam::>:role/DataZoneProvisioning->"         ]       }     }   } ] }

In case your secret is encrypted with a customized KMS key, append the important thing coverage with the next assertion and add a tag to the important thing: AmazonDatazoneEnvironment = All. You possibly can skip this step should you’re utilizing an AWS managed KMS key.

{     "Impact": "Permit",     "Principal": {         "Service": "logs.>.amazonaws.com",         "AWS": "arn:aws:iam::>:root"     },     "Motion": [         "kms:Decrypt",         "kms:Encrypt",         "kms:GenerateDataKey*",         "kms:ReEncrypt*"     ],     "Useful resource": "*"  },  {     "Sid": "AllowDatazoneRoles-DEV",     "Impact": "Permit",     "Principal": {         "AWS": "*"     },     "Motion": [         "kms:Decrypt",         "kms:Describe*",         "kms:Get*",         "kms:Encrypt",         "kms:GenerateDataKey",         "kms:ReEncrypt*",         "kms:CreateGrant"     ],     "Useful resource": "*",     "Situation": {         "StringLike": {             "aws:PrincipalArn": [                 "arn:aws:iam::>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",                 "arn:aws:iam::>:role/datazone_*",                 "arn:aws:iam::>:role/>",                 "arn:aws:iam::>:role/service-role/AmazonDataZoneRedshiftAccess->-*"             ]          }      }  }

Place a mechanism to generate the next payload to set off the dataset registration Lambda perform. The payload should include the related Redshift database, schema, and desk or view that you just wish to publish within the Amazon DataZone area. The next instance code assumes you may have three databases in your Redshift cluster and inside these databases you may have totally different schemas, tables, and views. You need to regulate the payload primarily based in your use case.

{     "supply": "redshift-user-initiated",     "detail-type": "Amazon Redshift dataset registration in Amazon DataZone",     "datasets": [         {             "clusterName": ">",             "dbName":">",             "schemas": [                 {                     "schemaName":">",                     "addAllTables":false,                     "addAllViews":false,                     "tables":[                         ">",                         ">"                     ],                     "views":[                         ">"                     ]                 }             ]         },         {             "clusterName": ">",             "dbName":">",             "schemas": [                 {                     "schemaName":">",                     "addAllTables":true,                     "addAllViews":true,                     "tables":[],                     "views":[]                 }             ]         },         {             "clusterName": ">",             "dbName":">",             "schemas": [                 {                     "schemaName":">",                     "addAllTables":true,                     "addAllViews":false,                     "tables":[],                     "views":[                         ">"                     ]                 }             ]         }     ] }

Stipulations in Amazon DataZone account (Account A)

Full the next steps to arrange your Amazon DataZone account (Account A):

Register to Account A and ensure you have already deployed an Amazon DataZone area and a venture inside that area. Consult with Create Amazon DataZone domains for directions to create a website.
In case your Amazon DataZone area is encrypted with a KMS key, add the information area account (Account B) to the KMS key coverage with the next actions:
```
"Motion": [     "kms:Encrypt",     "kms:Decrypt",     "kms:ReEncrypt*",     "kms:GenerateDataKey*",     "kms:DescribeKey" ]
```

Create an IAM function that’s assumable by Account B and ensure the function has a following coverage hooked up and is a member (as contributor) of your Amazon DataZone venture. For this publish, we name the function dz-assumable-env-dataset-registration-role. By including this function, you possibly can efficiently run the registration Lambda perform.

Within the following coverage, present the AWS Area and account ID akin to the place your Amazon DataZone area is created, and the KMS key ARN used to encrypt the area:

  {     "Model": "2012-10-17",     "Assertion": [         {             "Action": [                 "datazone:CreateDataSource",                 "datazone:CreateEnvironment",                 "datazone:CreateEnvironmentProfile",                 "datazone:GetDataSource",                 "datazone:GetDataSourceRun",                 "datazone:GetEnvironment",                 "datazone:GetEnvironmentProfile",                 "datazone:GetIamPortalLoginUrl",                 "datazone:ListDataSources",                 "datazone:ListDomains",                 "datazone:ListEnvironmentProfiles",                 "datazone:ListEnvironments",                 "datazone:ListProjectMemberships",                 "datazone:ListProjects",                 "datazone:StartDataSourceRun",                 "datazone:UpdateDataSource",                 "datazone:SearchUserProfiles"             ],             "Useful resource": "*",             "Impact": "Permit"         },         {             "Motion": [                 "kms:Decrypt",                 "kms:DescribeKey",                 "kms:GenerateDataKey"             ],             "Useful resource": "arn:aws:kms:>:> }:key/${DataZonekmsKey}",             "Impact": "Permit"         }     ] }

Add Account B within the belief relationship of this function with the next belief relationship:

{     "Model": "2012-10-17",     "Assertion": [         {             "Effect": "Allow",             "Principal": {                 "AWS": [                     "arn:aws:iam::>:root",                     "arn:aws:iam::>:root",                 ]             },             "Motion": "sts:AssumeRole"         }     ] }

Add the function as a member of the Amazon DataZone venture through which you wish to register your information sources. For extra data, see Add members to a venture.

Further instruments

The next instruments are wanted to deploy the answer utilizing the AWS CDK:

Deploy the answer

After you full the stipulations, use the AWS CDK stack supplied on the GitHub repo to deploy the answer for automated registration of information belongings into the Amazon DataZone area. Full the next steps:

Clone the repository from GitHub to your most well-liked built-in improvement setting (IDE) utilizing the next instructions:

git clone https://github.com/aws-samples/sample-how-to-automate-amazon-redshift-cluster-data-asset-publish-to-amazon-datazone

$ cd sample-how-to-automate-amazon-redshift-cluster-data-asset-publish-to-amazon-datazone

On the base of the repository folder, run the next instructions to construct and deploy assets to AWS:
Register to Account B (the information area producer account) utilizing the AWS CLI together with your profile title.
Be sure you have configured the Area in your credential’s configuration file.
Bootstrap the AWS CDK setting with the next instructions on the base of the repository folder. Present the profile title of your deployment account (Account B). Bootstrapping is a one-time exercise and isn’t wanted in case your AWS account is already bootstrapped.
Exchange the placeholder parameters (marked with the suffix _PLACEHOLDER) within the file config/DataZoneConfig.ts:
1. Amazon DataZone area and venture title of your Amazon DataZone occasion. Ensure all names are in lowercase.
2. The AWS account ID of the Amazon DataZone account (Account A).
3. The assumable IAM function from the stipulations.
4. The AWS Methods Supervisor parameter title containing the Secrets and techniques Supervisor secret ARN of the Amazon Redshift credentials.
Use the next command within the base folder to deploy the AWS CDK answer. Throughout deployment, enter y if you wish to deploy the adjustments for some stacks while you see the immediate Do you want to deploy these adjustments (y/n)?
After the deployment is full, sign up to Account B and open the AWS CloudFormation console to confirm that the infrastructure was deployed.

Take a look at automated information registration to Amazon DataZone

Full the next steps to check the answer:

Register to Account B (producer account).
On the Lambda console, open the datazone-redshift-dataset-registration perform.
Below TEST EVENTS, select Create new take a look at occasion.

For Occasion title, enter Redshift, and for Occasion JSON, enter the next JSON construction (change the cluster, schema, database, and desk names in response to your setting):

{   "supply": "redshift-user-initiated",   "detail-type": "Amazon Redshift dataset registration in Amazon DataZone",   "datasets": [     {       "clusterName": "YOUR_REDSHIFT_CLUSTER_NAME",       "dbName": "DATABASE_NAME",       "schemas": [         {           "schemaName": "SCHEMA_NAME_1",           "addAllTables": false,           "addAllViews": false,           "tables": [             "TABLE_NAME"           ],           "views": []         },         {           "schemaName": "SCHEMA_NAME_2",           "addAllTables": false,           "addAllViews": false,           "tables": [],           "views": [             "VIEW_NAME"           ]         }       ]     }   ] }

Select Save.
Select Invoke.
Open the Amazon DataZone console in Account A the place you deployed the assets.
Select Domains within the navigation pane, then open your area.
On the area particulars web page, find the Amazon DataZone information portal URL within the Abstract part. Select the hyperlink to the information portal.
For extra particulars about accessing Amazon DataZone, confer with How can I entry Amazon DataZone?
Within the information portal, open your venture and select the Knowledge tab.
Within the navigation pane, select Knowledge sources and discover the newly created information supply for Amazon Redshift.
Confirm that the information supply has been efficiently revealed.

After the information sources are revealed, customers can uncover the revealed information and submit a subscription request. The info producer can approve or reject requests. Upon approval, customers can devour the information by querying the information within the Amazon Redshift question editor. The next screenshot illustrates information discovery within the Amazon DataZone information portal.

Clear up

Full the next steps to scrub up the assets deployed via the AWS CDK:

Register to Account B, go to the Amazon DataZone area portal, and verify there isn’t any subscription in your revealed information asset. If there’s a subscription, both ask the subscriber to unsubscribe or revoke the subscription request.
Delete the revealed information belongings that had been created within the Amazon DataZone venture by the dataset registration Lambda perform.
Delete the remaining assets created utilizing the next command within the base folder:
```
npm run cdk destroy –all
```

Conclusion

Amazon DataZone affords a seamless integration with AWS providers, offering a strong answer for organizations like Volkswagen to interrupt down their information silos and implement efficient information mesh architectures via an easy implementation highlighted on this publish. Through the use of Amazon DataZone, Volkswagen addressed its speedy information sharing hurdles and laid the groundwork for a extra agile, data-driven future in automotive manufacturing. The automated information publishing from numerous warehouses, coupled with standardized governance workflows, has considerably lowered the guide overhead that after slowed down Volkswagen’s information engineering groups. Now, as an alternative of navigating a labyrinth of emails, tickets, and communication, Volkswagen’s information engineers and information scientists can rapidly uncover and entry the information they want, all whereas sustaining their safety and compliance requirements.

Through the use of Amazon DataZone, organizations can carry their remoted information collectively in ways in which make it easier for groups to collaborate whereas sustaining safety and compliance at scale. This method not solely addresses present information governance challenges but additionally creates a extremely scalable basis for future data-driven improvements. For steerage on establishing your group’s information mesh with Amazon DataZone, contact your AWS staff at present.