Saying OIDC Token Federation for Enhanced Delta Sharing Safety

We’re excited to introduce the Public Preview of OIDC Token Federation for Enhanced Delta Sharing Safety a serious safety and usefulness enhancement for when sharing with non-Databricks recipients. With this launch, knowledge suppliers can securely share knowledge with non-Databricks customers on any computing platform preferring to authenticate utilizing a customized Identification Supplier (IdP), equivalent to Azure Entra ID or Okta. This eliminates the necessity for static credentials, enhances safety, and allows fine-grained entry management—making certain that solely the best customers or machines have entry to shared knowledge.

5 Advantages of Safe OpenID Join Token Federation

Delta Sharing is the trade’s first open-source strategy to knowledge sharing throughout knowledge, analytics and AI. This implies you’re not locked into any particular vendor or platform. Delta Sharing from Databricks permits for 2 kinds of sharing: Sharing with Non-Databricks Recipients and Sharing with Databricks Recipients (also referred to as D2D Sharing). Examine out the weblog “How Delta Sharing Permits Safe Finish-to-Finish Collaboration” for extra particulars on each a lot of these sharing. For those who’re sharing knowledge with one other Databricks buyer into their Databricks account, you possibly can already use D2D Sharing, which gives a seamless and built-in expertise throughout the Databricks ecosystem.

Alternatively, when securely sharing with exterior customers who will not be on Databricks, Delta Sharing has at all times offered a easy and quick approach to share knowledge utilizing bearer tokens. Nonetheless, for non-Databricks customers who prioritize enhanced safety, OIDC Token Federation for Enhanced Safety presents a extra sturdy and versatile authentication mechanism. This strategy minimizes publicity dangers and ensures safe collaboration.

Key Advantages of utilizing OIDC Token Federation when sharing with non-Databricks customers on any computing platform:

1. IdP-Managed Identities: Prospects can use their present Identification Suppliers (like Entra ID or Okta) for authentication, avoiding the necessity to arrange new methods or processes.
2. Effective-Grained Consumer Entry Management: Prospects acquire exact management over who can entry knowledge, making certain solely the best individuals or methods have permissions.
3. Multi-Issue Authentication (MFA) Help: If their Identification Supplier helps Multi-Issue Authentication (MFA), it provides an additional layer of safety, making certain solely licensed customers entry shared knowledge.
4. Diminished Safety Dangers: Quick-lived tokens robotically expire, minimizing the possibility of unauthorized entry with out requiring guide intervention.
5. No Shared Secrets and techniques: Eliminates the necessity to distribute static credentials between Databricks, suppliers, and recipients.

How does the OIDC Token Federation work when sharing with non-Databricks recipients?

With OIDC Token Federation when sharing with non-Databricks recipients, every Delta Sharing recipient is configured with federation insurance policies, making certain that solely licensed customers or machines can entry shared knowledge. Here is the way it works:

1. Organising Entry for Non-Databricks Recipient

• The information supplier (Databricks consumer) configures an OIDC Token Federation coverage for the recipient, specifying their exterior IdP (e.g., Entra ID, Okta).
• The coverage defines which customers, purposes, or methods from the recipient’s id system are allowed to entry the shared knowledge.

2. Safe Authentication with Quick-Lived Tokens

1. When the recipient makes an attempt to entry shared knowledge, Delta Sharing Connector will authenticate on their behalf towards their configure IdP (e.g., Entra ID or Okta).
2. Upon a profitable authentication, id system creates a brief digital move, known as a JSON Internet Token (JWT), which incorporates details about who they’re. That is shared with Delta Sharing Connector.
3. The Delta Sharing Connector will mix the JWT token issued by the IdP alongside the Delta Sharing request and ship it to Databricks Delta Sharing Server.
4. Databricks Delta Sharing will validates the JWT towards the recipient’s coverage, and matches the principles set by the information supplier, equivalent to verifying who issued it, who it’s for, and who’s requesting entry.
5. Upon a profitable authentication, Databricks Delta Sharing server shares the requested knowledge.
6. Delta Sharing Shopper in flip returns it to recipient workflow.

This strategy eliminates the necessity for shared secrets and techniques. As a substitute, it makes use of non permanent authentication tokens that expire shortly and it permits exact management over who or what (a particular consumer or machine) can entry the information.

Three Authentication Eventualities Supported

OIDC Token Federation strategy helps each Consumer-to-Machine (U2M) and Machine-to-Machine (M2M) authentication flows, enabling a broad vary of use circumstances.

1. Consumer-to-Machine (U2M) Authentication

• A human consumer from the recipient group authenticates by way of their IdP.
• If Multi-Issue Authentication (MFA) is enabled within the recipient’s or supplier’s IdP, will probably be enforced.
• The consumer can then use instruments like Energy BI or Tableau to entry and analyze the shared knowledge simply.
• The information supplier can set guidelines to permit entry solely to particular individuals or teams, making certain tight management over who will get entry.

This demo exhibits the right way to securely share knowledge from Databricks to Energy BI with EntraID authentication

2. Machine-to-Machine (M2M) Authentication

Delta Sharing now helps two safe methods for non-Databricks recipient machines to authenticate robotically:

State of affairs 1: OAuth Shopper Credentials Grant Move

• The recipient or supplier group registers a Service Principal of their IdP. A service principal is sort of a “consumer id” for purposes or automated methods, permitting them to securely entry sources while not having a human to log in.
• No credentials are shared externally between Databricks, Supplier or Recipient, and secret administration is native—every part stays safe inside every group.
• Help for Python Delta Sharing Shopper and Spark Delta Sharing Shopper ensures that recipients can entry shared knowledge by means of scripts/automation.

This demo exhibits the right way to securely share knowledge from Databricks to Python Delta Sharing Shopper utilizing OAuth Shopper Credentials Grant

State of affairs 2: Managed Identification Authentication (Coming Quickly)

• For workloads working in cloud environments (e.g., Azure VMs), authentication happens robotically utilizing managed identities.
• No secrets and techniques or guide credential administration is required.
• Preliminary assist will give attention to Azure Compute, with potential growth to different cloud suppliers.

This demo exhibits the right way to securely share knowledge from Databricks to Python Delta Sharing Shopper utilizing Cloud supplier Managed Identification

Selecting the Identification Supplier:

Prospects can select to authenticate utilizing an Exterior Identification Supplier (recipient-managed) or an Inner Identification Supplier (provider-managed).

• Exterior Identification Supplier (Recipient-Managed): The recipient’s id system (like Entra ID or Okta) is used. The supplier units it up within the sharing coverage, so the recipient controls who from their group can entry the information.
• Inner Identification Supplier (Supplier-Managed): The supplier’s id system is used. The supplier manages authentication by including exterior recipients as company in their very own id system. This enables the supplier’s system to deal with entry on behalf of the recipient.

What’s Subsequent?

We are going to make it simpler to arrange safe knowledge sharing with pre-built templates for widespread OIDC Federation Insurance policies tailor-made for common id suppliers like Entra ID and Okta. Moreover, upcoming assist for managed id authentication will allow cloud-based workloads (e.g., Azure VMs) to authenticate while not having passwords or secrets and techniques, making certain a seamless and safe connection to Databricks Delta Sharing endpoints.

Get Began

OIDC Token Federation for Enhanced Safety when sharing with non-Databricks recipients is obtainable in Public Preview at this time to AWS, GCP and Azure clients. Learn the way Delta Sharing makes it simple for organizations to securely share knowledge with non-Databricks customers on any computing platform—with out compromising on safety.