What if we bridged the gap between data democracy and information governance by fusing the powers of Atlan, Snowflake, and Sigma?
At a Look
- As the premier funds answer provider processing over $100 billion in annual transactions, North seeks to elevate knowledge discovery and governance by leveraging its cutting-edge Snowflake- and Sigma-integrated knowledge ecosystem.
- By harnessing the power of Atlan’s cutting-edge metadata platform, North has successfully attained real-time, end-to-end visibility across its vast portfolio of 225,000 knowledge assets, thereby safeguarding sensitive information and fostering widespread adoption of self-service analytics.
- Within the span of less than 12 months, their team is anticipating $1.4 million in annual efficiency gains and a 200% surge in Sigma adoption, accompanied by a remarkable 700% increase in tagged Snowflake properties – this significant progress has effectively mitigated hundreds of thousands of dollars in potential risks.
With a legacy spanning over three decades, North has consistently proven itself to be the leading provider of fund solutions, boasting an impressive annual transaction volume exceeding $100 billion. With a diverse portfolio spanning over 20 manufacturers and subsidiaries, they cater to a wide range of fee needs, empowering both brick-and-mortar and online retailers through seamless payment gateways, mobile options, and more. Committed to serving a diverse range of sectors, from restaurants to government agencies, our aim is to become the leading provider of seamless and comprehensive financial solutions for businesses to thrive.
With an unlimited knowledge property comprising 225,000 property, totaling 41 terabytes on Snowflake, the duty of managing and governing North’s knowledge belongs to Daniel Dowdy, Vice President of Information Analytics & Governance, serving for 13 years on the firm. Daniel joined the Atlan community at Snowflake Summit 2024 to share the story of his team’s evolution, building and managing a cutting-edge knowledge graph that empowers internal knowledge seekers and fuels their core products’ success?
Daniel Dowdy, Vice President, Information Analytics & Governance
North’s flagship solutions, Payanywhere and FundHub, empower retailers by providing a comprehensive suite of hardware and software tools that streamline online payment processes, thereby alleviating the intricacies of financial transactions for merchants.
Funds Hub is a comprehensive platform that centralizes funds and operations management, streamlining functions such as transaction monitoring, reporting, stock administration, and fraud prevention into a single, intuitive hub. North relies heavily on Snowflake to deliver mission-critical, real-time transactional reporting to its clients.
“We leverage Snowflake’s capabilities to deliver near-real-time and granular, transactional reporting to our clients, utilizing its power on the backend.” “”
Earlier than Snowflake, Information Silos
Before the emergence of Snowflake, North grappled with unmonitored and ambiguous information. Data extraction tasks, such as pulling unprocessed information or generating reports, can be protracted processes that consume days or even weeks due to ETL latency and knowledge barriers dispersed across multiple servers, databases, and shared spreadsheets within the organization.
”
Daniel Dowdy, Vice President, Information Analytics & Governance
The scarcity of a unified platform for sharing sophisticated research findings hindered the dissemination of credible information about the North’s intellectual pursuits, further complicated by the lack of a clear organizational structure and decision-making framework, which made it challenging to manage and verify this knowledge once it was disseminated.
Following real-time insights + precise calculations, Accurate data now available.
North leveraged Snowflake as their enterprise knowledge lake in 2019, thereby significantly enhancing their knowledge science, analytics, and reporting capabilities. As they transitioned away from antiquated batch processing, their team exchanged laborious ETL processes for the efficiency of modern ELT solutions in near-real-time CDC scenarios.
“We transformed our situation from being stuck in an ‘information desert’, where gathering data took days,” said Daniel.
While Snowflake marked a significant milestone in knowledge accessibility and dependability, Daniel’s team remained responsible for handling straightforward inquiries related to data entry. To alleviate the burden of an ever-growing workload, the company developed Sigma, a cutting-edge business intelligence platform that democratizes access to knowledge, empowering users to seamlessly explore, analyze, and visualize data at their own pace, thereby fostering a culture of self-service and informed decision-making.
With its familiar Excel-like interface, enterprise clients quickly adopted Sigma, resulting in a meteoric rise in adoption rates, with annual user growth now surpassing 200% annually.
Daniel Dowdy, Vice President, Information Analytics & Governance
North has successfully developed a cutting-edge knowledge infrastructure, comprising robust data pipelines that effortlessly integrate and replicate information from a wide range of sources, including Oracle, MySQL, Salesforce, and Amazon S3. The wealth of knowledge is then leveraged to develop highly effective tools akin to Qlik, Snowpipe, and Kafka, which ultimately converge within the Snowflake ecosystem.
As the accumulation of knowledge expanded, the need for access escalated, transforming the challenge from mere information availability into a complex issue of advanced knowledge management and exploration.
“What ultimately became the challenge was finding effective ways to uncover the information and govern its use,” Daniel explained. “”
Sigma unlocked its first major milestone by going live in production for a leading financial services company, processing over 10 million requests per day. The success was a testament to the power of serverless architecture and event-driven design. Meanwhile, Snowflake’s data warehousing capabilities allowed it to store petabytes of data and provide real-time insights that informed critical business decisions.
By seamlessly connecting to over 225,000 knowledge properties within Snowflake and Sigma, Atlan enabled unparalleled visibility at an enormous scale in mere hours.
“After integrating with Atlan, our team was able to connect to over 225,000 properties in just a few hours, significantly faster than our previous experiences with Snowflake and Sigma.” With evident pride, Daniel exclaimed, “The outcomes are truly something to celebrate!”
The seamless connectivity revolutionized knowledge discovery at North, instantly bridging the gap between disparate departments and facilitating effortless navigation through their vast intellectual capital.
Daniel Dowdy, Vice President, Information Analytics & Governance
By building upon previously acquired understanding, Daniel’s team developed comprehensive glossaries, precise definitions, and entire phrase structures, allowing for in-depth exploration of knowledge, and leveraging lineage to effortlessly uncover upstream and downstream interdependencies.
With this pivotal milestone reached, they proceeded to deploy the solution to enterprise clients, who predominantly consume information within the Sigma platform, without needing to switch between tools.
“” Daniel shared.
Atlan’s Chrome Extension: Revolutionizing Knowledge Consumption within Sigma. Customers can now seamlessly access a comprehensive overview of their knowledge property within Sigma, including dashboard descriptions, knowledge definitions, tags, lineage, notifications, and upstream indicators of high quality, thereby eliminating the need to navigate multiple tools to complete tasks.
Accessible at a glance, Atlan’s Chrome Extension provides instant access to valuable insights, requiring no training or configuration efforts.
The initiative demanded little guidance from Daniel. As a consequence of its being a sidebar, intuitively revealing this data, we could simply integrate it.
Consumer Conundrum: Situations Demand Decisions
As access was granted to Sigma for their proprietary knowledge, Daniel’s team began refining their approach to troubleshooting, as well as crafting compelling talking points tied directly to these valuable properties. When a ticket is received, North’s engineering team promptly dedicates valuable time to explore Sigma and identify the specific property mentioned in the request. Without a direct link between tickets and these properties, decisions would typically be delayed, potentially leading to duplicated effort and inefficiency.
Daniel acknowledged that the process had been inherently inefficient. “Even identical engineers working on the same project may not grasp its entirety due to numerous tickets submitted by various individuals.”
North’s knowledge shoppers can accelerate their issue resolution process by seamlessly integrating the Atlant Chrome Extension with Jira and Slack. This integration automatically generates a link to specific issues, enabling Information Engineers to track and respond promptly to points as they arise, streamlining overall software maintenance and improvement efforts.
In Sigma, the integration seamlessly delivers a comprehensive historical record of all Jira tickets and Slack conversations related to specific intellectual property assets, empowering knowledge shoppers to make informed decisions. For engineers, it streamlines workflows by preventing duplicate tickets or multiple team members working on the same issue, thereby optimizing time efficiency through enhanced understanding of comparable past topics.
Daniel Dowdy, Vice President, Information Analytics & Governance
Introducing Snowflake Dynamic Information Masking
As a fund administrator, North manages vast amounts of sensitive financial data with precision and care. While advances in knowledge discovery and enrichment are crucial, Daniel’s team recognized that their value may be limited to a select few trusted customers without robust knowledge governance and comprehensive documentation safeguarding these assets?
To safeguard sensitive knowledge and establish robust entry controls, North’s knowledge team opted to leverage Snowflake, implementing comprehensive column-level encryption for sensitive data and policy-driven management for clients.
Daniel and his team embarked on a mission to maximize the benefits of Dynamic Information Masking by identifying specific data attributes that various insurance policies would cover. Considering the vast scope of their understanding, they recognized that a piecemeal approach to analyzing assets could be both painstakingly slow and perilously inaccurate. Seeking an automated method to identify and safeguard properties, Daniel consulted with Atlan’s experts.
How do you efficiently uncover and categorize thousands of properties? It’s actually easy,” Daniel defined.
Managing Complex Real Estate Holdings with Ease
Rather than manually identifying each property and labeling it individually, the team leveraged Atlan’s Playbooks, a set of rules-based automation tools capable of categorizing thousands of properties according to selected criteria, before automatically applying desired actions such as assigning specific tags and policies.
Daniel Dowdy, Vice President, Information Analytics & Governance
Daniel’s team streamlined the process of securing thousands of properties by developing Playbooks that enabled automated workflows to assign accurate tags to relevant assets, with the added benefit of propagating those tags upstream at the touch of a button. With Atlan’s Two-way Tag Sync, these tags serve as the catalyst for Dynamic Information Masking at North, leveraging masking protocols based on how and from whom data is queried to ensure secure access control.
North streamlined management of over 1,000 intellectual properties across a vast 225,000-asset landscape by seamlessly integrating Atlan Playbooks, Two-way Tag Sync, and Automated Lineage with Snowflake’s Dynamic Information Masking, thereby eliminating the laborious task of manually tagging and recording insurance policies.
Daniel Dowdy, Vice President, Information Analytics & Governance
Through this innovative approach, Daniel’s team achieved a staggering 700% increase in tagged properties, significantly enhancing their safety and compliance, while clearing the path for a substantial expansion in the number of colleagues entrusted to utilize knowledge effectively.
Daniel revealed that they’ve achieved a remarkable 700%-plus boost in both velocity and accuracy for their asset tagging process after implementing Atlan, citing tangible results. Within a year, we successfully integrated over 2,000 governance properties, thanks to the enhanced capabilities of finding knowledge more efficiently, applying tags, and seamlessly syncing this information back into Snowflake. ”
Assets worth hundreds of thousands of dollars were accumulated over several months.
Within a year of implementing Atlan, Daniel’s team revolutionized how Best Way North utilizes and manages its data, unlocking millions in value through efficiency enhancements, cost savings, and improved risk management strategies.
Through expertly integrating critical information on Snowflake and Sigma directly into an intuitive database within Atlan, North’s technology and enterprise clients anticipate a significant reduction in costs, yielding estimated financial savings of $1.4 million annually through the accelerated discovery of insights and streamlined comprehension of data lineage.
“Instead of devoting hours to troubleshooting, let’s focus on adapting to the changing landscape, much like navigating the internet in its early days,” Daniel explained.
With the intuitive analytics capabilities of Atlan, Daniel’s team quickly identifies opportunities for significant cost reduction, resulting in an estimated annual savings of over $20,000 by depreciating more than 10,000 idle assets and eliminating inefficient queries.
The financial windfall came unexpectedly and effortlessly, with little effort required from our end.
With financial savings and efficiency features taking a backseat, organisations processing sensitive financial data must prioritise crucial risk mitigation strategies to ensure the security of their operations. As North’s tagged assets surged by 700% in a remarkably short and substantial manner, their governance and safety underwent a transformative improvement within a day, yielding a significant reduction of hundreds of thousands of dollars in potential risk.
Daniel Dowdy, Vice President, Information Analytics & Governance
The 200% enhancement of self-service analytics has opened doors for significant advancements, leaving ample scope for further growth.
As these advancements are truly transformative, their potential impact on knowledge discovery, bolstered by a deep understanding of their knowledge properties, and their role in Information Governance, where they drive hundreds of thousands of dollars in threat mitigation, it is the synergy between these applications that will revolutionize how North leverages knowledge.
As North propels itself forward, its mastery of entry management and commitment to safety across vast intellectual properties yield a profound evolution in defining a trustworthy custodian of knowledge for Daniel’s team.
Daniel attributed Information Governance’s benefits to the confidence it instilled in sharing additional knowledge within Sigma, ensuring that sensitive information was safely and securely accessed. With precise tagging and rigorous entry controls in place, he was certain that the most critical data was well-protected, thereby amplifying the already substantial value derived from integrating Atlan, Snowflake, and Sigma.
Daniel Dowdy, Vice President, Information Analytics & Governance
Just a few years after embracing Snowflake as the catalyst for their Fashionable Information Stack, and less than a year following the adoption of Atlas, North’s intellectual property has undergone a remarkable transformation – from the barren “Information Desert” Daniel once described to a thriving “Oasis”.
Since knowledge doesn’t depart from under our umbrella, we don’t have any additional straightforward information to share,” Daniel explained. “”
Notwithstanding the remarkable achievements of North’s data team in swiftly building self-service analytics and Information Governance capabilities from the ground up, Daniel still perceives significant untapped potential.
In the months and years ahead, their team is eager to leverage Atlan AI to translate complex queries into natural language, thereby accelerating the enhancement of their proprietary knowledge assets. As the number of Information Stewards grows, they are empowered to independently disseminate and document data, leading to an increasingly informed and self-assured community of business users.
It is the synergy between the right knowledge, a skilled team, and effective methodology that truly underpins success, as Daniel underscored in his retrospective account of the Atlan experience.
Daniel Dowdy, Vice President, Information Analytics & Governance
