Thursday, December 5, 2024

Scaling Our B2B Software-as-a-Service (SaaS) Gross Margin Optimization Platform Using Rockset?

Trendy Snack-Sized Gross sales Coaching

Across industries, we deliver personalized sales enablement solutions via the cloud, empowering professionals to optimize their performance and drive business growth. Our comprehensive SaaS platform revolutionizes the hiring and onboarding process for new sales recruits by prioritizing personalized coaching and ensuring long-term employee retention.

High employee turnover rates among commission-based sales teams can have devastating financial consequences. Regardless, it can likely be mitigated through targeted guidance, dispensed consistently in manageable chunks. By designing personalized curricula for each new hire’s unique needs and learning styles, we enhance learner motivation and streamline training to accelerate their productivity and success on the job.

Does such real-time personalization necessitate a robust information infrastructure capable of instantaneously processing massive amounts of personal data? As our prospect base and information volumes expanded exponentially, our once-robust information infrastructure was pushed to the breaking point.

Only after discovering the real-time analytics database, known as [insert name], were we finally able to combine hundreds of thousands of event occurrences under one second, empowering our prospects to work with precise time-stamped information – not outdated data too stale to provide effective support for sales coaching.

Our enterprise requires scalability, concurrency, and low operational complexity.

Conveyed through a suite of rules-based constructs, ConveyYour offers bite-sized, interactive training sessions and assessments for new sales recruits via text messaging, empowering clients to monitor their progress in real-time using our intuitive dashboard.

It’s clear that the coaches have meticulously crafted their strategy, taking it right down to the final 15 seconds of play. The system effortlessly tracked students’ performance on the latest quiz, automatically adjusting class assignments as needed to ensure a tailored learning experience for each student.

More than 100,000 sales representatives have undergone training via ConveYour. Our innovative microlearning approach banishes trainee apathy, significantly enhances learning results, and dramatically minimizes employee turnover. For direct sales-driven companies that constantly recruit new representatives, often recent graduates or newcomers to sales, these wins are crucially essential.

Situational awareness has always been our top priority. We dispatch numerous millions of written communications to our sales teams annually. We’re actively tracking every interaction that new recruits have with our platform, going beyond just monitoring their gross sales performance.

Each year, this buyer employs approximately 8,000 sales representatives on a gross scale. Recently, approximately half of the users participated in a compliance training course delivered and monitored through ConveYou. As individuals progress through the entire spectrum of 55 classes, tracking their development yields an astonishing 50,000 distinct data points. By multiplying this figure by 4,000 repetitions, one can estimate approximately 8 million pieces of occasion-related data. Considerably fewer programs exist for every buyer.

To provide real-time insights at the fingertips of gross sales managers, we had to execute analytics runs in batches and store the results for later access. Managing the diverse array of caches proved to be an arduously challenging task. Inevitably, some cached data would become stale, leading to outdated results. Will this potentially lead to unhappy phone calls from our sales managers, complaining about the inaccurate compliance status of their representatives?

As our prospects expanded, we found ourselves craving greater scalability. What a fantastic perk to enjoy! Despite this, it was a significant disadvantage nonetheless.

On various situations, caching would hardly make a difference. We sought to achieve high concurrency and instantaneous query results. As an example, our team designed a customer relationship management (CRM) dashboard that provided real-time aggregated performance metrics for over 7,000 sales representatives, enabling data-driven insights and informed decision-making. This dashboard served numerous central managers, who struggled to wait for data to trickle in through weekly and daily reports that were too slow. As the sheer volume of data and diverse needs of supervisors escalated, the dashboard’s performance noticeably lagged behind.

Deploying supplementary server infrastructure potentially could have facilitated. While our usage may exhibit a pronounced seasonal pattern, with peak activity in the fall as companies onboard newly graduated talent, it experiences a significant downturn during other periods of the year. Deploying perpetual infrastructure to support unpredictable surges in demand would have been a costly and inefficient endeavor. Wouldn’t it have been ideal to have a scalable information platform, capable of growing or shrinking according to our needs?

The closing situation is a benchmark for measuring our success. Construction firm Conveyors employs a team of just five skilled builders. That’s a deliberate alternative. Can we reasonably maintain a small, agile, and productive staff? To unlock their full potential, we had to transition to top-tier SaaS tools, which was a critical hurdle for us to overcome.

Technical Challenges

Our reliable data infrastructure is built around a secure on-premises MongoDB database, which processes and stores all individual transactional data. Connected via an ETL pipeline is a MySQL database residing within Google Cloud, providing seamless access to both massive, long-running queries and lightning-fast ad-hoc queries leveraging smaller datasets?

The databases were clearly underperforming. Our “Reside” CRM dashboard experienced a significant delay, often taking up to six seconds to render results, and at times crashing entirely. The root causes were complex and multifaceted, contributing to this outcome. The sheer volume of data we collected necessitated extensive research, coinciding with sudden surges in simultaneous users, much like the morning and lunchtime rushes on dashboard checks by managers.

Despite this, the primary objective remained that MySQL should not be optimized for high-speed analytics. Without well-built indexes and optimised SQL queries, a MySQL query’s performance is likely to suffer significantly. If left unchecked, this issue could potentially compromise the quality of subsequent interactions with various stakeholders, compromising the overall effectiveness of your business.

With my team dedicating an average of 10 hours each week to troubleshooting, optimizing, and resolving SQL query and index issues solely to prevent database crashes, it was clear that we needed a more efficient solution.

As the frequency of new queries hitting MySQL increased, my anxiety levels skyrocketed to alarming heights.

Drawbacks of Various Options

We explored a multitude of possibilities. While considering a scaling solution for our MongoDB instance, we initially explored the idea of adding more slave nodes; nonetheless, after careful evaluation, we concluded that simply throwing additional resources at the problem would not address the underlying issue.

While exploring Snowflake’s approach, we found certain aspects of their methodology to be particularly compelling. Despite notable progress, a significant hurdle remained: the inability to seamlessly ingest real-time data. We simply couldn’t justify dedicating an hour to learn how to transfer data from Amazon S3 to Snowflake.

Despite exploring ClickHouse, we found that the numerous compromises required, especially regarding storage, outweighed its benefits. As a dedicated store of data, ClickHouse permanently and irreversibly records information. Erasing or updating existing content gradually transforms into a laborious process. And as professionals in our field, we are well aware that regularly backing up data and disconnecting unnecessary connections is crucial to prevent potential losses or security breaches. Once we’ve done so, there’s no need to conduct additional reviews, with these contacts remaining visible notwithstanding. Without real-time capabilities to ingest, process, and refresh data instantly, analytics are essentially outdated and lose their value.

We also experimented with Amazon Redshift, but ultimately rejected it due to its limitations with smaller datasets and the excessive manual effort required, which proved unsustainable in our workflow.

Scaling with Rockset

Through my exploration, I learned about Rockset. Rockset seamlessly combines the benefits of relational databases and NoSQL data stores. While capable of rapid data writing akin to a MongoDB or other transactional databases, it can also execute complex queries with remarkable speed.

We successfully deployed Rockset in December 2021. It took only one week. While maintaining MongoDB as our primary document-based database, we began real-time data streaming to both Rockset and MySQL instances, leveraging their capabilities to efficiently handle query requests.

Our proficiency in utilizing Rockset has been nothing short of remarkable. The speed at which an organization processes and incorporates new information is critical to staying ahead of the curve. Due to the integration of advanced technology and streamlined processes, updating and backfilling information has become remarkably swift. The constant need to correct and revise data in real-time causes immense frustration for me. Since the contact is about to be eliminated and you’re performing a JOIN immediately afterwards, there’s no requirement for that contact to appear in subsequent reviews?

Could also have a profound impact. With Rockset’s autonomous compute and storage capabilities, our team of small staff members enjoys a significantly reduced IT workload. With no database maintenance required, you enjoy complete peace of mind.

With Rockset’s intuitive user interface and built-in SQL assistance, my builders are remarkably efficient in their work. Convergence of indexing and computerized question optimization eliminates the need for engineers to dedicate time to optimizing question effectiveness. Questions run quickly out of the field. Our average response time has plummeted from a sluggish six seconds to an impressive 300 milliseconds. Regardless of dataset size, whether small or massive – we’ve seen instances ranging from a few thousand to a staggering 15 million occurrences within our extensive collections. We have successfully minimized the incidence of question errors and timed-out queries to zero.

I don’t fear that granting access to a novice developer would inadvertently compromise our entire customer base’s data by crashing the database. In the worst-case scenario, an unpleasant query will likely consume additional RAM resources without hesitation. However it would. Nonetheless. Simply. Work. Finally, that immense burden has been lifted from my shoulders. I no longer need to serve as a database gatekeeper.

Additionally, Rockset’s real-time efficiency enables us to bypass the need for batch analytics and stale caches. With our advanced technology, we can now combine two million instances of information in under a second. Prospects gain unparalleled transparency with access to exact, time-stamped data, free from outdated interpretations.

We leverage Rockset for internal reporting, seamlessly ingesting and analyzing our digital server utilization data in collaboration with our trusted internet hosting partner, DigitalOcean. We frequently synchronize data with a Rockset collection as an employee of Cloudflare, enabling seamless reporting on pricing and network topology metrics. Using Digital Ocean’s native console is significantly less straightforward than monitoring our usage and efficiency in a simpler way.

As our proficiency in Rockset continues to flourish, we’re currently undertaking a comprehensive migration from MySQL to this high-performing solution. As older data is retroactively migrated from MySQL to Rockset, the process of transitioning all APIs and SQL queries is gradually being relocated from MySQL to Rockset.

When seeking seamless integration of real-time analytics with scalable capabilities for tech-driven companies like yours, we highly recommend exploring the features offered by Rockset.

Related Articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles