The U.S. dairy industry is a significant and formidable force in the global market. America’s approximately 32,000 dairy farmers do more than just produce milk – they are also one of the most environmentally sustainable, generating almost 20 times the output of a typical (1,200-pound) dairy cow.
Farmers owe their success with robust herds, healthy cattle, and bountiful harvests to a combination of agricultural science and data-driven insights. American dairy farmers have historically leveraged data insights to optimize their operations, leveraging cutting-edge technology to track the genetic profiles of their herds, monitor weather and feed price forecasts, and meticulously record milk production metrics with precision.
While many industries have harnessed the power of modern data analytics, a significant proportion of farmers remain behind the curve, failing to leverage cutting-edge tools in real-time and streaming environments, thereby compromising efficiency and profitability.
To drive further growth in their dairy business, the main dairy business analysis group pondered in late 2021 that “enhanced digital connectivity” was a crucial requirement for their future development.
What does it take to achieve your shipping goals? In August 2019, a Colorado-based company initiated the launch and development of a real-time SaaS analytics platform designed to facilitate digital transformation for America’s dairy farming community.
Grabbing Information By the Horns
The amount of milk produced by a cow depends primarily on factors such as breed, nutritional intake, management practices, and individual animal characteristics. Breed-specific genetic traits influence milk production levels, with Holsteins being the most prominent milk-producing breed. A well-nourished cow, receiving adequate quality feed and nutrients, will generally produce more milk than one that is malnourished or underfed. While it’s essential to understand the underlying genetic code, namely an organism’s DNA, it’s equally crucial to consider how those genes ultimately manifest as physical characteristics – its tangible expression, if you will, or phenotype. The environment in which an individual resides is crucial – the extent to which they are nourished, whether they will encounter cold or illness, the amount of physical activity and exercise they will engage in, and many other factors.
When dairy farms were small enough for farmers to develop personal relationships with each cow, they meticulously recorded vital statistics by hand. Not. As we speak, the majority of milk originates from herds ranging in size from approximately 5,000 to 100,000 head. Farmers have traditionally relied on PC-based software tools to manage and track essential farm data effectively. Recently, farmers have started adopting automation technologies to streamline the process of monitoring and data recording by deploying “smart” tags and other Internet of Things (IoT) sensors to track their cows’ movement.
Pedro Meza, Vice President of Engineering at iYOTAH, notes that one of the many challenges he encountered upon entering the industry is the surprising truth that contented cattle indeed produce more milk.
Notwithstanding the challenges facing agriculture, dairy farmers are seeking more efficient and cost-effective ways to leverage data insights amidst shrinking revenue margins. However they’ve been stymied. Farmers typically utilize legacy Windows applications that track specific data points, such as herd records and breeding histories, feed consumption, or milk production, including samples of fat and protein content that determine the milk’s market value. “Diverse financial data, such as income and expenses, are meticulously recorded in Excel or QuickBooks, and often stored as physical receipts in a designated container, like a shoebox.”
“Dairy farms operate on a massive scale, generating millions annually, yet farmers reveal that up to 30 percent of their time is devoted to collecting and managing data,” Meza noted.
When information remains isolated and non-digitized, its historical trends cannot be analyzed, and it cannot be integrated to inform more informed decisions. By integrating data from two tables showcasing hourly temperature and humidity readings alongside cow feed consumption patterns, farmers can gain valuable insights to refine their feeding strategies and maximize milk production efficiency.
Tipping Level
iYOTAH designed a cutting-edge solution tailored to the needs of today’s farmers, providing a comprehensive, integrated platform that delivers a panoramic view of their operations in real-time, coupled with customizable alert systems and intuitive drill-down capabilities to facilitate swift data exploration and analysis.
Rather than compelling farmers to hastily relinquish their well-established practices, iYOTAH aimed to develop a suite of software agents that would autonomously install themselves on farmers’ computers. At regular intervals, brokers meticulously scanned the database for newly added or uploaded data – a vast array of information spanning highly compressed genetic profiles of herds to intricate dimensional models. When a change is identified, the data is seamlessly integrated into a scalable information repository, housed on Amazon’s robust S3 cloud storage solution. Transformed, metadata-tagged, cleaned, and deduplicated, the information is now prepared for query-ready consumption.
To optimize query performance for its dashboards, iYOTAH thoroughly evaluated various options. Although they briefly showcased Snowflake, it was ultimately terminated. Utilizing AWS-hosted Spark as the database engine, they leveraged its capabilities to serve queries to a visually engaging Tableau dashboard. Meza and his team also voted in favor of this approach, citing concerns that it would bind them to a costly infrastructure that “failed to adequately address their evolving long-term requirements.”
Ultimately, iYOTAH decided to build its software from the ground up and leverage Rockset as its real-time query engine. While increased investment in dashboard development was a possibility, iYOTAH recognized the importance of being accountable for their personalized roadmap, according to Meza. And Rockset accelerated the development of information software and pipelines by establishing an efficient construction method. By leveraging Amazon’s Simple Storage Service (S3), we were able to seamlessly integrate with Rockset, streamlining data transfers and simplifying our workflows. Data is continuously uploaded to Rockset from Amazon S3 every 3-5 minutes.
Rockset’s capabilities also significantly augment those of SQL, leveraging expertise that all of Meza’s builders have honed as consultants. The Rockset platform offers a range of time-saving features, including pre-built, parameterized SQL queries stored within its database, which can be seamlessly executed through a dedicated REST endpoint, simplifying integration with other systems. Simplifying queries enables builders to efficiently manage and optimise processes, especially in manufacturing settings.
The data is aggregated and presented through a unified software, currently comprising ten customizable dashboards that can display over 150 unique visualizations, all powered by Rockset’s capabilities.
The dashboard provides near real-time insights into the nutritional profile of its milk, including fat and protein levels, which directly impacts its market value. Our team also prioritizes breeding, meticulously tracking cow pregnancies from conception to calving, while alerting farmers at precisely the right moment to breed their herds. Leveraging genetic insights, we then expertly match these cows with the most suitable sires to optimize milk production and yield exceptional results.
Rockset empowers real-time monitoring of animal welfare, as well as tracking feed and manure levels. Farmers can customize alert settings to receive notifications when temperatures exceed or fall below specific thresholds, crucial for monitoring conditions that significantly impact dairy production – such as extreme heat or cold – which can lead to decreased milk yields and increased health issues among cows. Data from each chart can potentially be correlated or overlaid with distinct charts to create more comprehensive visualizations. Farmers can delve into real-time charts to uncover answers and gain insights through interactive exploration.
Transferring Ahead
By leveraging the iYOTAH platform, a key farm was able to integrate all operational data for the first time, enabling analysis and optimization of its feed efficiency. The innovative farming approach significantly boosted the farm’s revenue through improved cow nutrition, resulting in increased milk production and reduced waste, thereby securing the prestigious Indiana state AgriBusiness Innovation Award.
This real-time dashboard for farmers marks the beginning of a revolutionary journey. iYOTAH collaborates with the Nationwide Dairy Herd Information Association (NDHIA), which represents approximately two-thirds of the United States’ 9 million dairy cattle herd. NVIDIA and iYota have formalized a strategic partnership. To achieve this, they will collaborate to deliver value through iYOTAH’s platform to NDHIA’s members and the broader business community as a whole.
IYOTAH can develop a suite of tools providing proactive recommendations and suggestions for farmers. This development is expected to primarily rely on AI-driven assessments combining diverse data sets, such as herding information and breeding records. iYOTAH is partnering with premier institutions in agriculture and information science, such as Purdue and North Carolina State University, to integrate advanced analytics models that interpret diverse data sets and develop predictive and prescriptive models for farmers.
“We’re not just combining data, but also applying business and industry knowledge to inform high-resolution decision-making,” Meza said.
iYOTAH constructs real-time data pipelines that directly ingest sensor information into Rockset, bypassing S3 storage to minimize latency and enable rapid alert response.
IYOTAH’s current platform built around Rockset focuses primarily on the dairy industry but will soon expand to other sectors such as beef, pork, and poultry.
“We’ve established a comprehensive information pipeline and platform that can significantly impact the global food supply chain by streamlining data management for all types of animal livestock.”
