The moment an innovative AI video generator is launched, it’s only a matter of time before someone leverages its capabilities to create a viral sensation – such as a comical clip of actor Will Smith savoring a plate of spaghetti.
The internet’s obsession with Smith savoring a steaming hot bowl of noodles has reached new heights, transforming the humble act into a benchmark for assessing the realism of cutting-edge video generators. In February, Smith revealed the pattern on his Instagram post himself.
Google’s Veo 2 has achieved it.
Finally, we get to enjoy some delicious spaghetti.
— Jerrod Lew (@jerrod_lew)
Will Smith’s next venture into the AI neighborhood will undoubtedly make waves in 2024 alongside pasta, marking yet another milestone in his illustrious career. A teenage programmer has developed a mobile application that leverages artificial intelligence to manage Minecraft game worlds and assess their construction capabilities. In Britain, a programmer has developed a novel platform where artificial intelligence (AI) models engage in video game competitions, including popular titles such as Pictionary and Joust.
There exist additional pedagogical assessments to gauge the proficiency of AI systems. What was behind the catastrophic failures of those seemingly anomalous devices remains unclear.
While many industry-standard AI benchmarks may seem impressive to experts in the field, they often lack meaningful insights for the average person. Companies often tout the capabilities of their AI systems, boasting that they can respond to math problems on Olympiad exams or provide plausible answers to PhD-level queries. But many people, including you, primarily utilize chatbots to resolve problems such as
Crowdsourced industry metrics don’t inherently possess greater accuracy or revelatory power.
Consider, as an illustration, a widely recognized standard that many AI enthusiasts and developers strive to adhere to diligently. In the Chatbot Arena, users can freely assess the proficiency of artificial intelligence in executing various tasks, such as developing an online application or generating an image. While many raters have a predisposition to favor consultancies stemming from the AI and technology sectors, their voting patterns are largely driven by personal, subjective biases that are difficult to quantify.
Ethan Mollick, a professor of management at Wharton, recently pointed out another limitation of many AI industry benchmarks: they often fail to assess a system’s performance relative to that of an average human.
“It’s a travesty to acknowledge that there aren’t 30 distinct evaluation standards from disparate organizations in medicine, law, and recommendations, yet people continue to apply these flawed frameworks, regardless.”
Unconventional AI evaluation metrics, such as Join 4, Minecraft, and a video of Will Smith eating spaghetti, may be empirical but lack widespread applicability and generalizability. Because an AI excels in replicating Will Smith’s tone and style does not necessarily mean it will successfully generate, for instance, a well-constructed burger recipe.
I was advised by a professional that AI benchmarking neighbourhoods should focus on the far-reaching consequences of artificial intelligence, rather than measuring its capabilities within narrow domains? That’s wise. Despite initial doubts, bizarre benchmarks seem unlikely to disappear anytime soon. While they’re undoubtedly entertaining – who wouldn’t enjoy watching AI create intricate Minecraft structures? In fact, they’re straightforward to understand. As my colleague Max Zeff notes, the {industry} struggles to condense the complex knowledge of AI into bite-sized advertising that resonates with audiences.
What novel performance standards will capture attention globally by the end of 2024?
