When someone advises you to “know your limits,” they’re probably recommending that you set boundaries and prioritize self-care by engaging in activities that help you grow, such as careful planning and preparation. While seemingly straightforward for robots, the motto’s true significance lies in conveying the importance of understanding task constraints, specifically limitations imposed by the surrounding environment, to execute duties safely and efficiently.
What happens when a robot tasked with cleaning your kitchen doesn’t quite grasp the physics at play? To devise a logical multi-step plan for cleaning the room thoroughly, the machine could initially gather information about the space’s dimensions and any specific requirements or constraints. It may then employ a combination of spatial reasoning and problem-solving abilities to create an optimal cleaning sequence, taking into account factors like furniture layout, obstacles, and access points. Although Large Language Models (LLMs) excel at processing text data, they may overlook crucial details about a robot’s physical limitations when trained solely on text, such as its reach capacity or nearby obstacles that must be avoided? Working solely with language models, and you’re more likely to discover yourself meticulously removing pasta stains from your flooring.
Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) employed vision models to perceive the robot’s surroundings, modeling its physical constraints. The team’s technique involves creating a detailed LLM (likely referring to Line of March or Lines of Movement, but unclear) which is then simulated to ensure the plan is robust and realistic. If the original plan proves unexecutable, the language model will continually produce alternative scenarios until it finds one that aligns with the robot’s capabilities.
Researchers have developed a trial-and-error technique, dubbed “Planning for Robots by way of Code for Steady Constraint Satisfaction” (PRoC3S), which enables robots to execute complex tasks such as writing individual letters, drawing stars, and arranging blocks while ensuring long-term plans meet all constraints. In the not-too-distant future, PRoC3S is poised to empower robots with the ability to tackle complex tasks in dynamic settings such as homes, where they may be asked to complete multi-step chores like “prepare my breakfast”.
“According to PhD student Nishanth Kumar SM ’24, the synergistic combination of LLMs and classical robotics methods like job and movement planners enables open-ended problem-solving, as these autonomous entities cannot execute complex tasks independently.” We’re generating a dynamic simulation in real-time, reflecting the robotic environment’s current state, and then testing multiple potential motion plans to determine the most suitable course of action. Innovative fashioning enables us to craft a remarkably realistic virtual environment, empowering robots to simulate possible actions throughout every stage of a long-term strategic plan.
The staff’s research was presented last month at the Conference on Robot Learning (CoRL) in Munich, Germany.
Fostering Self-Awareness in Robots: Defining Parameters for Open-Ended Tasks?
MIT CSAIL
The researchers employ a Large Language Model (LLM) that has been pre-trained on vast amounts of textual data sourced from across the internet. Prior to assigning tasks to PROCS, the team provided a language model with a prototypical task (e.g., drawing a square) closely related to the target task (drawing a star). The job pattern comprises a descriptive outline of the exercise, a long-term strategy, and specific details regarding the robot’s environment.
Despite their promising start, these ambitious plans ultimately faltered. In simulations, ProCS3 successfully generated stars and letters with high accuracy in nine out of 10 instances consistently. It can also build digital structures by stacking blocks into pyramidal shapes and intricate patterns, and position devices with precision, akin to carefully arranging fruit on a platter. During each of these digital demonstrations, the CSAIL method consistently outperformed similar techniques like and , achieving its intended objective more reliably.
The CSAIL engineers subsequently introduced their methodology to the real world. Experts refined their methodology by programming a robotic arm to create precise linear patterns using interlocking blocks. The PRoC3S system empowered the machine to successfully place blue and purple blocks within corresponding bowls, while also relocating all objects towards the centre of a desktop.
Kumar and co-lead writer Aidan Curtis, a PhD scholar at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), assert that these discoveries highlight the potential of large language models (LLMs) to create reliable and trustworthy plans that can be effectively implemented in practice. Researchers aim to develop a domestic robot that can efficiently process and respond to basic requests, such as “bring me some chips,” by accurately determining the necessary actions required to fulfill the task. In a simulated digital environment, PROC3S can aid a robotic checkout process by discovering a viable operational plan, thus enabling the procurement of a satisfying snack.
To advance their research, the team plans to leverage a more advanced physics simulator and extend its capabilities to tackle more complex, long-term challenges through more efficient data-search techniques. Moreover, the team intends to deploy PROCS3 to design cell robots inspired by quadrupeds, which will be capable of walking and scanning their surroundings as part of various tasks.
According to Eric Rosen, a researcher at The AI Institute, “The potential for chatbots like ChatGPT to misguide robotic actions and induce unsafe or inaccurate behaviors due to hallucinations is a valid concern that warrants further exploration.” To address this challenge, PRoC3S employs advanced algorithms for high-level task guidance, combining them with AI techniques that carefully consider the context to ensure verifiably safe and accurate actions are taken. “This combination of planning-based and data-driven strategies may prove crucial in developing robots capable of grasping and consistently executing an expanded range of tasks beyond their current capabilities.”
Kumar and Curtis’ co-authors also include CSAIL associates, specifically MIT undergraduate researcher Jing Cao and professors from the Department of Electrical Engineering and Computer Science at MIT: Leslie Pack Kaelbling and Tomás Lozano-Pérez. The research received partial funding from the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research, the Army Research Office, MIT Quest for Intelligence, and the Artificial Intelligence Institute.
