From Spatulas to Screwdrivers: How AI is Teaching Robots to Master Tools

From Spatulas to Screwdrivers: How AI is Teaching Robots to Master Tools
đź‘‹ Hi, I am Mark. I am a strategic futurist and innovation keynote speaker. I advise governments and enterprises on emerging technologies such as AI or the metaverse. My subscribers receive a free weekly newsletter on cutting-edge technology.

Could your next handyman be a robot? MIT’s latest AI thinks so.

MIT researchers have developed a groundbreaking technique to train robots using diverse datasets, enabling them to master multiple tools and adapt to new tasks. By leveraging generative AI models called diffusion models, they combine various data sources to create a general policy for robots. This approach, known as Policy Composition (PoCo), allows robots to perform tasks like hammering nails and flipping objects with a spatula, leading to a 20% improvement in performance compared to traditional methods.

The PoCo technique is revolutionary in its ability to integrate data from different domains, such as human demonstrations and robotic simulations. This not only enhances the robot's dexterity but also its ability to generalize across various tasks. The MIT team trains separate diffusion models on specific datasets, each learning a strategy for completing a particular task. These models are then combined into a comprehensive policy, enabling robots to switch tools and adapt to new challenges.

Image: Courtesy of the researchers

The implications of such advancements in AI and robotics are vast. As robots become more adept at using tools and performing various tasks, they are poised to become an integral part of the global workforce. Multi-modal large language models (LLMs) further enhance this potential by enabling robots to process and integrate information from multiple sources, such as visual, tactile, and linguistic data. This multi-modal capability allows robots to understand and execute complex tasks that require a combination of skills and knowledge.

Imagine a future where robots can not only assemble products in factories but also assist in medical surgeries, conduct scientific research, and even perform household chores. These robots, equipped with multi-modal LLMs, will be able to understand instructions, adapt to new environments, and learn from their interactions. This will lead to a more efficient and versatile workforce, capable of performing tasks that are currently challenging or hazardous for humans.

Moreover, the integration of multi-modal LLMs will enable robots to communicate more effectively with humans. They will be able to comprehend natural language commands, interpret visual cues, and respond appropriately, making them valuable collaborators in various industries. This will not only increase productivity but also enhance safety and precision in critical tasks.


However, this technological progress comes with challenges. Ensuring that these AI-driven robots are used ethically and responsibly is crucial. There must be clear guidelines and regulations to prevent misuse and ensure that the benefits of this technology are shared widely. As we move towards a future where robots become an essential part of our workforce, we must address issues of job displacement and ensure that humans and robots can coexist harmoniously.

The advancements in AI and robotics, exemplified by MIT's PoCo technique and the integration of multi-modal LLMs, herald a new era of intelligent machines capable of performing a wide range of tasks. These robots will not only enhance productivity and efficiency but also open up new possibilities for innovation and collaboration. How can we ensure this AI-driven progress remains beneficial and ethical?

Read the full article on MIT News.


đź’ˇ If you enjoyed this content, be sure to download my new app for a unique experience beyond your traditional newsletter.

This is one of many short posts I share daily on my app, and you can have real-time insights, recommendations and conversations with my digital twin via text, audio or video in 28 languages! Go to my PWA at and sign up to take our connection to the next level! 🚀

upload in progress, 0

If you are interested in hiring me as your futurist and innovation speaker, feel free to complete the below form.

I agree with the Terms and Privacy Statement
Dr Mark van Rijmenam

Dr Mark van Rijmenam

Dr. Mark van Rijmenam is a strategic futurist known as The Digital Speaker. He stands at the forefront of the digital age and lives and breathes cutting-edge technologies to inspire Fortune 500 companies and governments worldwide. As an optimistic dystopian, he has a deep understanding of AI, blockchain, the metaverse, and other emerging technologies, and he blends academic rigour with technological innovation.

His pioneering efforts include the world’s first TEDx Talk in VR in 2020. In 2023, he further pushed boundaries when he delivered a TEDx talk in Athens with his digital twin , delving into the complex interplay of AI and our perception of reality. In 2024, he launched a digital twin of himself offering interactive, on-demand conversations via text, audio or video in 29 languages, thereby bridging the gap between the digital and physical worlds – another world’s first.

As a distinguished 5-time author and corporate educator, Dr Van Rijmenam is celebrated for his candid, independent, and balanced insights. He is also the founder of Futurwise , which focuses on elevating global digital awareness for a responsible and thriving digital future.


Digital Twin