At the Conference on Robotic Learning (CoRL) in Munich, Germany, Hugging Face and NVIDIA announced a partnership to accelerate robotics research and development by bringing together the open source robotics community.
Hugging Face’s LeRobot open AI platform, combined with NVIDIA AI, Omniverse, and Isaac robotics technologies, enables researchers and developers to drive progress across a wide range of industries, including manufacturing, healthcare, and logistics.
Open source robotics in the age of physical AI
The era of physical AI, robots that understand the physical properties of their environments, is rapidly transforming global industries.
To drive and sustain this rapid innovation, robotics researchers and developers need access to open source, extensible frameworks that span the robot training, simulation, and inference development process. Models, datasets, and workflows released under a shared framework allow you to immediately use the latest advances without having to rewrite code.
Hugging Face’s leading open AI platform serves more than 5 million machine learning researchers and developers, providing tools and resources to streamline AI development. Hugging Face users can access and fine-tune the latest pre-trained models and build AI pipelines with a common API that includes over 1.5 million models, datasets, and applications freely accessible on the Hugging Face Hub.
Developed by Hugging Face, LeRobot extends successful paradigms from a library of transformers and diffusers into the realm of robotics. LeRobot offers a comprehensive suite of tools for data acquisition, model training, and sharing simulation environments, as well as the design of low-cost manipulator kits.
NVIDIA AI technology, simulation, and open source robot learning modular frameworks such as NVIDIA Isaac Lab can accelerate LeRobot’s data collection, training, and validation workflows. Researchers and developers can share models and datasets built with LeRobot and Isaac Lab, creating a data flywheel for the robotics community.
Scaling robot development with simulation
Developing physical AI is difficult. Unlike language models that use extensive Internet text data, physically-based robotics relies on physical interaction data along with visual sensors, which is difficult to collect at scale. Collecting real-world robot data for dexterous manipulation across numerous tasks and environments takes time and effort.
To facilitate this, Isaac Lab, built on NVIDIA Isaac Sim, enables demonstration and trial-and-error robot training in simulation using high-fidelity rendering and physics simulation, creating a realistic synthetic environment. and create data. Isaac Lab combines GPU-accelerated physics simulations with parallel environment execution to generate vast amounts of training data (equivalent to thousands of real-world experiences) from a single demonstration provides the functionality to
The generated motion data is used to train the policy through imitation learning. After successful training and validation in simulation, the policy is deployed to a real robot and further tested and tuned to achieve optimal performance.
This iterative process leverages the accuracy of real-world data and the scalability of simulated synthetic data to ensure a robust and reliable robotic system.
Sharing these datasets, policies, and models on Hugging Face creates a robotics data flywheel that allows developers and researchers to build on each other’s work and accelerate progress in the field. .
“The robotics community thrives when we build together,” said Animesh Garg, assistant professor at Georgia Tech. “Adopting open source frameworks like Hugging Face’s LeRobot and NVIDIA Isaac Lab will accelerate the pace of AI-powered robotics research and innovation.”
Foster collaboration and community engagement
Planned collaborative workflows include collecting data through remote operations and simulations in Isaac Lab and storing it in the standard LeRobotDataset format. The data generated using GR00T-Mimic is used to train robot policies through imitation learning and then evaluated in simulation. Finally, the validated policy is deployed to real-world robots using NVIDIA Jetson for real-time inference.
The first steps in this collaboration have already taken place, demonstrating a physical picking setup using LeRobot software running on NVIDIA Jetson Orin Nano, providing a powerful and compact computing platform for deployment. .
“Combining the Hugging Face open source community with NVIDIA hardware and Isaac Lab simulation has the potential to accelerate innovation in AI for robots,” said Remi Cadene, Principal Scientist at LeRobot.
This effort builds on NVIDIA’s community contributions in generative AI at the edge, supporting the latest open models and libraries such as Hugging Face Transformers, large-scale language models (LLMs), small-scale language models (SLM), optimizing inference for multimodal vision language models. (VLM) and an action-based version of VLM, the Vision Language Action Model (VLA), diffusion policy, and voice model, all with strong community-driven support.
Together, Hugging Face and NVIDIA aim to accelerate the efforts of a global ecosystem of robotics researchers and developers that are transforming industries from transportation to manufacturing to logistics.
Learn about NVIDIA’s robotics research papers at CoRL. This also includes VLM integration, temporal navigation, long-term planning, etc. for improved environmental understanding. Check out workshops with NVIDIA researchers at CoRL.
