Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI. learn more
Nvidia announced new AI and simulation tools to accelerate robot learning and humanoid development.
The world’s largest technology company by valuation (valued at $3.432 trillion) is helping robot developers significantly accelerate the development of AI-enabled robots with tools unveiled this week at the Conference on Robotic Learning (CoRL) in Munich, Germany. He said it will be able to accelerate.
This lineup includes the general availability of the Nvidia Isaac Lab robot learning framework. Six new humanoid robot learning workflows for Project GR00T, an effort to accelerate humanoid robot development. New world model development tools for video data curation and processing, including Nvidia Cosmos tokenizer and Nvidia NeMo Curator for video processing.
The open source Cosmos tokenizer provides superior visual tokenization for robot developers by decomposing images and videos into high-quality tokens with very high compression rates. Running up to 12x faster than current tokenizers, NeMo Curator delivers video processing curation up to 7x faster than non-optimized pipelines.
In conjunction with CoRL, Nvidia also published 23 papers and 9 workshops related to robot learning, and also released training and workflow guides for developers. Additionally, Hugging Face and Nvidia announced that they will collaborate with LeRobot, Nvidia Isaac Lab, and Nvidia Jetson to accelerate open source robotics research for the developer community.
Accelerate robot development with Isaac Lab
Nvidia Isaac Lab is an open source robot learning framework built on Nvidia Omniverse, a platform for developing OpenUSD applications for industrial digitization and physical AI simulation.
Developers can use Isaac Lab to train robot policies at scale. This open source, integrated robot learning framework applies to all embodiments, from humanoids to quadrupedal robots and cobots, to handle increasingly complex movements and interactions.
Leading commercial robot manufacturers, robot application developers, and robotics research institutes around the world rely on Isaac Lab. These include 1X, Agility Robotics, The AI Institute, Berkeley Humanoid, Boston Dynamics, Field AI, Fourier, Galbot, Mentee Robotics, Skild AI, Swiss-Mile, Unitree Robotics, and Xpeng Robotics.
Project GR00T: Foundation for general-purpose humanoid robots
Humanoids are coming. Building advanced humanoids is extremely difficult and requires multiple layers
A technical and interdisciplinary approach to enable robots to effectively perceive, navigate, and learn skills for human-robot and robot-environment interactions.
Project GR00T is an effort to develop accelerated libraries, foundational models, and data pipelines to accelerate the global humanoid robot developer ecosystem.
Six new project GR00T workflows provide humanoid developers with a blueprint for achieving the most challenging humanoid robot capabilities. This includes GR00T-Gen for building AI-powered generative OpenUSD-based 3D environments.
“Humanoid robots are the next wave of embodied AI,” Jim Huang, senior research manager for embodied AI at Nvidia, said in a statement. “NVIDIA research and engineering teams collaborated internally and across our developer ecosystem to build Project GR00T to accelerate the advancement and development of humanoid robot developers worldwide.”
Today, robot developers are building world models, or AI representations of the world that can predict how objects and the environment will react to the robot’s actions. Building these world models is incredibly computationally and data intensive, with the models requiring thousands of hours of hand-picked real-world image or video data.
Nvidia Cosmos tokenizers simplify the development of these world models by providing efficient, high-quality encoding and decoding. They set new standards in minimizing distortion and temporal instability, enabling high-quality video and image reconstructions.
Providing high-quality compression and up to 12x faster visual reconstruction, the Cosmos tokenizer paves the way for scalable, robust, and efficient generative application development across a wide range of visual domains.
Humanoid robot company 1X has updated its 1X World Model Challenge dataset to use the Cosmos tokenizer.
“Nvidia Cosmos tokenizers provide extremely high temporal and spatial compression of data while preserving visual fidelity,” Eric Jang, vice president of AI at 1X Technologies, said in a statement. Masu. “This allows us to train world models using long-range video generation in an even more computationally efficient way.”
Other humanoid and general purpose robot developers, such as Xpeng Robotics and Hillbot, are developing to manage high-resolution images and videos using Nvidia Cosmos tokenizers.
nimo curator
NeMo Curator now includes a video processing pipeline. This allows robot developers to improve the accuracy of world models that process large text, image, and video data.
Curation of video data is challenging due to its huge size and requires scalable pipelines and efficient orchestration for load balancing across GPUs. Additionally, filtering, captioning, and embedding models must be optimized to maximize throughput.
NeMo Curator overcomes these challenges by streamlining data curation and significantly reducing processing time with automatic pipeline orchestration. Supports linear scaling across multi-node, multi-GPU systems to efficiently process over 100 petabytes of data. This simplifies AI development, lowers costs, and speeds time to market.
availability
Nvidia Isaac Lab 1.2 is available now and open source on GitHub. Nvidia Cosmos tokenizer is available now on GitHub and Hugging Face. NeMo Curator for video processing will be available later this month.
A new Nvidia Project GR00T workflow is coming soon to help robotics companies more easily build capabilities for humanoid robots.
For researchers and developers learning how to use Isaac Lab, new introductory developer guides and tutorials are now available, including a migration guide from Isaac Gym to Isaac Lab.
