We have implemented into Isaac Sim and Groot our custom hand tracking equipment. The next step for us is to generate synthetic data and we have stumbled upon this repository explaining the architecture: https://github.com/NVIDIA-Omniverse-blueprints/synthetic-manipulation-motion-generation. Is this the latest repository?
#Haptikos Exoskeletons - Data Generation
wraith citrus
summer glen
Hi @wraith citrus - I am checking with colleagues to verify, but from what I can gather: yes, that's the latest repo for synthetic manipulation motion generation. It's the NVIDIA Isaac GR00T Blueprint, a reference workflow for generating large-scale synthetic motion trajectories from a small number of human demos, built on Omniverse and Cosmos.
A few things to keep in mind:
• The container bundles Isaac Lab 2.0.2 and Isaac Sim 4.5.0, so make sure that aligns with your current setup
• Local deployment needs an RTX A6000 (48GB VRAM) minimum
• The Cosmos augmentation step requires an H100 or higher (80GB VRAM) on a separate node from the Isaac Lab simulation
• It runs as a Jupyter Notebook workflow via Docker Compose
Since you're already doing custom hand tracking in Isaac Sim and Groot, your demo data could feed into this pipeline as the starting point for trajectory generation. Check out the generate_dataset.ipynb notebook for the full walkthrough.
wraith citrus
Thank you very much @summer glen will be following then this architecture!!
wraith citrus
Greetings @summer glen ! I am posting the procedure of capturing our data. We are doing the teleoperation on an Ada6000 with 20 fps. In this video you can see the procedure. We have recorded 30 iterations and annotated from 5-15 correct recordings and our accuracy is ranging from 15-50%. What do you believe we can make different to ensure we have a greater success rate?
wraith citrus
Greetings again!! Now, we are iterating on the data generation. We have seen significant differences when annotating the demos and even the same set of recordings, with a different annotation, resulted in one case at 92% accuracy at data visualization and 0% on the other. Also, we pointed out that the gif in the site, https://isaac-sim.github.io/IsaacLab/main/source/overview/imitation-learning/teleop_imitation.html#…, that shows how to execute the pick and place task, doesn't seem to have physics in the wheel. Is this a bug, setup in Isaac Sim or just a wrong gif? Also, what is the most important factor when choosing the right recordings?
stoic pike
Thanks a ton for contributing this @wraith citrus
Your PR has been merged into Isaac Teleop now: https://github.com/NVIDIA/IsaacTeleop/pull/486