Computer Vision Examples is the next step in the evolution of depth sensor examples (incl. “Azure Kinect and Femto Bolt Examples”, “Kinect-v2 Examples”, etc.). Instead of a depth sensor though this asset uses a plain web camera or video recording as input, and AI models to provide depth estimation, body tracking and other streams. The package contains over thirty demo scenes.
The avatar-demo scenes show how to utilize user-controlled avatars in your scenes, gesture demo – how to use discrete and continuous gestures in your projects, fitting room demos – how to overlay or blend the user’s body with virtual models, background removal demo – how to display user silhouettes on virtual background, etc. Short descriptions of all demo-scenes are available in the online documentation.
This package works with plain web cameras and video clips that can be played with Unity video player. It can be used with all versions of Unity – Free, Plus & Pro.
How to run the demo scenes:
1. Create a new Unity project (Unity 2023.2 or later, as required by Unity’s Sentis package).
2. Open the ‘Package Manager’-window in Unity editor, click the ‘+’-button, select ‘Install package by name’ from the menu and enter ‘com.unity.sentis’. Then hit Enter or the ‘Install’-button. This will install ‘Sentis’ – the Unity package for AI model inference.
3. Import this package into the Unity project.
4. Open ‘File / Build settings’ and switch to ‘PC, Mac & Linux Standalone’.
5. Check if ‘Direct3D11’ is the first option in the ‘Auto Graphics API for Windows’-list setting, in ‘Player Settings / Other Settings / Rendering’.
6. Please first open and run a demo scene that contains body tracking from a subfolder of the ‘ComputerVisionExamples/DemoScenes’-folder (e.g. AvatarDemo1-scene or OverlayDemo2-scene). Stand in front of the camera to calibrate. This is needed only once, to estimate the camera intrinsic parameters.
7. Open and run a demo scene of your choice from a subfolder of the ‘ComputerVisionExamples/DemoScenes’-folder. Short descriptions of all demo-scenes are available in the online documentation.
Current limitations:
1. Because of the intensive GPU utilization, please avoid using the CVE-package on mobile platforms for now, if possible. The performance may not be good enough, and the device may overheat.
2. Currently, the CVE-package will not run on WebGL platform, because it doesn’t support compute shaders.
Downloads:
* The CVE-asset may be purchased and downloaded from the Unity Asset store. All future updates will be available free of any charge, at least for now.
Free for education:
The package is free for academic use. If you are a student, lecturer or university researcher, please e-mail me to get a free copy of the CVE-asset for academic and personal use.
One request:
Please don’t share this package or its demo scenes in source form with others, or as part of public repositories, without my explicit consent.
Documentation:
* The basic documentation is in the Readme-pdf file, in the package.
* The online documentation is available here
Troubleshooting:
* Please note, this is the one of the first releases of the CVE-package. Don’t expect everything to be perfect. If you get any issues or errors, please contact me to report your issues. Then have some patience, until the issues get resolved.
* If you get errors in console, like ‘Texture2D’ does not contain a definition for ‘LoadImage’ or ‘Texture2D’ does not contain a definition for ‘EncodeToJPG’, please open the Package Manager, select ‘Built-in packages’ and enable ‘Image conversion’ and ‘Physics 2D’ packages.
What’s New in Version 1.3:
1. Updated ‘Computer Vision Examples’ to work with Sentis v1.4.0.
2. Updated stream providers and UniCam-interface to allow non-synched frames.
3. Updated webcam device manager to allow audio when it plays back a video.
4. Added Keep-last-pose setting to AvatarController, to keep user’s last position before losing it.
5. Fixed following of offset-node and applying the muscle limits in AvatarController-component.
Videos worth more than 1000 words:
User motion tracking in a video and its retargeting on 3d avatar(s), with the help of ‘Computer Vision Examples for Unity’. The same can be applied to live webcam capture.
User segmentation (background removal) in a video, with the help of ‘Computer vision examples for Unity’. The same can be applied to live webcam capture.
Point cloud (scene mesh) of a video recording. The camera moves slightly left and right, to increase spatial perception. The same can be applied to live webcam capture.
This is great idea, i brought from asset store and try it now.
This looks really promising! I would like to ask if the package includes the same demos that is available in the Kinect examples? Does the scripts work similarly, so there is a new interface below the KinectController gameobject everything else works like before? Thanks
Yes, exactly. My idea was everything to be backwards compatible. The demo scenes here are less than before, but they will increase in time, as I improve the asset and add more features.
It’s a great project. We see a new example by Rumen after the Azure Kinect Examples.
Thank you!
This looks great! Can you tell us which models you are using (especially for Image Segmentation)?
Hi Moritz, for pose estimation and segmentation I’m using the MediaPipe’s Pose Landmarker model.
I’ve installed the sentis and your package.
but I got below errors. (I use Unity2023.3.26f1)
How can I solve it?
——————————————————————
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(191,50): error CS1061: ‘string’ does not contain a definition for ‘name’ and no accessible extension method ‘name’ accepting a first argument of type ‘string’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(210,41): error CS0117: ‘TensorFloat’ does not contain a definition for ‘AllocZeros’
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(471,42): error CS1061: ‘TensorFloat’ does not contain a definition for ‘dataOnBackend’ and no accessible extension method ‘dataOnBackend’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\PoseLmBodyStreamProvider.cs(386,41): error CS0117: ‘TensorFloat’ does not contain a definition for ‘AllocZeros’
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(514,50): error CS1061: ‘TensorFloat’ does not contain a definition for ‘dataOnBackend’ and no accessible extension method ‘dataOnBackend’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(620,61): error CS1061: ‘IWorker’ does not contain a definition for ‘ExecuteLayerByLayer’ and no accessible extension method ‘ExecuteLayerByLayer’ accepting a first argument of type ‘IWorker’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(763,20): error CS1061: ‘TensorFloat’ does not contain a definition for ‘CompleteOperationsAndDownload’ and no accessible extension method ‘CompleteOperationsAndDownload’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
You probably try to install an older CVE-package with Sentis v1.4. The latest Sentis release has breaking changes that cause these errors. Please download and install CV Examples v1.3 from Unity asset store, and the errors should go away.
I’ve solve what I asked. Thanks.