Computer Vision Examples (Lite) is the next step in the evolution of depth sensor examples (incl. “Azure Kinect and Femto Bolt Examples“, “Kinect-v2 Examples“, etc.). Instead of a depth sensor though this asset uses a plain web camera or video recording as input, and AI models to provide depth estimation, body tracking, object tracking and other streams. The package contains over thirty demo scenes.
The avatar-demo scenes show how to utilize user-controlled avatars in your scenes, gesture demo – how to use discrete and continuous gestures in your projects, fitting room demos – how to overlay or blend the user’s body with virtual models, background removal demo – how to display user silhouettes on virtual background, etc. Short descriptions of all demo-scenes are available in the online documentation.
This package works with plain web cameras and video clips that can be played with Unity video player. It can be used in all versions of Unity – Free, Plus & Pro.
How to run the demo scenes:
1. Create a new Unity project (Unity 6.0 or later, as required by the ‘Inference Engine’ package).
2. Open ‘Window / Package Manager’ in Unity editor, go to Unity Registry and install the ‘Inference Engine’-package. This is the Unity package for AI model inference.
3. Open ‘File / Build settings’ and make sure that Windows, macOS or Linux is the currently active platform.
4. Open and run a demo scene of your choice from a subfolder of the ‘ComputerVisionExamples/DemoScenes’-folder. Short descriptions of all demo-scenes are available in the online documentation.
Current limitations:
1. Because of the intensive GPU utilization, please avoid using the CVE-package on mobile platforms. The performance is not good enough, and the device may overheat.
2. Please don’t use the CVE-package on WebGL platform.
Downloads:
* Computer Vision Examples (Lite) can be downloaded from Unity Asset store.
One request:
Please don’t share this package or its demo scenes in source form with others, or as part of public repositories, without my explicit consent.
Documentation:
* The basic documentation is in the Readme-pdf file, in the package.
* The online documentation is available here
Troubleshooting:
* Don’t expect everything to be perfect. If you get any issues or errors, please contact me to report the issues.
* If you get errors in console, like ‘Texture2D’ does not contain a definition for ‘LoadImage’ or ‘Texture2D’ does not contain a definition for ‘EncodeToJPG’, please open the Package Manager, select ‘Built-in packages’ and enable ‘Image conversion’ and ‘Physics 2D’ packages.
What’s New in Version 1.7:
1. Upgraded CVE-package to Unity 6.0; Upgraded Sentis to InferenceEngine.
2. Upgraded core stream providers to v1.7. Fixed issues in various sync modes.
3. Updated background removal demo scenes to detect multiple users.
Videos worth more than 1000 words:
User motion tracking in a video and its retargeting on 3d avatar(s), with the help of ‘Computer Vision Examples for Unity’. The same can be applied to live webcam capture.
User segmentation (background removal) in a video, with the help of ‘Computer vision examples for Unity’. The same can be applied to live webcam capture.
Point cloud (scene mesh) of a video recording. The camera moves slightly left and right, to increase spatial perception. The same can be applied to live webcam capture.
This is great idea, i brought from asset store and try it now.
This looks really promising! I would like to ask if the package includes the same demos that is available in the Kinect examples? Does the scripts work similarly, so there is a new interface below the KinectController gameobject everything else works like before? Thanks
Yes, exactly. My idea was everything to be backwards compatible. The demo scenes here are less than before, but they will increase in time, as I improve the asset and add more features.
It’s a great project. We see a new example by Rumen after the Azure Kinect Examples.
Thank you!
This looks great! Can you tell us which models you are using (especially for Image Segmentation)?
Hi Moritz, for pose estimation and segmentation I’m using the MediaPipe’s Pose Landmarker model.
I’ve installed the sentis and your package.
but I got below errors. (I use Unity2023.3.26f1)
How can I solve it?
——————————————————————
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(191,50): error CS1061: ‘string’ does not contain a definition for ‘name’ and no accessible extension method ‘name’ accepting a first argument of type ‘string’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(210,41): error CS0117: ‘TensorFloat’ does not contain a definition for ‘AllocZeros’
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\WebcamDepthStreamProvider.cs(471,42): error CS1061: ‘TensorFloat’ does not contain a definition for ‘dataOnBackend’ and no accessible extension method ‘dataOnBackend’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\PoseLmBodyStreamProvider.cs(386,41): error CS0117: ‘TensorFloat’ does not contain a definition for ‘AllocZeros’
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(514,50): error CS1061: ‘TensorFloat’ does not contain a definition for ‘dataOnBackend’ and no accessible extension method ‘dataOnBackend’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(620,61): error CS1061: ‘IWorker’ does not contain a definition for ‘ExecuteLayerByLayer’ and no accessible extension method ‘ExecuteLayerByLayer’ accepting a first argument of type ‘IWorker’ could be found (are you missing a using directive or an assembly reference?)
Assets\ComputerVisionExamples\KinectScripts\Interfaces\Providers\SentisManager.cs(763,20): error CS1061: ‘TensorFloat’ does not contain a definition for ‘CompleteOperationsAndDownload’ and no accessible extension method ‘CompleteOperationsAndDownload’ accepting a first argument of type ‘TensorFloat’ could be found (are you missing a using directive or an assembly reference?)
You probably try to install an older CVE-package with Sentis v1.4. The latest Sentis release has breaking changes that cause these errors. Please download and install CV Examples v1.3 from Unity asset store, and the errors should go away.
I’ve solve what I asked. Thanks.
Hello, is there any way to use several cameras for blob detection?
Hi there. Theoretically yes, but there isn’t camera pose provider and multi-camera calibration in CV-Examples yet, and have not tested this use case too. You should set the camera poses manually. The model inferences needed by multiple cameras could cause some additional lag too.
Calibrating multiple cameras poses no issue; it can be accomplished by positioning several cameras within a scene and using multiple planes with blobs. However, model lag presents a significant challenge. Additionally, creating a single virtual camera from multiple cameras for blob detection is another complex task.
Pingback: Azure Kinect and Femto Bolt Examples for Unity | RF Solutions - Technology, Health and More
Pingback: Kinect v2 Examples with MS-SDK | RF Solutions - Technology, Health and More
I’ve noticed that when using a computer vision example in a 1920×1080 , such as in a background removal scene, the recognition effect is only effective in the central area of the scene. Beyond this central area, it loses effectiveness. How can I make it work effectively across the entire screen? Please let me know, thank you.
Yes. Please enable the ‘Detect multiple users’-setting of the UniCamInterface-component in the scene.