Azure Kinect Examples for Unity

Azure Kinect Examples for Unity, v1.7.1 (also available in Unity Asset store) is a set of Azure Kinect (aka ‘Kinect for Azure’, K4A) examples that use several major scripts, grouped in one folder. The package currently contains over twenty five demo scenes. Apart of the Azure Kinect sensor (aka K4A), the K4A-package supports the “classic” Kinect-v2 (aka Kinect for Xbox One) sensor, as well as Intel RealSense D400-series sensors.

The avatar-demo scenes show how to utilize Kinect-controlled avatars in your scenes, gesture demo – how to use discrete and continuous gestures in your projects, fitting room demos – how to overlay or blend the user’s body with virtual models, background removal demo – how to display user silhouettes on virtual background, point cloud demos – how to show the real environment or users as meshes in your scene, etc. Short descriptions of all demo-scenes are available in the online documentation.

This package works with Azure Kinect (aka Kinect for Azure, K4A), Kinect-v2 (aka Kinect for Xbox One) and Intel RealSense D400-series sensors. It can be used with all versions of Unity – Free, Plus & Pro. Please note, the body tracking demo scenes don’t work with the Intel RealSense D400-series sensors.

How to run the demo scenes:
1. (Azure Kinect) Download and install the latest release of Azure-Kinect Sensor SDK. The download link is below. Then open ‘Azure Kinect Viewer’ to check, if the sensor works as expected.
2. (Azure Kinect) Follow the instructions on how to download and install the latest release of Azure-Kinect Body Tracking SDK and its related components. The link is below. Then open ‘Azure Kinect Body Tracking Viewer’ to check, if the body tracker works as expected.
3. (Kinect-v2) Download and install Kinect for Windows SDK 2.0. The download link is below.
4. (RealSense) Download and install RealSense SDK 2.0. The download link is below.
5. Import this package into new Unity project.
6. Open ‘File / Build settings’ and switch to ‘PC, Mac & Linux Standalone’, Target platform: ‘Windows’ & Architecture: ‘x86_64’.
7. Make sure that ‘Direct3D11’ is the first option in the ‘Auto Graphics API for Windows’-list setting, in ‘Player Settings / Other Settings / Rendering’.
8. Open and run a demo scene of your choice from a subfolder of the ‘AzureKinectExamples/KinectDemos’-folder. Short descriptions of all demo-scenes are available in the online documentation.

* The latest Azure Kinect Sensor SDK (v1.3.0) can be found here.
* The latest Azure Kinect Body Tracking SDK (v0.9.4) can be found here.
* Older releases of Azure Kinect Body Tracking SDK can be found here.
* Instructions how to install the body tracking SDK can be found here.

* Kinect for Windows SDK 2.0 can be found here.
* RealSense SDK 2.0 can be found here.

Free for education:
The package is free for academic use. If you are a student, lecturer or university researcher, please e-mail me to get a free copy of the K4A-asset directly from me.

One request:
My only request is NOT to share this package or its demo scenes in source form with others, or as part of public repositories, without my explicit consent.

Downloads:
Please purchase the package at one of the following stores. In this case all future updates will be available to you, free of any charge:
* The K4A-asset may be purchased and downloaded in my online store.
* It may be purchased and downloaded in Unity Asset store, as well.
* If you’d like to try the free version of the K4A-asset, you can find it here.

Documentation:
* The basic documentation is in the Readme-pdf file, in the package.
* The K4A-asset online documentation is available here.
* Many K4A-package tips, tricks and examples are available here.

Troubleshooting:
* If you get “‘KinectInterop.DepthSensorPlatform’ does not contain a definition for ‘DummyK2′” in the console, please delete ‘DummyK2Interface.cs’ from the KinectScripts/Interfaces-folder. This dummy interface is replaced now with DummyK4AInterface.cs.
* If you get syntax errors in console like “The type or namespace name ‘UI’ does not exist…”, please open the Package manager (menu Window / Package Manager) and install the ‘Unity UI’ package. The UI elements are extensively used in the K4A-asset demo scenes. Recently, for unknown reasons, Unity has decided to remove core packages (like the UI-package) from the standard distributions.
* If the Azure Kinect sensor cannot be started, because StartCameras()-method fails, please check again #6 in ‘How to run the demo scenes‘-section above.
* If you get a ‘Can’t create the body tracker’-error message, please check again #2 in ‘How to run the demo scenes‘-section above. Check also, if the Body Tracking SDK is installed into its by-default folder.
* If the body tracking stops working at run-time or the Unity editor crashes without notice, update to the latest version of the Body tracking SDK. This is a known bug in BT SDK v0.9.0.
* The RealSense-interface is in experimental state. Known issues are that depth and color frames are out of sync, and the body tracking doesn’t work with RealSense sensors.
* If there are errors like ‘Shader error in [System 1]…’, while importing the K4A-asset, please note this is not really an error, but shader issues due to missing HDRP & VFX packages. You only need these packages for the Point-cloud demo. All other scenes should be started without any issues.
* If there are compilation errors in the console, or the demo scenes remain in ‘Waiting for users’-state, make sure you have installed the respective sensor SDKs and the other needed components. Please also check, if the sensor is connected.

What’s New in Version 1.7.x:
1. Added KinectNetServer-component to share sensor streams over the network, and KinectNetServer -scene to act as network server for the connected sensor.
2. Added NetClientInterface-component to receive the sensor streams over the network, as well as NetClientDemo1-demo to show the network sensor functionality.
3. Added ‘Finger orientations’-setting to AvatarController, to determine whether the model should reflect the user’s finger orientations or not.
4. Added ‘Mesh texture’-setting to SceneMeshRendererGpu & UserMeshRendererGpu components, to select between the color and infrared textures (thanks to Alan).
5. Added UserSkeletonCollider-component to UserMeshDemo-scene, to provide collisions with physical objects (thanks to Daniel Gontz).
6. Fixed “missing hands” issue, when the user is far from the sensor (thanks to Nayden).
7. Added caching of space tables, to improve the scene load time (thanks to Gianluca Martini).
8. (1.7.1) Updated KinectNetServer & NetClientInterface to exchange transformed depth & color frames.
9. (1.7.1) Fixed user-tracking bug in Kinect2Interface and AvatarMatcher-script (thanks to Hao Tseng).
10. (1.7.1) Added ‘Users face backwards’-setting to Kinect2Interface (thanks to Leif Dehmelt).

Videos worth more than 1000 words:
Here is a holographic setup, created by i-mmersive GmbH, with Unity 2019.1f2, Azure Kinect sensor and “Azure Kinect Examples for Unity”:

 

58 thoughts on “Azure Kinect Examples for Unity

  1. Hey Rumen,
    Would love to try out this package but am not seeing where i can download / access it? Is it only visible to certain readers?

  2. Hi Rumen.
    I’m working on Azure Kinect and Unity and really really want to try it out. Where can I download this package.

    Thanks,
    Kien Le

  3. Hey, i’m using 2019.1.10f1 and none of the avatar demos are working with the azure kinect. Unity immediately quits after pressing play. I have installed and tested both the sensor and body tracking sdks. Do you have any suggestions? I know you said there was a bug in the body tracking sdk. Does anyone know if this has been fixed yet?

    • Hi Chris, sorry, but the K4A-asset is quite new and issues are possible. Your issue is a bit odd though. May I ask you send me the Unity editor’s log file, along with the versions of Sensor SDK and Body Tracking SDK you have installed. Here is where to find the log-file: https://docs.unity3d.com/Manual/LogFiles.html My e-mail address is in the About-page of this website.

      • I have met the same issue with Azure Kinect Body Tracking SDK ver 0.9.0 and Azure Kinect Examples for Unity v1.1. After I updated the Azure Kinect Body Tracking SDK to ver 0.9.1, the problem has been solved.

      • Hey, Sorry for the delayed response, but updating to the .9.1 body tracking sdk fixed the issue. Works well now, thanks!

    • Hi Chris, this might not be your issue, but I noticed that if you have the Kinect Sensor already open when you try to run any of the Demos, it will crash immediately. So make sure you don’t have any other service or process that has opened the sensor before trying to use the Unity Project.

      • Andrew, thank you for providing this great tip! It may be useful to many of the users of K4A-sensor. As a matter of fact, I haven’t heard anything from Chris since reporting this issue. I suppose the issue got resolved by itself.

  4. Hi Rumen, thanks a lot for this! Quick question tho.
    In Kinect 4 Azure Interface when I play a recorded MKV with a depth track that works in the official Azure Kinect Viewer I get
    ArgumentException: Result is not of a recognized result type.
    Also depth_track_enabled of Playback is never set to True (color_track_enabled: True, depth_track_enabled: False, ir_track_enabled: False, imu_track_enabled: False, depth_delay_off_color_usec: 566)
    Any idea? Not sure if posting here is the best way to report something like this.

    • Hi Gauthier, yes I know what you mean. This is a bug in the C# wrapper of Sensor SDK. It is fixed in v1.2 of the K4A-asset that I published today. Please update from the download link in your invoice, or from your account page (if you’ve created one). Import the updated asset, try again and finally please tell me, if the issue is resolved or not.

  5. Hi Rumen,
    Rergarding those : “HDRP & VFX packages”
    Could you let us know which packages we need to download from the asset store to avoid the shader issues ?
    Thx,
    Seb

    • Hi Seb, what shader issues do you mean? If you mean the shader errors while importing the K4A-asset, just ignore them. They will not affect the demo scenes or script compilation. The HDRP & VFX packages are only needed for the point-cloud demo, along with the visual effects that produce the errors while importing. If you want to run this demo scene, see the instructions here: https://ratemt.com/k4adocs/Point-CloudDemo.html I personally would recommend to create a separate Unity project for it.

  6. Hello again, since I already purchased the unity examples package would it be possible for me to get a copy of the unity asset store version for free? I know it is the same, but it would make it a lot easier to integrate into future projects as the versions change.

    Cheers,
    Chris

    • Hi, please e-mail me your invoice number and PayPal account. I’ll refund you the money and then you can purchase the same asset on the Asset store.

  7. Hi Rumen –
    Thanks for making this asset! I’m getting the ‘Can’t create the body tracker’ error when I try to run the demo scenes, though the sensor does get turned on. The Sensor SDK and Body Tracking SDK are working fine too. I’m on Windows 10, Unity 2019.2.2f1, Sensor SDK 1.2.0, Body Tracking SDK 0.9.2. Any suggestions?

    • Hi Sean, please e-mail me and attach a screenshot of your project’s root folder, so I can take a look. This is the parent folder of the Assets-folder. Please also tell me the path, where the body tracking SDK is installed on your machine, and don’t forget to mention your invoice number, as well.

  8. Hi Rumen, I have a problem with the sensor recognition once I build my proyect. The sensor is recognized perfectly in the editor but whenever I build the proyect there is no image. Thanks in advance.

      • Hi there, I am also having this issue and can’t seem to figure out what’s going on. Would love to be in the loop on debugging this if possible. Aside from this, everything has been working great so far!

      • Hi Mark, please e-mail me and send me the Player’s log file (see the link above), so I can look more closely at what’s going on at run-time.

  9. I have 2 errors not covered in the docs, firstly I just now installed the latest versions of the tracking library and viewer as of today, and verified they work. Using Unity 2019.1.8f1 on a fresh project.

    AzureKinectOpenDeviceException: result = K4A_RESULT_FAILED
    and then
    Failed opening Kinect4AzureInterface, device-index: 0

    • Actually the error depends on what scene is open, the lights on the azure Kinect are all on when the scene plays, and the last line of the console is 1 sensor(s) open, but it never tracks anyone. In the backgroundremoval 1 demo the first error is: Can’t create body tracker for Kinect4AzureInterface0!

      • If only the body tracking doesn’t work, please make sure you have installed the Azure Kinect Body Tracking SDK into its by-default folder, i.e. ‘C:\Program Files\Azure Kinect Body Tracking SDK’. The K4A-asset expects to find it there. If it must be installed somewhere else, this would require a slight change in the code.

        If the sensor doesn’t work at all (i.e. scenes like BlobDetectionDemo or SceneMeshDemo don’t work too), please make sure you have installed Azure Kinect SDK v1.2.0. If the problem persists, please e-mail me for further analysis of the issue. I would appreciate it, if you send me the Editor log as well, so I can check more closely what happens. Here you can see where to find the Editor log: https://docs.unity3d.com/Manual/LogFiles.html

  10. That was my problem, my C drive is full so I’d installed on a separate drive, changing it fixed everything, thank you so much!

    Unrelated to your product it’s depressing how narrow the field of view on the depth sensor is on the azure, you can’t fit 6 people in it’s tracking area like you could with the previous kinect.

    Thanks again, your product helps so many people, I’ve used it since kinect v1 and really appreciate your effort.

  11. Hello Rumen F
    I‘m very happy to use your unity Azure Kinect Pakage。now i have some problems to ask you。i want to use Multiple divece in unity 。 i follow your tips to set kinectcontroller prefabes。but the sec deviece doesn’t work。there is erro about the sec device that is “Can’t create body tracker for Kinect4AzureInterface1!“。i try to close creat body function then test the the sec device, it doesn’t work either . what i need to do to fix this erro and make the sec device work?

      • Thanks very much! I bought in the Asset Store and works GREAT – excellent work! I was looking through the docs and wasn’t able to locate the licensing but I did find your repo with the free (limited) code base and it was MIT – is this version also covered under MIT? Thanks again – really appreciate your work!

  12. Yes, I saw your requests and notes on not sharing and will DEFINITELY not be sharing and will respect that for sure. Thank you for the work you’ve done, it’s a blessing!

  13. Hi! I’m an university student, and i’m going to do a project About motion tracking.
    I read in the document that for educational purpose is free to use?

    • Yes, please send me an e-mail request for a free copy of the K4A-asset from your university e-mail address. This is to prove you really study or work there.

    • Sorry, but I don’t send any packages to e-mail addresses, without getting an e-mail from them first. This is just a basic security measure.

  14. Hi Rumen! Has been a huge help using your package for my project. I have been trying to set up pose detection for multiple models by instantiating a pose detector for each new avatar created by the UserAvatarMatcher script, but even though I am able to instantiate multiple pose detectors and assign the appropriate PoseModelHelper and AvatarController scripts to each pose detector, the pose detection doesn’t work. Is it even possible to run pose detection for multiple avatars in the scene? If so, do you have any idea what I might be doing wrong?

    • Hi Mark, could you please e-mail me and send me your script (along with the meta-file) and the scene, where the script is used. I’ll try to find out what goes wrong. Please also mention your invoice number in the e-mail.

  15. Is there a way to replace the live azure kinect data stream with a pre-recorded data stream? I’m trying to stress test my scene running for many hours but would prefer not to rely on actively moving in front of the camera for a long time.

  16. Can we get the something like the interaction demo in KinectV2 back?
    That’s the most important part (for me) 🙂

Leave a Reply