Kinect v2 Examples with MS-SDK and Nuitrack SDK

Kinect2_MsSDKKinect v2 Examples with MS-SDK and Nuitrack SDK is a set of Kinect-v2 (aka ‘Kinect for Xbox One’) examples that use several major scripts, grouped in one folder. The package contains over thirty demo scenes.

Apart of the Kinect-v2 and v1 sensors, the K2-package supports Intel’s RealSense D400-series, as well as Orbbec Astra & Astra-Pro sensors via the Nuitrack body tracking SDK. Please mind that Nuitrack SDK is in early stage, and issues are possible.

The avatar-demo scenes show how to utilize Kinect-controlled avatars in your scenes, gesture-demos – how to use the programmatic or visual gestures, fitting room demos – how to create your own dressing room, overlay demos – how to overlay the body parts with virtual objects, etc. You can find short descriptions of all demo-scenes in the K2-asset online documentation.

This package works with Kinect-v2, Kinect-v1 (aka Kinect for Xbox One & Kinect for Xbox 360), Intel RealSense D415 & D435, Orbbec Astra & Astra-Pro, and other Nuitrack supported sensors, as well. It can be used in all versions of Unity – Free, Plus & Pro.

Free for education:
The package is free for academic use. If you are a student, lecturer or academic researcher, please e-mail me to get the K2-asset directly from me.

Free copy of ‘Kinect Mocap Animator’:
As of 17.Aug.2018, I provide the ‘Kinect Mocap Animator’-package free of charge for personal use to anyone, who requests it by e-mail and has a valid invoice for the K2-asset.

One request:
My only request is NOT to share this package or its demo scenes in source form with others, or as part of public repositories, without my explicit consent. Please respect my work.

Customer support:
* First, please check if you can find the answer you’re looking for on the Tips, tricks and examples page, as well as on K2-asset Online Documentation page. See also the comments below the articles here, or in the Unity forums. If it is not there, you may contact me, but please don’t do it on weekends or holidays. This is the only time I can have some rest.
* If you e-mail me, please include your invoice number. I provide free e-mail support for the first 3 months after the product purchase (limited to max 10 issues). More information regarding e-mail support can be found here.
* Please mind, you can always update your K2-asset free of charge, from the Unity Asset store.

How to run the demo scenes:
1. (Kinect-v2) Download and install Kinect for Windows SDK v2.0. The download link is below.
2. (Kinect-v2) If you want to use Kinect speech recognition, download and install the Speech Platform Runtime, as well as EN-US plus the other needed language packs. The download links are below.
3. (Nuitrack or Orbbec) If you want to work with Nuitrack body tracking SDK, look at this tip. If you want to work with Orbbec Astra or Astra-Pro sensors via OpenNI2, look at this tip.
4. Import this package into new Unity project.
5. Open ‘File / Build settings’ and switch to ‘PC, Mac & Linux Standalone’, Target platform: ‘Windows’.
6. Make sure that ‘Direct3D11’ is the first option in the ‘Auto Graphics API for Windows’-list setting, in ‘Player Settings / Other Settings / Rendering’.
7. Open and run a demo scene of your choice, from a subfolder of ‘K2Examples/KinectDemos’-folder. The short descriptions of all demo scenes are available here.

* Kinect for Windows SDK 2.0 (Windows-only) can be found here.
* MS Speech Platform Runtime v11 can be downloaded here. Please install both x86 and x64 versions, to be on the safe side.
* Kinect for Windows SDK 2.0 language packs can be downloaded here. The language codes are listed here.

Documentation:
* The official online documentation of the K2-asset is available here.

Downloads:
* The official release of the ‘Kinect v2 with MS-SDK’ is available at Unity Asset Store. All updates are free of charge.

Troubleshooting:
* If you get compilation errors like “Type `System.IO.FileInfo’ does not contain a definition for `Length’”, you need to set the build platform to ‘Windows standalone’. For more information look at this tip.
* If the Unity editor crashes, when you start demo-scenes with face-tracking components, look at this workaround tip.
* If the demo scene reports errors or remains in ‘Waiting for users’-state, make sure you have installed Kinect SDK 2.0, as well as the other needed components, and the sensor is connected.

* Here is a link to the project’s Unity forum: http://forum.unity3d.com/threads/kinect-v2-with-ms-sdk.260106/
* Many Kinect-related tips, tricks and examples are available here.
* The official online documentation of the K2-asset is available here.

Known Issues:
* If you get compilation errors, like “Type `System.IO.FileInfo’ does not contain a definition for `Length’“, open the project’s Build-Settings (menu File / Build Settings) and make sure that ‘PC, Mac & Linux Standalone’ is selected as ‘Platform’ on the left. On the right side, ‘Windows’ must be the ‘Target platform’, and ‘x86’ or ‘x86_64’ – the ‘Architecture’. If Windows Standalone was not the default platform, do these changes and then click on ‘Switch Platform’-button, to set the new build platform. If the Windows Standalone platform is not installed at all, run UnityDownloadAssistant again, then select and install the ‘Windows Build Support’-component.
* If you experience Unity crashes, when you start the Avatar-demos, Face-tracking-demos or Fitting-room-demos, this is probably due to a known bug in the Kinect face-tracking subsystem. In this case, first try to update the NVidia drivers on your machine to their latest version from NVidia website. If this doesn’t help and the Unity editor still crashes, please disable or remove the FacetrackingManager-component of KinectController-game object. This will provide a quick workaround for many demo-scenes. The Face-tracking component and demos will still not work, of course.
* Unity 5.1.0 and 5.1.1 introduced an issue, which causes some of the shaders in the K2-asset to stop working. They worked fine in 5.0.0 and 5.0.1 though. The issue is best visible, if you run the background-removal-demo. In case you use Unity 5.1.0 or 5.1.1, the scene doesn’t show any users over the background image. The workaround is to update to Unity 5.1.2 or later. The shader issue was fixed there.

What’s New in Version 2.17.3:
1. Updated Nuitrack-interface to utilize the depth-to-color coordinate mapping for RealSense D400-series (thanks Hyeon Myeong Park).
2. Fixed shoulder-joints orientations, when the body is turned (thanks to Victoria Bloom).
3. Fixed hand-cursor position in InteractionManager, when ‘Screen space – Camera’ is used for UI (thanks to Marcelo Roca).
4. Added hand-to-fist tracking in AvatarController for non-Kinect sensors (thanks to Francesco Peciarolo).
5. Added ‘Standard user distance’-setting to KinectHolographicViewer-scene (thanks to Taha Farooq).
6. Fixed color-image orientation in scene-mesh visualizer for different sensors (thanks to Leo Yoon).
7. Added left & right foot emulation to the Nuitrack body tracking (thanks to Luca Boro).
8. Fixed the Nuitrack-interface scripts not to break the UWP-builds.

Videos worth more than 1000 words:
Here is a video by Ricardo Salazar, created with Unity5, Kinect v2 and “Kinect v2 with MS-SDK”, v.2.3:

..a video by Ranek Runthal, created with Unity4, Kinect v2 and “Kinect v2 with MS-SDK”, v.2.3

..and a video by Brandon Tay, created with Unity4, Kinect v2 and “Kinect v2 with MS-SDK”, v.2.0:

672 thoughts on “Kinect v2 Examples with MS-SDK and Nuitrack SDK

  1. Hi Rumen: I now get about 10 errors: Assets/KinectScripts/AvatarController.cs(12,14): error CS0101: The namespace `global::’ already contains a definition for `AvatarController’
    such as these. I think I followed your directions pretty well. Any ideas?

    • New
      Hi, I just purchased this. I got this error when trying to run the sample scene.

      Assets/K2Examples/KinectScripts/Interfaces/NuitrackInterface.cs(529,23): error CS0246: The type or namespace name `nuitrack’ could not be found. Are you missing an assembly reference?

      Help

      • Hi, what platform is selected in your Build Settings? And, are using the latest version (v2.17.3) of the K2-asset or an older one?

      • Yes, I’m using the current version (v2.17.3). How can i access the previous version?
        My target platform is Windows and i tried both x86 and x86_64 and got the same error

  2. hi ,
    Loving the asset
    I am working in untiy 2017.3 version.
    i have an issue in “KinectFittingRoom1” demo scene . there is color image turn opposite side.

    i am changing the shader “userBlendedshader” to “ForegroundBlendedShader” but the depth is gone ..

    please help me !!!

    and also Background removal scene not working

    • Hi, I just checked both FittingRoomDemo1 & BackgroundRemovalDemo1 with Kinect-v2 in Unity 2017.3 editor, and it worked as expected. Are you using Kinect-v2 or some other sensor?

      Please contact me by e-mail and send me some screenshots, and the Unity log-file as well/ This way I could see what happens at your end. Please mention your invoice number, too. Here is how to find the Unity log-files on your platform: https://docs.unity3d.com/Manual/LogFiles.html

      • Ok, already answered. Please check the version of the K2-asset you’re using – see ‘WhatsNew-Kinect2-MsSdk.pdf’ in the _Readme-folder. I think your copy is quite old, because similar issues were fixed already, in the more recent releases.

    • Assets/K2Examples/KinectScripts/Interfaces/NuitrackInterface.cs(529,23): error CS0246: The type or namespace name `nuitrack’ could not be found. Are you missing an assembly reference?

      • Hi, what platform is selected in your Build Settings? And, are using the latest version (v2.17.3) of the K2-asset or an older one?

      • If you are using Kinect-v2 or Kinect-v1, the selected platform should be ‘PC, Mac & Linux Standalone’ with target platform ‘Windows’ and architecture ‘x86_64’. This should fix the errors.

      • To remove the Nuitrack-interface, delete KinectScripts/Interfaces/NuitrackInterface.cs, the KinectScripts/Interfaces/Nuitrack-folder and comment out the references to NuitrackInterface in KinectScripts/KinectInterop.cs.

      • I tried this

        To remove the Nuitrack-interface, delete KinectScripts/Interfaces/NuitrackInterface.cs, the KinectScripts/Interfaces/Nuitrack-folder and comment out the references to NuitrackInterface in KinectScripts/KinectInterop.cs.

        No errors but demo scenes not working.
        please note that i am working with kinect V2

      • There is something generally wrong in your setup. I don’t remember anyone else so far having similar problems with Kinect-v2 on Windows standalone platform. I suppose you have installed the Kinect SDK 2.0, as per setup instructions.

        In this regard, may I ask you to e-mail me the Unity log file after running and stopping the scene, so I can see what happens. Here is where to find it: https://docs.unity3d.com/Manual/LogFiles.html Please also mention your invoice number in the e-mail.

  3. Hi, is it possible for an ordinary camera (webcam) for motion detection?

    Another question, On your scene KinectFaceTrackingDemo3. Is it possible to get exact data of face only? I mean we just need to get background removal of face and the neck and we want to remove the quad boundary being seen on background removal.

    • The motion detection is done by segmenting users from the depth camera image, as far as I remember. If you want to use external color camera along with it, you would need to calibrate the depth camera and the new color camera, to get correct color image overlays.

      To the 2nd questions: See the source of the SetFaceTexture-component in the scene. It extracts the face texture and applies it to the given target object (in this case – the Quad). You can modify the code, to apply the texture to an object of your choice. Hope I understood your issues correctly.

      • Thanks Rumen! As of now I am trying to work with SetFaceTexture quad component for other rigged models (combination of avateering + setfacetexture for face but i only want to expose the head). But I cant figure out about the the SetFaceTexture error saying that:

        Texture has out of range width / height
        UnityEngine.Texture2D:Resize(Int32, Int32)
        SetFaceTexture:Update() (at Assets/K2Examples/KinectDemos/FaceTrackingDemo/Scripts/SetFaceTexture.cs:166)

        I tried to copy the properties of the quad from original KinectFaceTrackingDemo3 demo but it seems it doesnt work. I made the quad (FaceView based from your example) as a child of the neck portion of the rigged model. Can you tell me about the cause of error?

        PS: Didnt modify anything on SetFaceTexture script.

        Thanks!

      • The face texture uses the face part (determined by face-width & face-height parameters, around the head) of the color image or the background-removed image (called foreground texture). That’s why the BackgroundRemovalManager-component is in the scene, too. If you look above line 166 in the script, there are some checks if the face rectangle is not beyond the limits of the foreground or color-camera textures (they have equal resolutions). Maybe this check estimates negative values for width or height of the face texture sometimes. It’s easy to debug and find out, I think. There is nothing special about this script, really 😉

  4. Hi,
    I am using “Orbbec Astra Pro” And “NUI Track” now.
    but Color is not displayed in “KinectOverlayDemo 1 scene”.

    Can you tell me how to use Color stream with “Kinect v2 Examples with MS-SDK and Nuitrack SDK” in “Orbbec Astra Pro”?

    This sentence uses Google Translate.

    thank you.

    • This is one of the fixes in the upcoming v1.17.2 of the K2-asset. I submitted it yesterday to Unity asset store, but it is not yet published. If you want to get it earlier, please e-mail me your request, along with the invoice number you got from the Asset store.

      • Hi,
        Kinect v2 Examples with MS-SDK and Nuitrack SDK
        In Current Ver 2.17.2

        I confirmed that “Orbbec Astra Pro” And “NUI Track” works properly with “KinectOverlayDemo 1 scene”.

        This sentence is using Google translation.

        It was saved.

        Thank you

  5. Hello Rumen,

    I built and ran the projects and to my folly, I tried to see if the Orbbec Astra would still work (detect skeletons) if several of those apps ran in the background. The demos didn’t crash but they didn’t work either, and now even when I reset everything, the whole thing has stopped working. I’m not using the trial version of Nuitrack also. Any thoughts on how I can fix this..?

  6. Hello! The package is absolutely fantastic. So much stuff for 20$. My goal is to track a person that is lying down on his back. It doesn’t work because of the depth difference is too small between the body and the table on which the person is lying on I guess. Do you have any idea on which settings I should tune so that it can work, if possible at all? Thanks for everything!

    Vincent

    • The Kinect’s face-tracking subsystem tracks the positions of the eyes. See KinectFaceTrackingDemo5-scene in KinectDemos/FaceTrackingDemo-folder. But if under eye-tracking you mean tracking the direction the user is looking, this is not supported. In this case, better use the forward-direction of the head.

  7. Hi Rumen, I use your Asset with a Kinect v2 atm. Apart from installing the SDKs and following your tips, do I have to make any changes to my Unity Project to get it to work with an Orbbec device? Cheers.

  8. Hi Rumen, Is it possible to create fake body that can be detected by kinect as user? Or any idea how can I accomplish this? I will use this for testing purposes. Thanks.

    • Hi, for testing purposes you can make recordings with the Kinect Studio (part of Kinect SDK 2.0) and then play them back, after you run the scene.

  9. Hello! i have used this asset for making body tracking system.
    I change my Kinect V2 into Intel Realsense D415(with Nuitrack).
    And i play the KinectAvatarsDemo1 included in this asset.
    Of course it doesn’t work. Demo Scene used kinect’s library and abandoned others.
    So i remove the kinect’s library in my computer.
    But demo couldn’t find libnuitrack dll file, and DummyK2Interface is linked
    although i’ve never installed regarding library.

    How can i play demo scene with nuitrack? I’ve already installed nuitrack SDK.

  10. Hi, I’m working Realsense D435 with kincet v2 package. I change “sensorAlwaysAvailable” from “true” to “false” when I run scene AvatartDemo. I see message “No sensor found. Make sure you have installed the SDK and the sensor is connected.” . But i setting SDK Realsense. Please help me. Thanks…

  11. Hello Rumen, happens when moving from one scene to another gets slow, cursor or hand is put difficult to grasp objects, I would like if I can recommend something to improve and make more fast my game

    • See ‘Howto-Use-KinectManager-Across-Multiple-Scenes.pdf’ located in K2Example/_Readme-folder in the K2-asset, as well as its implementation in KinectDemos/MultiSceneDemo. To run this demo, you need to put the scenes (in the same order) into the Scenes-in-Build list of Build settings.

      I suppose you have KinectManager-components in each scene, but there should be only one – in the startup scene. See the demo.

  12. Hello Rumen, we are using your Kinect v2 plugin for unity and we face a problem, after attaching avatar controller component to our own model when testing with kinect shoulders drop and deform the mesh. We tried different rig variations. But so far the only solution we got was to use avatar controller classic, however if using it there are jitters with model parts like feet and arms. Is there anything you can recommend us to do?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s