Kinect has a traditional RGB camera in it, as found in a multitude of webcams and mobile phones, and it's capable of a standard 640x480 resolution, operating at 30 frames per second. Alongside this are the depth sensors. These bathe the area in an infra-red wash, colour-coding the scene based on how far away the objects are. This is the key to Kinect's unique capabilities. Not only does it allow games to know where everyone and everything is in 3D space, but it also means that even without the RGB data it can operate just fine in any lighting conditions - even pitch black.
The depth map is the most crucial weapon in Kinect's arsenal, and it can also be integrated with the traditional RGB webcam image in a process known as registration, although the integration of the two planes together does incur a small additional CPU load. However, even without registration, we can see that developers are making use of it in the launch titles, visualising it directly into the game.
Perhaps the most dramatic example of this is in Ubisoft's Your Shape: Fitness Evolved. Here your on-screen persona is effectively a post-processed rendition of the depth map, with the main figure (i.e. the player) cut out, with additional particles effects overlaid to create a much smoother look.
Latency
Quote:
However, it is important to point out that there are a huge range of factors that conspire to make Kinect react differently depending on the game being played. Rare's Nick Burton, who we spoke to at the hands-on event, was very specific in pointing out that his game operates with a latency of 150ms, not including display lag.
We've seen in previous Digital Foundry tests on console game latency that 30FPS titles typically operate at 100ms in a best-case scenario. While Kinect Sports can run at 60FPS (certainly the bowling does), the scan rate of Kinect is limited to 30FPS, so at the very best we should never assume to see any kind of response better than the 100ms "standard".
Over and above that there are other considerations. For example, in terms of the lag, there's going to be a fixed "cost" in acquiring the depth map and the RGB info, and there's also going to be an additional latency caused by the transfer of USB - you'd think that these are the baseline fixed costs of operating the system. Then there's the processing of the acquired data, and the different configurations from which developers can access it
So, what are these elements exactly and how would they impact performance? An RGB image with the depth map aligned (or registered, as Microsoft calls it) along with voice command analysis is clearly going to require more CPU work than processing skeletal data and some gesture-recognition alone.
This presumably explains why Burton was so keen to point out that his game operates at that specific level of latency. Even in our own latency tests, we can see an enormous spread of controller lag across many titles running at the same frame-rates and we can imagine the situation being a lot worse with Kinect, definitely in the short term.
As pointed out in our initial hands-on yesterday and indeed way back in the original gamescom Digital Foundry vs. Project Natal feature, the lag factor definitely feels like more of an issue simply because we are used to our actions being transmitted to the console at the highest, fastest level.
Our hands and fingers are capable of extremely fast reaction times, so while jettisoning the joypad may appear to make the games more accessible, the human body as a whole cannot react anywhere near as fast as our brain-to-finger reflexes. This is why Kinect games that, on paper, have the same overall latency as some more traditional joypad games, actually feel a lot less responsive.
Factoring in the inherent latency of human movement is perhaps one of the greatest challenges facing Kinect developers, and what's more this is this is likely to be more or less of an issue depending on the person playing.
CPU time
Quote:
But just how much of a burden on the CPU are the NUI libraries, as they're referred to by Microsoft? Bearing in mind the fact that we have excellent motion-tracking on a consumer-level device, and that the processing chip in the original Natal design was removed in favour of a software layer implemented on the 360 itself, you could be forgiven for thinking that game performance would be impacted by the additional load.
Well, the initial rumours about the use of one core of the Xenon CPU are clearly way off beam based on the games we've seen. Kudo Tsunoda's Kinect Adventures is running some very attractive visuals using the Unreal Engine, which probably wouldn't take kindly to being run on an effectively crippled CPU.
However, the best example we've seen yet is the demo of the Kinect rendition of Forza Motorsport at the Microsoft media briefing. There's little doubt that the action on-stage wasn't scripted (as per the Project Natal Experience the night before) and here we are seeing the same silky smooth 720p60 that we've come to expect from Turn 10's excellent racing titles.
Probably the best source we have on the actual additional load that Kinect incurs on the Xenon CPU comes from the Wired magazine article on the then-Project Natal run a few months back. In that story, Xbox's technological frontiersman Alex Kipman pegged the load at 10 to 15 per cent depending on the tasks being asked of the various libraries.
Bearing in mind what Kinect is achieving, that's a pretty impressive achievement. This does not fully contradict the "one core" rumours that were previously circulating - it may simply be the case that Kinect only uses a certain percentage of that single processing unit, most likely spread across the two available hardware threads.
So while today's Forza demo suggests that Kinect can be backwardly imported into existing games, scheduling those significant NUI libraries into an existing pipeline could be problematic, especially in a system as sensitive to lag as this one.
Check out the lag in this video.
http://www.eurogamer.net/videos/kinect-latency-tests?size=med
27CDB6E-AE6D-11cf-96B8-444553540000" width="437" height="265" id="viddler"><param name="movie" value="http://www.viddler.com/simple_on_site/9b762fbe" /><param name="allowScriptAccess" value="always" /><param name="allowFullScreen" value="true" /><param name="flashvars" value="fake=1"/><embed src="http://www.viddler.com/simple_on_site/9b762fbe" width="437" height="265" type="application/x-shockwave-flash" allowScriptAccess="always" allowFullScreen="true" flashvars="fake=1" name="viddler" ></embed></object>
