If you're going to put a computer on your head, what will you do with it? That's the question we asked ourselves when donning Microsoft HoloLens, the wearable headset that's capable of displaying interactive holographic images.

Watch the videos that Microsoft has made about HoloLens and it looks magical; a headset you put on that paints games, video calls and 3D tools on the wall around you. CEO Satya Nadella called it mind-blowing; "think of it as Kinect-plus-plus, and then square it," he said enthusiastically. And when you try it, it does feel magical – even though the way we got to try it out was very carefully scripted, with no opportunity to experiment.

Unlike virtual reality systems like Oculus Rift and Sony's Project Morpheus, HoloLens isn't everything you see. Yes, you might see the surface of Mars in life-size 3D all around you – including under your feet. But if there's someone standing next to you, you can look at them through the holographically-printed lenses that give the system its name and actually see them.

Even more cleverly, you can look at a computer; HoloLens detects the screen and doesn't project over it. It's not quite as clear as looking at a screen without the lenses on, but you can easily use a computer while you're wearing HoloLens. You don't just see the computer screen; HoloLens cuts out the whole monitor and bezel plus your keyboard and mouse – it's not clear if that's customisable and the very edge of the monitor flickers in and out of existence as HoloLens detects it and projects around it.

Features like this make the system far more flexible than a simple set of VR goggles – as well as meaning that you can wear it while you're walking around the house without worrying about bumping into walls (or other people). If, of course, you were so inclined.

You can do one thing on your PC and another on your HoloLens – or drag your mouse cursor off your screen and use it in the HoloLens world (with the right software – and as the device is a Windows 10 PC underneath, making software that works with both won't be that hard).

Virtual holograms in the real world

HoloLens is more of an augmented reality system. But unlike most AR tools which automatically slap labels on the world around you – over a camera feed on the screen of your phone or as a heads-up display with Google Glass – HoloLens is more about apps that paint things onto the real world.

microsoft hololens what s it really for we test the microsoft ar wearable image 4
Microsoft

That might be an immersive virtual reality, like the surface of Mars in the OnSight system that NASA's Jet Propulsion Lab is creating to let scientists control the Mars rover. Or it might be a hovering video call in Skype, where someone who doesn't have a HoloLens can see what you're seeing, then draw on their screen and you see that shape drawn over the real world like a magical highlighter. Nadella called it "mixed reality" and that's what we saw – although at a very early stage.

This would be amazing for customer support – imagine getting someone to talk you through changing your car battery or fitting the spare part you bought online. It could be a fantastic way for an architect or a builder to see information about a building; the builder could see where pipes and wires are in the wall so they don't drill into them, an architect could see their building where it's going to be before a brick has been laid. Surgeons could learn how to perform operations, or see MRI scans overlaid on the patient in front of them. Industrial designers could look at their CAD screens and at a 3D version of the thing they're building – and turn it over or walk around it.

And of course, HoloLens will be a fantastic prospect for gaming, because instead of the game being on your screen it can be spread around your house. The Minecraft-style HoloBuilder demo we tried put castles and lakes, farms and sheep on and underneath the furniture – and even inside the walls once we blew a virtual hole in them. Just playing with the pre-made landscape was enormous fun; building it should be delightful too. The kind of games you could make that mix a physical and digital world would be magical; imagine casting spells you can see in the room or fighting monsters hand to hand that shatter your furniture when they kick it out of the way.

What you see with HoloLens isn't as clear and bright as these promotional images from Microsoft, though, but it's still clear, colourful and vivid enough. It's a world away from the tiny window of information from Google Glass, and it doesn't replace the world around you the way Oculus VR does.

The tracking is good too; HoloLens knew what we were looking at and the virtual objects stayed perfectly in place when we walked around, crouched down and bent over to look at them. The 3D is convincing, and we didn't have any sign of headaches or nausea. There's no flickering or glitchy jumping around, and very low latency – when you move your head what you're looking at is just there. It's only where you're looking though, not in your peripheral vision.

Because the HoloLens camera can see what you're looking at, it can see gestures you make with your hands out in front of you. The basic gesture is an "air tap", moving one finger with your fist clenched. It's easy to learn and pretty accurate within space – and it's surprisingly hard to be precise in mid air. You can also use voice commands, on their own or combined with gestures or your gaze – the command applies to what you're looking at.

microsoft hololens what s it really for we test the microsoft ar wearable image 3
Microsoft

In all of the apps we tried with HoloLens, what was really impressive was the way the system could pick voice commands out perfectly – in a UK accent as well as an American one – even if you were talking to other people before and after the command. In our brief tests, voice recognition was more accurate than on Xbox One, and more responsive.

Microsoft showed us another app too, though we didn't get to try it out. HoloStudio allows you make 3D models in mid air (and send them off for 3D printing). It looks reasonably powerful but it's like Paint or Solitaire; it's not a showcase app designed to impress – it's meant to be simple enough for anyone to use because it's there to teach you how to use gestures and gaze and voice together to control a holographic app – in the same way Solitaire was designed to teach people how to drag and drop.

Getting smaller

Alex Kipman – who you'll probably know from Kinect – talked at the announcement about making technology disappear with holographic computing. With the prototypes we tried, HoloLens certainly doesn't do that. At this stage, it's a sensor rig you drape over your head and strap on tightly to put the lenses in the right place, plus a small box you hang around your neck (that's the "Holographic Processing Unit" – a custom chip that handles information from the sensors to track your gaze and head movements), all tethered to a power cable dangling from the ceiling like a strange umbilical cord.

Even the final hardware (which Microsoft didn't let us try) is far from inconspicuous – unlike a normal pair of glasses, you're not going to forget you're wearing it. It's far less conspicuous than Oculus Rift, though, and future versions of the hardware are likely to get smaller, lighter and more unobtrusive.

Unlike many other VR and AR and natural interface systems, you don't need to tether it to a PC or a phone, nor put a remote control band on your arm or a ring on your finger. You just put it on and the world around you gets an extra dimension.

Of course it will be a pricey dimension, and there are lots of things that it could do wrong. We've only tried the prototype and if the technology doesn't deliver – if the gaze and gesture controls aren't accurate, the battery life is disappointing, or the system is just too complicated for people - then HoloLens will end up as a novelty.

microsoft hololens what s it really for we test the microsoft ar wearable image 2
Microsoft

Kinect has been a huge sales success, but hasn't become a mainstream gaming accessory, and don't forget all those video headsets that never took off. With even Google Glass being removed from the shelves recently, that could be a signifier to the state of wearable technology in its current form. Perhaps we're just not ready to be donning giant helmets and draped in wires just yet

However, HoloLens is the most credible video headset we've seen so far. It's fun to use and could also be really useful, so we're hoping it will be a success – in whatever future form it arrives as.