Wearable computers and hidden projectors
“The problem of registering virtual images with the user’s view of the physical world is the main focus of AR research.”
So says Professor Bruce Thomas head of the Wearable Computer Labs at the University of South Australia and leading expert in the field of augmented reality. Since 1998 its been Thomas’s goal to create the ultimate user interface with the “eureka moment”, as he puts it, when they’ve managed to build the computers from Star Trek.
“The ultimate is walking into a room and the computer knows exactly what we want and does it perfectly without needing to ask,” dreams Thomas to Pocket-lint down the long distance line from Adelaide.
“I don’t want to walk with a phone held out 30cm in front of me. Apart from anything else, my arm will get tired but the biggest challenge we come across is how do we make this large unmanageable problem of making seamless UIs into something smaller that we can solve.”
Inspired by technology in development in military defence labs, Thomas decided that wearable computers were the next big thing around 15 years ago and began to tackle one of the larger areas for AR head on by taking it out of the controlled laboratory environment and into the real world.
“We wanted to start with with user interfaces in an outdoor setting because we recognised that there are issues with tracking devices, inputs and sensors and all the registration errors that arise when you try to relate the positions of the real and virtual words.”
The first project from Thomas and his team was a navigational tool called Map In My Hand which involved the user donning an HMD (Head Mounted Display) and strapping an old 486 PC to their backs. Working with students, and tech geeks at that, it wasn’t long before the success of the tool led to the team’s first AR application in the shape of playing a live video game on top of the background of the real world. ARQuake is an augmented reality version of the hit FPS title where you can walk around the real world shooting virtual monsters, and before you get carried away, sadly it’s not available for public play.
The WCL started by mapped out their university campus and turning it into a Quake environment by dissolving away the details but keeping the shape of the boundaries as defined by a 3D grid. Then it was a matter of attaching laptops to people’s backs along with an HMD and $5,000 worth of specialist GPS and tracking equipment to overcome the inaccuracies of the standard satellite location system from within 5m down to a more manageable 50cm. Several hurdles later and the team had a tour de force of AR on their hands as well as lots of fun running round the university shooting imaginary creatures while wearing some pretty outrageous kit. More than anything though, this was about a major success with one of the age old problems of AR - positioning and tracking.
Based on the success of ARQuake, much of the years following, up until 2006, were dedicated to taking things a step forward. Now that tracking issues had been mastered the WCL began working on an outdoor AR system where the user could create and manipulate virtual objects in the real world. It was known as Tinmith, so called because “This Is Not Map In The Hand” as its lead student wanted to make sure.
“Ultimately, the goal initially was to make a modelling system for buildings and objects larger than yourself or very far away. Things that just wouldn’t be possible to manipulate without AR,” Thomas explains.
Again, it was a case of creating a backpack mounted as a large utility belt complete with a high powered 14-inch laptop and connected to an HMD as well as two gloves by Bluetooth, the fingertips of which each corresponded to a different set of menus and widgets displayed in the virtual space.
As you look at the demo videos and the gloves involved in the UI, it’s hard not to see this as an exciting precursor to that back breaking holy grail that is the Minority Report experience or sifting through objects and media that simply hang in the air in front of the user. A key difference - apart from the level of development and how advanced the Hollywood version is - is that as mouth-watering as the Tom Cruise version of future computing is, it’s the Tinmith system that actually serves more of a practical purpose. It’s about modelling a space and being able to visualise architectural plans or landscaping in the actual environment where the building work is to be done. Whether it’s creating a residential garden or building the Olympic Park, it’s something that’s simply not possible or convenient to do on location with a keyboard and mouse.
“Tinmith is great for seeing a physical world relationship with something you’re trying to create because of the way you can change between the real to the virtual and any mix of AR in between. You can even change environments as well as the space, so you could use weather overlays to make sunny Adelaide look like a rainy day in London.
“Tinmith is perfect for working with objects you can’t see or ones that are far away which you can bring closer, but there’s also the opposite where you can get collaboration or ‘through walls viewing’.”
For the latter, the example Thomas gives is of a command centre sending instructions to a team of fire fighters outside trying to stop a blaze in the Australian bush. The blend of the virtual, supplied by the command crew, and the real, from the view of the men on the ground, could be invaluable not only just in communication from the team on the ground to HQ but also for the modelling of exactly what the fire fighters need to do to get the blaze under control and sending that information back out to them.
The major drawback of both Tinmith and ARQuake and number of solutions like it is that they require a lot of heavy, expensive kit to be worn by the user - something which Thomas and the WCL team have been quick to recognise.
“One of the biggest barriers is the HMDs. They don’t look good. People are image conscious, even at the workplace, and they don’t want to wear one of these. I can see them in situations where you need protective eyewear anyway but otherwise, not really. It doesn’t matter how functional the HMD is. It’s not going to overcome this problem.
“If you make the technology invisible, then people are more likely to use it. Just look at mobile phones. They used to be bricks before someone made them small enough to catch on.”
With this in mind, the WCL has spent the last few years tackling AR user interfaces in a different way. If backpacks and HMDs are the problem, the solution has been to lose both entirely by creating AR that anyone can see in a controlled, indoor environment where the computer can sit on a desk and the virtual overlays can all be done with projectors.
Spatial Augmented Reality, as Thomas and his team have called it, uses the projectors to add computer generated images to the real world to make an immersive environment which more than one person can appreciate at the same time. What’s more, tools used to interact with it no longer have to be confined to the traditional standards.
The demo video shows an air brush tool which, thanks to the tracking technology perfected in the years of previous projects, knows exactly where it is in the environment and which direction its facing with six degrees of freedom sensor on board. It can then be used to spray virtual graphics onto a real object just as if it were actual paint.
“Effectively, there are two boxes,” explains Thomas. “One is in the real world and the other in the virtual. The real box has sensors on it so, when it receives information from the air brush tool, the computer knows where to generate the graphics on the virtual surfaces which it overlays via the projector.”
Very clever stuff, which becomes even more mind boggling with the use of the air brush palate involving colours and, more impressively, stencils that you can even create yourself as you go. It’s not hard to work out the practical applications for product manufacture.
“Imagine having a new photocopier to design. First it’s virtual, then 3D and then the massive jump all the way to a really expensive $100,000 prototype. What I’m proposing is to design a photocopier in the same way from virtual to 3D and then project the designs onto a box which tells more about the overall shape and how it will work. You can pretend to press the buttons, open out the drawer and check it works in a real world environment on a 1:1 scale of reality at a fraction of the cost.”
Unsurprisingly, the WCL has had plenty of commercial interest in the project and Thomas and his team have taken the industrial design idea even further. Below is another way that the same technology can be used with a different tool this time to act as a guide to how an object should be manufactured. Here the 3D blueprints are projected onto a solid block of raw material where they act as a cutting guide for the object to be made.
While the applications of this kind of AR are obvious, the gesture-based tools - impressive as they are - are still limited and it’s here where Thomas has another team focusing its work.
“We want to come up with interfaces that are more compelling than a mouse and keyboard. Mice and keyboards are great if you want to write a document but not so hot for modelling shapes and organic objects”
The answer has been to come up with something that the WCL has called Digital Foam. The ideas is that you can create any shape out of a selectively conductive material which responds by providing life like moulds and indentations to a virtual surface of the same object.
“We innately like to touch objects,” explains Thomas. “We make gestures but this is more like actual manipulation of props.”
It’s perhaps harder this time to spot exactly how such an interface might be useful in the real world but according to Thomas, that’s not really the issue for the WCL.
“We don’t always know what the applications for these things are but we don’t have to. Someone else will come along with specialist knowledge of their field and say, ‘Wow, this will be great for dentistry!' or whatever it is.”
What Thomas does recognise though is that simplicity is the key. “Less is more” is the favourite phrase in the Wearable Computer Labs, so we’re told.
“A good interface is a minimal one. Sometimes elements of information might be all we need and it depends on at what point in the task. I see navigational AR on the street as three-mode - 60 per cent audio cues, 30 per cent head down with how many metres to go and only 10 per cent with full visual AR.”
As to whether this branch of technology can make it to the consumer in a meaningful way, Thomas seems to have little doubt with plenty of ideas of what could be possible in the near future.
“A navigation application that actually works will be the killer app. To be able to just go to any restaurant anywhere in the world with confidence, convenient instructions and very good visual cuing will be irresistible. At the moment there are sensor problems and compasses in phones aren’t very good but computer vision will overcome these.”
Advertising is another industry which has been quick to use AR but holding sheets and codes out in front of webcams for pop up pictures hasn’t impressed a man who saw this kind of thing over 10 years ago.
“They need to get over the gimmicks. We’ve got a $5,000 watch company using cheap paper AR to promote their product and that just doesn’t make any sense.”
Tissot's AR campaign might not have excited Thomas but one sector where the professor is quite clear that it’ll be done properly is in gaming.
“There are lots of avenues for gaming, especially since the arrival of 3DTV and cheap projectors. This is really a eureka moment that’s already here but perhaps not quite realised. They’ll need to extend the gaming community first but the next form of entertainment will be leveraging social networks.”
With the social revolution the biggest rise in internet usage in the last five years, there’s an enormous group of people out there ready to get on board - whether that’s on Facebook, Twitter or dedicated gaming platforms with their own burgeoning communities it doesn’t really matter, so long as both the hardware and connectivity is in place. So what kind of AR can we look forward to?
The grand idea, according to Thomas, is the concept of bringing your online friends into your lounge. AR can be used to virtualise them right into your living space and, at the same time, you into theirs, rather than having to meet them in some online forum or world.
“This isn’t Second Life. You want people in your space,” stresses Thomas. “My most enjoyable time with friends is having a cup of coffee with them, not being on Skype, and with the technology available right now, that reality is very, very close. Maybe it won’t be good enough for musical chairs just yet but there’s no reason why it can’t be done to sit down in a certain dedicated area.
“We already have so much and we don’t integrate them in a sensible fashion. The Internet becomes the backbone to everything to get around all that and, once we can combine 3DTV with table top computing, phones and tablets, we’ll be there, and it’s going to be absolutely massive.”
For more information on what Qualcomm is doing with Augmented Reality please click: http://www.qualcomm.co.uk/products/augmented-reality
And for more on AR Week head over to our AR Week homepage on Pocket-lint