05 November 2009
Dr Dave Merchant

Last month we looked at how computer software can help with everything from design calculations to fire and crowd control, by accurately simulating engineering properties, fluid dynamics and even human behaviour. These "no humans involved" (NHI) simulations are involved in every aspect of your life, from the crash-worthiness of a car to the width of exits from a football ground. But the fundamental problem with NHI is that it doesn't cope well with complex moral and emotional responses.

Computer-generated "agents" can do a good job of mimicking basic emotions, such as fear, bravery or indecision. But in many simulations, chaos theory applies; a very small change in the actions of a few people in a crowd dramatically changes the result.

A "brave" person may be more likely to go back into a burning building to save someone (and we can model that just fine), but a mother's actions as she tries to get several children out of a plane cabin is far harder to put into code. The fact you may have argued with someone in the ticket queue will affect your decision to help them in extremis, even if you'd deny it later. If your phone rings, what you do depends on whose name is on the screen. To bring these nuances into an NHI program we'd need to simulate the entire back-story of each agent. It's easier when we have a ready-programmed version up and running: you.

Virtually you

In avatar-based simulated learning (s-learning), each computerised character is controlled by a real human, essentially as if they're playing a video game. Avatars may represent human figures, or may not, but they should act like humans.

It actually depends on who that human is, and what they're playing, because players they certainly are, even if the simulation is deadly serious and, crucially, they know they're playing. They know they can't be killed by the fire, and unless they're proficient gamers they can find it difficult to control their avatars realistically in high-stress situations, which is precisely what we want them to experience. There's often a difference between what they would do, and what they think they'd do; in the game, we see the latter.

We need to be very careful to design the simulation around that issue, so people are either given lots of training in how to "play", or the interface is simplified so it becomes intuitive. We can, for example, allow the software to control the character up to a certain point while a human watches, then pause and ask the human to decide between two actions before running a bit more code.

The agent never learns what you're thinking, but knows enough to follow orders. It may sound like we've just taken the tabletop exercise and put it on a computer screen, but by actively simulating the response to each decision we allow for any off-the-wall thing they may decide to do ("release the attack squirrels!"), and can work out where a crowd of 10,000 will end up in an hour because of it. If you're sitting round a desk, someone has to work that out by hand, and they can't. That's why we're simulating it.

In some cases, the players can avoid waggling joysticks, if in the real scenario they would only be giving orders from afar - the best example being public order training for police commanders. They make verbal decisions; the simulation reacts, showing them dots on a map or faked CCTV images, while an instructor hidden in a corner types in the instructions. For the players, it's 100% realistic.

Simulations are quite costly for small-scale users, so we can fake it, creating a whole branching set of alternate video clips, chosen by the viewer. All the clips are pre-defined, but the outcome seems to evolve with your choices. "Interactive" health and safety software and DVDs use this branching system, but it's not a simulation - you only think you have free will. For true freedom, we need to go elsewhere.

It's another world

The University of Derby owns a quarry: six and-a-half hectares of hole, full of plant and machinery, explosives and people in hi-vis clothing, where they train students in partnership with the Institute of Quarrying. Most weeks, a few of them are crushed, buried, blown apart or vanish without a trace; and nobody cares. The HSE has never investigated, no environmental groups objected. Nobody's complaining because it's in-world. Not in the world. The second one.

Second Life, the online virtual world created by Linden Lab in 2003 is, to some residents, far more important than the one with the soil and air and chickens they were born into. Things that happen inside the game (in-world) are often tied to things happening in reality (off-world), and to many it's no longer just a game. Careers are run, off-world fortunes are won and lost, marriages are made and broken (sometimes in-world and sometimes not!), and many companies are now entering Second Life to replicate their off-world products and services. Many colleges and universities have in-world campuses (buildings, islands, arenas, etc) for recruitment, public education courses and seminars - see the blog at http://virtualworldwatch.net for some of what's going on in the UK.

Derby's quarry is there, on their own private island, and students get to "play" at being in charge while the lecturers (and other students) each control their own avatars - some better-behaved than others.

If someone reverses a truck too close to the face, it falls over and they die. If someone else happens to be under it, they die too. Then they both get to try again. Of course the students are fully aware they're in a game, and they aren't using 3D headsets or a Holodeck, but with careful scenario design and supervision by the lecturers they can gain experience of the workflows, and be allowed to make mistakes.

Blasting a virtual quarry face does not qualify you to blast a real one, but it does mean you can try it 30 times a day, and see what happens if you mess up the shock tubes. The limiting factor at the moment is the avatar control problem; the students aren't typical gamers, and Second Life isn't realistic enough to allow complex tasks (for one, you have no hands), so occasionally someone will shoot off into the air because they pressed the Fly button. But Derby is using Second Life for convenience, and there's no technical reason why the "game engines" responsible for commercial titles can't be put to use in training and simulation.

We've been using specially written programs to create the views from the "windows" of flight simulators for decades, and Liverpool John Moores University has a 360° ship simulator in their Lairdside Maritime Centre that can sail their three real bridge rooms into any port on the planet. You can even hire the place for team-building exercises, and once you've tried to thread a supertanker through the Panama Canal, you won't look back. (Though, of course, with a 360° view, you can if you want to.)

Towering example

Making a new game, or a custom level for an existing game, is far easier than it used to be. Creating something that trains people in health and safety, forklift driving or crowd control is no different to avoiding aliens or stealing cars, and several docu-games have been written based on real events. In one infamous example, players had to escape from the World Trade Centre during the 9/11 terrorist attacks. It was so realistic it sparked an outrage and was dropped, but if it had come out before 9/11 it'd just have been another game, and one with real uses in training.

Docu-games rely on the high-quality graphics of modern PCs and consoles to influence your emotions as realistically as possible, but the game play is based on moral and emotional decisions rather than how fast you can fire a gun. This is ideal for users with little or no gaming experience, but they still rely on predefined branching scenarios rather than true simulations.

As development costs come down, the market for a game no longer needs to be particularly huge - but it's still a video on a screen. Your character can walk about, but you can't step inside. If you want a Holodeck-style immersive experience, you'll have to wait at least 20 years - but if you want to immerse the computer's world into yours, we can do that today.

Augmented reality (AR) adds computer-generated stuff to the real world you're looking at, with software aware of your viewpoint so you can move about and the AR content stays where it seems to be. First developed in the mid-90s to help engineers at Boeing, it's a halfway point between reality (seeing the world around you as is) and virtuality (seeing a totally computer-generated image, as in a game).

You're used to one form of AR - the ads seemingly painted on sports pitches on TV are all computer-generated by software that knows the location and direction of the camera and can work out where on-screen the image should appear. It's good enough so most people still think they're done by a guy with a paintbrush, and players can run "over" them - but most importantly if the sponsor changes, we just click a mouse. International matches can have different logos for each TV network, but of course you still don't get to step into the TV screen and walk about. For that, we need to use your own eyes as the camera - or rather replace them. Relax, this might sting.

Immersive 3D AR relies on a head-mounted display (HMD) with a camera and a tiny video screen in front of each eye, which starts by showing what the camera sees (so the HMD is effectively not there). Computer software analyses what you're looking at (either using GPS and direction info from the HMD, or image recognition) and adds things to the views in 3D.

Immerse yourself

Originally developed for the military (the new F-35 joint strike fighter uses AR linked to external cameras, so the pilot can see "through" the fuselage), immersive AR is increasingly used for training and to allow intuitive interaction with complex data - using the Mirage system from Canadian company Arcane Technologies, surgeons can see MRI images of a brain projected onto a real model of a head sitting on their desk (or even a real head), pick it up and turn it round, juggle with it, or slice it in half. In psychological therapy, Mirage assists in phobia treatment by exposing the patient to an AR spider scampering about at the command of the therapist's mouse, realistic enough for the patient to feel genuine anxiety but in a safe way. They can reach out and (almost) touch it.

In the workplace, AR can show in-vision instructions to an engineer repairing a complex machine, helping them follow the process better than referring back to printed booklets, or by fitting infra-red cameras it can let them see temperature - added over their normal visible-light vision rather than replacing it. AR could make hot objects flash red and add numeric values, but unlike with the "full-replacement" views from thermal imagers and night-vision, the person wearing an AR headset can move normally without losing their sense of balance or direction, and look down to write something on their clipboard.

At present, AR is limited by machine vision: the ability of the software to understand where it is, based on just the camera images. Some programs need to be helped by a walk-through in learning mode, some need marker dots to accurately identify the rotation of things against an unknown background, but research into machine vision is moving at enormous speed and in a few years it'll be able to see without help.

Professional AR systems use differential GPS or take external cues for their position. LifeClipper3, developed at the University of Basel, is a particular favourite of mine as it uses QR code stickers on objects of interest to calculate the HMD position as the user wanders about the city. Lifeclipper2 uses dGPS and attitude sensors for the same thing. Want to "see" the contents of every pipe in your refinery, including real-time sensor data and repair instructions?

Dive in

The CAVE system, developed at the University of Illinois and now available commercially, uses six projectors to display stereoscopic images on the faces of a cubic room; sensors control the images by detecting the position of special goggles. This allows users to move round an object seemingly in the middle of the space. Fans of CSI will know the idea from its virtual autopsy lab. The CyberWalk project has the closest thing to a Holodeck - a room where the user's position is tracked by cameras using marker balls on the HMD, so it's accurate to a few millimetres. The players' vision is entirely virtual, and a 40-tonne system of 2D conveyor belts allows them to walk about forever in any direction while remaining in the middle of the room. It's a one-person system, but it's the first step to replacing an avatar with a real human (at least until you reach some stairs). The views through the headset are nowhere near photo-realistic yet, but in a decade they will be - since improvements in graphics are driven by the billion pound gaming industry. We're even developing systems to let you feel the things you see, by limiting your movements through mechanical gloves and oversuits. Combine that with ROV robots and you can be anyplace you want to be, no commuting involved.

S-learning does raise an important question for the health and safety community, and the lawyers. You're perfectly happy that an airline pilot has "learned" on a simulator, mostly because you know they sit on the right for a while - but would you let someone drive a car alone if they've only ever done it in a computer game? Does a simulated fire drill count? We're only a few years off the ability to do both of those in a computer just as completely and accurately as in the real world, but when we get there the definition of "competent" is going to need some work. We'll probably meet in-world to argue about it.

Categories:
Training, Professional Skills, Features

Related articles: