To use 3D or not to use 3D? That's the question. Which properties are the most important for robot vision? The natural world sheds light on the answer.

Does my robot really need 3D vision? What type of vision system do I need? These are questions which come up a lot within robotics. Some people say that 3D is necessary because humans have stereo vision. But, is 3D vision really better than 2D vision?

Some recent research provided a new perspective on this issue, from an unlikely source. Research into how the praying mantis insect sees has shed some light on which properties we are most important for robot vision.

How praying mantises see in 3D

Photo by Jared Belson
It turns out that mantises have a very efficient and accurate 3D vision system.

An intriguing article caught my eye last year about 3D vision. It was some new research into the amazing visual system of the praying mantis insect.

Researchers from Newcastle University found that the insect does see in three dimensions, but not in the normal way (i.e the way that we see in 3D). Normally, stereo vision is obtained by taking two images of the same scene from two slightly different locations. These two images are combined by the brain — or by the computer in robot vision — and the differences between them are used to extract depth information using the distance between objects in both images.

Like humans, praying mantises have two eyes which face forward. They can also precisely detect depths for catching their prey, as humans and other predators can. As a result, you might think that they would use the same visual system as other predators use. But, you'd be wrong.

It turns out that mantises have a very efficient and accurate 3D vision system. However, the system only works properly in one specific situation: when the objects it sees are moving (e.g. its prey) and when both of the insect's eyes see the same object. When objects are not moving, it's possible that the praying mantis doesn't see 3D at all (the researchers didn't test this so we can't be sure). Perhaps they just see in 2D.

When each of the insect's eyes is shown a different, unrelated moving image — which wouldn't happen in nature — the mantis will inaccurately detect 3D moving objects. This could be similar to an optical illusion in humans, where our visual system shows us things which aren't there.

The researchers also found that the mantis was easily able to detect camouflaged objects and wasn't distracted by changes of light. In these situations, the mantis performed much better than humans.

In an article for ZDNet, robotics journalist Greg Nichols discussed how these new insights can be applied to robotics. Because the mantis's vision is very efficient (correlating to its very small brain) this type of stereo vision could be perfect for low computation robots.

What we can learn by looking into a mantis's eyes

There is a useful lesson that we can learn from the praying mantis:

Match the type of vision system to the specific needs of the task.

The praying mantis insect has a specialized vision system which works very well for the specific application for which it is needed, i.e. catching moving prey which comes close to it. However, the mantis's 3D vision would probably perform badly in other situations. For example, it wouldn't be able to detect the position of a static pen to be able to pick it up. Humans are very good at this, but we find it hard to detect camouflaged moving objects.

In robotics, it's not a good idea to invest in the most complex, expensive vision system in an attempt to have a system which will work in every situation. The reality is that there is no vision system which works in every situation!

The vision system of a praying mantis works better than human vision in some situations but worse in other situations. This is also true for robot vision.

3D robot vision vs 2D robot vision

The research was specifically looking at the 3D vision capabilities of the insect.

In robot vision, it's sometimes thought that 3D vision is better than 2D vision. Some people think that it's only used less because it's more expensive. However, this is also not true. Just as the mantis's vision performs better in some situations than others, 3D vision can perform worse than 2D in some situations. For example, 3D requires more complex algorithms so can end up slower, it can be less robust to detecting some materials, etc.

Each of the Top 10 Challenges for Robot Vision will affect different vision technologies to varying degrees. Some challenges, like "background" will affect 2D more than 3D. Other challenges, like "movement" might affect 3D more.

• Read more: Top 10 Challenges for Robot Vision

5 insect-inspired properties that will affect robot vision

What properties are going to be important for your robot application?

Let's have a look at 5 of the top properties, inspired by the visual capabilities of our praying mantis.

1. Speed of detection

The praying mantis needs to detect prey at ultra-quick speeds. Although most robots don't need to be so rapid, detection speed is still very important.

Faster robot vision means that you can reduce the cycle time. In turn, this improves the productivity of your robot cell.

2. Background robustness

One of the amazing things about the mantis's vision is its ability to detect very camouflaged moving objects.

Robot vision relies on having a good contrast between the object in the foreground and the background. The best vision solutions are robust to different backgrounds and can detect objects on almost any material.

3. Image definition control

The resolution of the praying mantis is quite poor, as with many insects. They are able to function despite seeing in very low definition.

The required resolution of robot vision depends on the needs of your task. Sometimes, detection algorithms are more robust with lower resolutions. Other times, high resolution is necessary. The best systems allow you to choose the resolution based on your needs.

4. Lighting robustness

The praying mantis is able to see in very bad and changeable lighting situations. In fact, their vision seems to be entirely based on movement and not on light differences, as our vision is.

Robot vision is often very sensitive to lighting changes. Therefore, the best systems introduce some robustness to lighting, such as active and strobed lighting.

5. Useful object detection

The praying mantis has a perfect example of a vision system which detects exactly what it needs to detect (moving prey and predators) and little else.

Robot vision is only useful if it can detect what you need it to detect. This is why features like easy-to-program template matching are so important. You want to be able to quickly program your desired objects, press play, and let the robot get on with it.

Which properties are most useful for your robot vision application? Tell us in the comments below or join the discussion on LinkedIn, Twitter, Facebook or the DoF professional robotics community.