Replicators in Europe

In the 7th Framework Program of the European Community a project has started in 2008, in which modular robots are developed by many research parties in Europe (Universität Stuttgart, Universität Graz, Universität Karlsruhe, Scuola Superiore Sant’Anna, Sheffield Hallam University, Fraunhofer Gesellshaft, Institut Mikroelektronickych Aplikaci, Ubisense, Ceske Vysoke Uceni Technicke v Praze and Almende B.V., see http://www.replicators.eu) that go beyond the swarm mode and are able to form robot organisms.  This is possible by sophisticated docking devices on several sides of the individual robot modules. By interlocking the docking units of a pair of robot modules, a rigid connection is made and robotic body forms – like snakes, scorpions, 4-legged spiders and wheels – are created.
The robot organisms can crawl around by shifting their center of gravity and are not restricted to the tiny screw drives or continuous tracks on the individual modules anymore. For that reason the Replicator robots can navigate through environments with obstacles that are impossible to overcome by robots that only have wheels. There is a wild range of applications to think of. A bucket of robots can be emptied in a collapsed mine (or a building after an earthquake). Then the robots start to assemble and to search for survivors. Or the robots swarm over the surface of a to be inspected planet in a NASA or ESA mission, a few fall in a cavity in the landscape and the rest comes to rescue, making it possible to continue to carry out the mission.

There have been several modular robot projects in the last decade, but this one is special because the individual robots are gifted with highly sophisticated sensors as has never seen before in modular robotics. In the current prototype each of the robot modules carries a laser scanner, infrared and ultrasound sensors, microphones and multiple cameras (and for example, a camera streaming images at a rate of 10 Hz, of a size of 256*256 pixels, coded by 3 8-bit RGB values amounts to a bandwidth of almost 16 Mbit/s.). A robot organism consisting out of say 10 modules has to cope with an extraordinary amount of sensor data. For that reason this type of modular robots, far more than before, needs elaborate sensor data processing and fusion mechanisms in place.

Sensor fusion is an advanced area of research that comes across new challenges when it is confronted with the agility of robots not just carrying the sensors around, but anticipating changes in perceptual input because they know they are driving their own selves around, and which are moreover also capable of proactive search for some preferred sensory input. This highly autonomous nature asks for sensor fusion techniques that enable the robots to categorize and classify the to be known world for themselves. The robots need to learn how another robot looks like, how a power outlet looks like, what sounds a robot typically makes, what sound a power outlet typically makes (none…) and preferably combinations of those different sensory modalities.

At Almende B.V. (http://www.almende.com) we implemented an adaptive resonance theory (ART) model to auto-classify visual-acoustic data. The acoustic data had to be preprocessed by an echo state network. This is a recurrently connected “reservoir” of neurons, where the only connections that can change are the ones that go to the output (or readout) neurons. So, within the reservoir we will have neurons having all kind of firing patterns, and we simply select at the output the ones we want to combine. The output of the reservoir will abstract representations of the original raw audio input. The visual data is also preprocessed. In this case a spatial attention mechanism is implemented using a combination of the saliency model of Itti and Koch and that of Frintrop (for performance purpose). Color, intensity and other characteristics can be used to have a certain blob of pixels “pop out” of the scene. This salient entities can subsequently be used by SIFT (scale-invariant feature transform) to come with a set of features (key points). Similar to audio, we now have an abstract representation of the original image input.

The visual as well as the audio abstractions are fed into a multi-directional unsupervised ARTMAP, an adaptive resonance theory (ART) variation. The reason that an ART variant is used, is because this allows the robot to learn fast new visual-acoustic objects without forgetting the old ones it already knows. Standard ARTMAP uses two ART models. A pattern to the supervisor network activates a node in the so-called “map field”, as well as an incoming pattern in the supervised network. If those nodes are different, the latter starts to search for a better match. The crux is to see both networks as presenting just another modality. The sound profile that is heard when a robot encounters another robot, can be used as “supervised” information for the visual profile, or the other way around. The actual implementation allows for dynamic binding of this information, or the robot will always expect a unique visual-acoustic object, while in the real world different visual objects might be able to make the same sounds, and the other way around. The necessity for this becomes clear in experiments in the physical realistic Delta3D simulator. A robot was wandering around and detected an interesting object, a power outlet! In the meanwhile the robot heard a sound from a robot, but sadly one that was sneaking up behind him! Thanks to the dynamic binding of visual-acoustic classes the robot learned later (by encountering silent power outlets) that this sound was not made by outlets at all, but by the robots in the arena. The robot is smarter than a gosling classifying its mother early in life… To go into more details about the implementation would lead too far for now. A lot of information can be found on the tech blog http://replicator.almende.com. Enjoy!