Pixelbots: Pixels with personality

About the Project

In today’s digital media landscape, we are constantly surrounded by displays, from the LCDs found on the phones in our pockets to the ubiquitous screens that greet us whenever we enter a store, airport or educational institution. This project aims to take these displays and enable human gestures, movements and abilties to be applied to them. This helps bring displays into a kinetic form and break the stereotypical rectangular or oval experience for digital displays. Pixels become mobile entities and their positioning and motion are used to produce a novel user experience.

This has been done through the use of robotic displays which basically comprises of a robot swarm where each robot can be remotely controlled and is equivalent to one mobile pixel with RGB LED for controllable color. This display can then be used to reproduce images or animations through a goal generation algorithm that computes a goal position for each mobile pixel. A control loop is then used to firstly allocate positions to each robot, compute a velocity and enable collision avoidance so that the mobile pixels can quickly reach their desired location.

The next step in the project is to then enable gesture based human control over this robot swarm. This has been done by providing two modes of interaction: free-form and shape contrained. In free-form interaction, users can select individual robots and change their positions or move them along a trajectory. In shape-constrained interaction, user can only control a subset of the degrees of freedom so that the configuration is maintained. This work can be extended to several essential applications in addition to entertainment such as surveillance or search-and-rescue operations.

Project Demonstrations

Funding & Partners

This project has received funding from the Netherlands Organisation for Scientific Research (NWO) Applied Sciences with project Veni 15916. Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the NWO. Neither the NWO nor the granting authority can be held responsible for them.

In today's digital media landscape, we are constantly surrounded by displays, from the LCDs found on the phones in our pockets to the ubiquitous screens that greet us whenever we enter a store, airport, taxicab, doctor's office, or educational institution. This plethora of displays both allures us and contributes to the media's saturation of our lives. The truth remains that we are never far from the next form of information display. Disney Research's Pixelbots takes this truth as an inevitability and brings the display into a kinetic form, breaking the screen out of the confines of a rectangular or oval experience. Billed as "Pixels with Personality," the Pixelbots' enticing presence rests on our ability to project human qualities onto objects that move as we do in physical space.

A taxonomy for gesture-based interaction between a human and a group (swarm) of robots is described. Methods are classified into two categories. First, free-form interaction, where the robots are unconstrained in position and motion and the user can use deictic gestures to select subsets of robots and assign target goals and trajectories. Second, shape-constrained interaction, where the robots are in a configuration shape that can be modified by the user. In the later, the user controls a subset of meaningful degrees of freedom defining the overall shape instead of each robot directly. A multi-robot interactive display is described where a depth sensor is used to recognize human gesture, determining the commands sent to a group comprising tens of robots. Experimental results with a preliminary user study show the usability of the system.

Real-time interaction with a group of robots is shown with a hand-held tablet, which tracks the robots and computes collision-free trajectories. Efficient algorithms are described and experiments are performed in scenarios with changing illumination. Augmented reality and a multi-player setup are also described.

About the Project

Project Demonstrations

Funding & Partners

Related Publications