Adding Physical Objects to an Interactive Game Improves Learning and Enjoyment

This delightful contribution explores EarthShake, a mixed-reality game that helps children learn some basic principles of physics by bridging the physical and virtual worlds via depth-sensing cameras.

The work includes not only an interactive prototype that is put to the test by 4-8 year old children (a particularly demanding user demographic if ever there was one!), but also through careful experimental design that teases out many insights illustrating how and why the use of three-dimensional (3D) physical objects in mixed-reality environments can produce better learning and enjoyment than flat-screen 2D interaction.

Computer technologies can be especially empowering when brought to bear in the context of the physical environment. This has long been suspected as a benefit of so-called “tangible interfaces”—that is, interfaces employing physical stand-ins or props as proxies for digital objects—yet precisely how, or why, or under what circumstances tangibles might bring benefits has remained murky, particularly when combined with mixed-reality environments, i.e. sensing systems that detect the 3D world and incorporate it directly into the interactive experience. One can hypothesize many possible reasons that tangibles could be beneficial to learners in mixed-reality environments:

Is it the three-dimensional nature of the objects?

Do the potential benefits derive from making interaction more enjoyable?

Or perhaps it is the embedding in reality, and the sensory cues that the real world affords, that forms the critical difference—as compared to watching videos of the same activities, for example.

In addressing these questions, the carefully controlled studies isolate various possible effects and confounds, and thereby convincingly demonstrate many aspects of exactly how these mixed-reality environments benefit learners. The results demonstrate that learning benefits accrue through embodied cognition, improved mental visualization (as evidenced by children’s hand gestures, for example), and via the mere observation of physical phenomena in the full richness of sensory cues available in the real world—cues that are inherently absent when watching a video recording of the same activity on a flat, two-dimensional screen.


Nesra Yannier, Scott Hudson, Eliane Wiese, Ken Koedinger. 2016. Adding Physical Objects to an Interactive Game Improves Learning and Enjoyment. ACM Trans. Comput.-Hum. Interact. 23, 4, Article 26 (August 2016), 33 pages.