iSpy

iSpy: Automatic Reconstruction of Typed Input from Compromising Reflections
Rahul Raguram, Andrew White, Dibyendusekhar Goswami, Fabian Monrose and Jan-Michael Frahm.
ACM Conference on Computer and Communications Security (CCS) (Oral Presentation), Oct 2011.
[journal (pdf)] [conference (pdf)] [video]

Press: Engadget, Gizmodo, New Scientist, PhysOrg

Today, personal mobile devices, such as smartphones, are all around us. While the ubiquity of these powerful personal computing devices has changed how we communicate and store information, it also provides new possibilities for the surreptitious observation of private messages and data. In this work, we demonstrate the use of computer vision and machine learning techniques to compromise the privacy of users typing on virtual keyboards. Specifically, we show that so-called compromising reflections (in, for example, a victim’s sunglasses) of a device’s screen are sufficient to enable automated reconstruction, from video, of text typed on a touchscreen keyboard. Despite our deliberate use of low cost commodity video cameras, we are able to reconstruct fluent translations of recorded data, even in very challenging scenarios. We believe these results highlight the importance of recalibrating our expectations of privacy in response to emerging technologies.