Abstract—Multi-touch has become the dominant interaction technique on shared displays, such as interactive tabletop surfaces. Alternative techniques include in-air gestures, interactive pens, and conventional pointing devices such as a mouse. A theoretically appealing but less explored approach is tangible interaction. Tangibles are physical objects that can act as both control and representation for the underlying system, allowing users to create, access, and manipulate digital information. Tangibles can offer a comparatively richer interaction experience by providing additional sensory information, such as pressure and friction, and by extending the interaction and display space, for instance through off-screen content control or feedback. One limitation, which often discourages researchers and designers from using such techniques, is the need for additional hardware or devices (as tangibles). Since touchscreen smartphones are gradually becoming ubiquitous, the possibility of using such devices as tangibles may encourage researchers to explore the matter further. In this hands-on tutorial, we will discuss and explore how touchscreen-based smartphones can be used as tangibles to interact with shared displays, and participants will be guided through the process of designing and prototyping their own mobile-based tangible interactions on an interactive tabletop surface.

Ali Mazalek, Ahmed Sabbir Arif. 2014. Mobile-based tangible interaction techniques for shared displays. In Proceedings of the 16th international Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '14). ACM, New York, NY, USA, 561-562. DOI: http://dx.doi.org/10.1145/2628363.2645668

Date: Tuesday, September 23, 2014
Time: 9:00-12:00

• Tutorial Paper
• Presentation Slides

Tutorials MobileHCI 2014

Organizers

• Ali Mazalek, Ryerson University, Toronto, Ontario, Canada
• Ahmed Sabbir Arif, Ryerson University, Toronto, Ontario, Canada

Audience & Background

This tutorial is intended for mobile HCI enthusiasts who are interested in exploring mobile-based tangible interactions. Participants must have experience developing with HTML5 and JavaScript. In addition, a basic understanding of gesture recognition and touch-based interaction approaches is useful. Participants are expected to bring their own laptops for development.

Structure

The tutorial begins with an overview of the concept of tangible interaction based on recommended reading materials, provided by the organizers ahead of time. Open discussion addresses design possibilities and practices for mobile-based tangible interactions. Participants then divide into groups of four, and each group is provided with two Android mobile devices, a development toolkit containing a gesture recognizer, JavaScript libraries, and sample source code. They are also provided with several example scenarios, where the use of mobile devices as active tangibles may benefit users. Refer to the tutorial paper for exmaples.

Schedule

• 9:00—9:10 AM: Introduction
• 9:10—9:25 AM: Tangible Tabletops and Mobile Devices
• 9:25—9:40 AM: Interaction Scenarios and Demonstration
• 9:40—10:30 AM: Setup and First Exercise (download a simple solution to the first exercise )
• 10:30—10:50 AM: Coffee break in hallway
• 10:50—11:45 AM: Design and Development Activity
• 11:45—12:00 PM: Presentation and Discussion

Reading Materials

• Kaltenbrunner, M. reacTIVision and TUIO: A tangible tabletop toolkit. Proc. ITS 2009. ACM (2009), 9-16.
• Kurtenbach, G. and Buxton, W. User learning and performance with marking menus. Proc. CHI 1994. ACM (1994), 258-264.
• Steimle, J., Khalilbeigi, M., Mühlhäuser, M., and Hollan, J. D. Physical and digital media usage patterns on interactive tabletop surfaces. Proc. ITS 2010. ACM (2010), 167-176.
• Ullmer, B. and Ishii, H. Emerging frameworks for tangible user interfaces. IBM Systems Journal 39, 3-4 (2000), 915–931.
• Wobbrock, J. O., Wilson, A. D., and Li, Y. Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. Proc. UIST 2007. ACM (2007), 159-168.
• Wu, A., Mendenhall, S., Jog, J., Hoag, L.S., and Mazalek, A. A nested API structure to simplify cross-device communication. Proc. TEI 2012. ACM (2012), 225-232.

Prepare Your Laptop for the Tutorial

• Set up local web server
  • Instructions for Windows and Mac
• Install node.js and npm. node.js is a platform for building fast and scalable network applications, and npm is the default package manager for node.js
  • How to install node.js and npm
  • Make sure to intall all the dependencies using command $ npm install
• Install tuio.js, a JavaScript implementation of the TUIO library for multitouch and tangible interaction in the web browser
  • $ npm install tuio
• Install socket.IO, a JavaScript library for realtime web applications
  • $ npm install socket.io
• Download reacTIVision fiducial markers

Sample Source Code

• Download the sample source code from here
• Unzip it and move it to your localhost folder (on Mac ~username/sites)
• Follow the instructions in the tuio/example_fileshare/readme.txt file
• Download a simple solution to the first exercise from here