Conceptually the work is based on sensory augmentation and interaction focuses on gaze based techniques and physiological sensing. It builds on context-awareness (Schmidt, 2013) and augmented reality.

Augmented Reality

Mostly, the term “augmented reality” refers to extending the world with visual information, but augmented reality can also be used to augment other human senses, for example, tactile augmentation (Mateevitsi et al., 2013), sonic augmentation (Bederson, 1995), or taste or smell augmentation (Narumi et al., 2011). Applications are to help blind persons navigate (Heyes, 1984 and Johnson et al., 2006), to provide assistance in a working environment (Bannat et al., 2008), or just to retrieve information about buildings or sights while traveling (Feiner et al., 1997). This visual information can be provided in the real world using a projector (Pinhanez, 2001) or using a head-mounted display (Starner et al., 1997). The project aims to build on work in augmented reality, but focuses on advancing interaction concepts and intuitive control. Current augmented reality research strongly concentrates on the technical realization of the presentation.

Technologies to Extend Visual Perception

Systems can be designed to map information outside the visible spectrum to a range that can be perceived by humans, e.g. Schönauer et al. (2013) present a system that combines the near and far infrared spectrum into a single view using a depth sensor and a thermal camera. The visual output cannot be interactively controlled and is designed to aid perception for fire fighters. In the automotive domain night vision cameras are integrated into cars to provide an alternative representation of the environment. Such a presentation aids the driver by providing a sharper and more detailed image of the road, especially when visibility is poor, such as at night or in adverse weather conditions (e.g., fog). Other research projects explore the expansion of the field of view, e.g. using an ultra-fast-time-of-flight imaging system, as described in Velten et al. (2012). An approach to overcoming occlusion and extending the view through walls is shown in Colley et al. (2014). These examples show that capture technologies are available to amplify human vision. The proposed project’s aim is to utilize such technologies to create new sensory experiences. No research thus far has explored intuitive and natural interaction with such technologies.

Sensory Augmentation and Substitution

Sensory substitution allows one sense to provide information that normally is provided by another sense. Sensor argumentation provides humans with new extended senses by overlaying an existing sense with new information from another source. This area has been extensively researched for people with disabilities, in particular blind people. Several approaches show that systems can be designed where visual perception is substituted by other senses. The tongue has a dense sensing resolution for tactile stimuli, and researchers have developed electrotactile tongue displays (Dublon, 2012) as vision prosthetics that allow blind persons to navigate in natural environments. It is also possible to substitute sound for visual perception, as shown in Ward et al. (2010), where a video stream is translated into a sound pattern. This example also shows that users were able to develop a detailed visual phenomenology within months of immersive use of the system. Nagel et al. (2005) demonstrate the feasibility of creating a new sense for orientation by augmenting the somatosensory system. Using vibrotactile stimulation around the waist to represent orientation information, some participants revealed qualitative changes in sensory experience. Other research has explored the creation of a sense for proximity using a helmet with tactile output (Bertram, 2013) or a specially designed suit. These research results hint at the basic feasibility of the proposed research; however, some of the results are inconsistent and many lack rigorous evaluation. The proposed project will take a systematic research approach to create prototypes as well as a conceptual and theoretical foundation.

Eye-Gaze Interaction

By observing the movement of the human eye, the gaze direction can be detected. Such observation can determine what a participant is looking at, how long the person’s gaze dwells on a specific region, or in what order the person focuses on certain areas. Early eye-tracking systems focused on disabled users and today many research efforts aim to support users with serious disabilities (Majaranta and Räihä, 2002). Different solutions for interaction with standard user interfaces based on eye-tracking technology (e.g., Ware and Mikaelian, 1986; Jacob 1990) have been studied. Typical scenarios in which eye tracking is used include gaze-supported pointing and target selection on (large) screens (e.g., Ware and Mikaelian, 1986; Stellmach and Dachselt, 2012). Gaze gestures offer an alternative input modality to control systems by performing certain predefined (relative) eye movements to trigger certain activities in desktop (Drewes and Schmidt, 2007) or mobile environments. These examples show that tracking a user’s eye gaze is a promising approach for creating an intuitive control mechanism for amplified senses.

Brain Computer Interaction for Communication and Control

Using EEG signals for explicit control has long been a topic of research especially to aid disabled or locked-in patients. Applications include classifying EEG signals to control a wheelchair (Tanaka et al. 2005, Rebsamen et al. 2006, Millan et al. 2009) or to move a cursor (Kalcher et al. 1993). These are only a few examples of the many brain-computer interface (BCI) applications in the control domain. P300 spellers are also one widely researched application of EEGs where the EEG detects P300 signals and uses them to spell a letter on a virtual keyboard (Farwell et al. 1988, Serby et al. 2005, Kanoh et al. 2011).
Beyond the medical field commercial and open source BCI devices have emerged. e.g., Neurosky, Emotiv, OpenBCI. Samadi et al. (2014) purposed a hands-free system to solve a Rubik’s cube where a robot manipulated the cube according to users’ EEG signals. NeuroPhone (Campbell et al. 2010) and ThinkContacts (Perkusich et al. 2011) are mobile phone BCI applications designed to help users dial phone numbers only via utilizing P300 EEG signals. In this project we will build this work and utilize EEG signals in implicit and explicit interaction with other modalities such as gaze tracking as a source of input to control information display or seamless task switching.

Measuring Physical Activities through Physiological Sensing

Modern electromyography (EMG) measures muscle activity and supports the design of muscle-computer interfaces. Saponas et al. (2008) show that it is possible to detect different gestures through EMG sensors worn on the forearm using widely known machine-learning algorithms. Such a wristband could be worn like jewellery or integrated into clothing. Benatti et al. (2014) present a low-cost approach to include EMG in smart clothes to detect hand gestures. With minimal physical action, basically just changing muscle tension, systems can be controlled. In addition to explicit interaction, implicit information can be collected, for example, measuring the activity of the Trapezius muscle can help determine a user’s current stress level (Wijsman et al., 2013). Additionally, Gibert et al. (2009) show that facial EMG can be used to detect facial expressions and thereby emotions such as anger or happiness. EMG around the eyes offers an alternative way to detect eye gaze activity. Overall, these examples show that EMG can be used for implicit and explicit interaction and it is expected that EMG interfaces can be designed in a natural and intuitive way.

Measuring Cognitive Activities through Physiological Sensing

The use of physiological sensing to measure physiological response and cognitive activities is a widely researched field. In particular, the use of EEG signals to assess and classify cognitive tasks has been explored and a basic feasibility is shown; see Hosni et al. (2006) and Sahami et al. (2014). Implicit interaction using eye gaze can help to speed up task switching (Kern et al., 2010) and that it can be used as a means to classify perception activities to move towards a cognitive quantified self (Kunze et al., 2013). In this project, we will build on this work and explore the use of gaze tracking as a means for implicit interaction with a large amount of content on a large physical space. Additionally, we plan to extend our work on task classification based on combining EEG and gaze tracking as an input modality for seamless transition between tasks.

Electro-Stimulation and Pain as Output

Electrostimulation is widely used in the medical field (TENS devices). In the context of virtual reality Kruijff et al. (2006) have describe neuromuscular electrical stimulation for pseudo-haptic feedback and Tamaki et al. (2011) have shown that stimulation can be an effective output channel to induce gestures. In the artistic community, e.g. http://en.wikipedia.org/wiki/PainStation, the use of pain as output media has be used to create motivation. In this project we will explore how minimal neuromuscular electrical stimulation, at the perception threshold, can be used as stimulus.

References

Bannat, A., Wallhoff, F., Rigoll, G., Friesdorf, F., Bubb, H., Stork, S., … & Zäh, M. F. 2008. Towards optimal worker assistance: a framework for adaptive selection and presentation of assembly instructions. In Proc. 1st Intern. Cotesys Workshop.

Bederson, B. B. 1995. Audio augmented reality: a prototype automated tour guide. In Conf. Companion CHI ’95, ACM, pp. 210-211. DOI=10.1145/223355.223526

Bedri, H., Feigin, M., Everett, M., Filho, I., Charvat, G. L., & Raskar, R. 2014. Seeing around corners with a mobile phone?: synthetic aperture audio imaging. In SIGGRAPH ’14 Posters, ACM, p. 84:1, DOI=10.1145/2614217.2614282

Benatti, S., Benini, L., & Farella, E. 2014. Towards EMG control interface for smart garments. In ISWC ’14 Adjunct, ACM, pp. 163–170. DOI=10.1145/2641248.2641352

Bertram, C., Evans, M. H., Javaid, M., Stafford, T., & Prescott, T. 2013. Sensory augmentation with distal touch: the tactile helmet project. In Proc. Biomimetic and Biohybrid Systems, Springer, pp. 24-35. DOI=10.1007/978-3-642-39802-5_3

Campbell, A., Choudhury, T., Hu, S., Lu, H., Mukerjee, M. K., Rabbi, M., & Raizada, R. D. 2010. NeuroPhone: brain-mobile phone interface using a wireless EEG headset. In Proc. MobiHeld ’10, ACM, pp. 3-8. DOI=10.1145/1851322.1851326

Colley, A., Koskenranta, O., Väyrynen, J., Ventä-Olkkonen, L., & Häkkilä, J. 2014. Windows to other places: exploring solutions for seeing through walls using handheld projection. In Proc. NordiCHI ’14, ACM, pp. 127-136. DOI=10.1145/2639189.2639226

De Luca, A., Weiss, R. & Drewes, H. 2007. Evaluation of eye-gaze interaction methods for security enhanced PIN-entry. In Proc. OZCHI ’07, ACM, pp. 199-202. DOI=10.1145/1324892.1324932

Drewes , H. & Schmidt, A. 2007. Interacting with the computer using gaze gestures. In Proc. INTERACT ’07, Springer, pp. 475-488. DOI=10.1007/978-3-540-74800-7_43

Dublon, G., & Paradiso, J. A. 2012, May. Tongueduino: hackable, high-bandwidth sensory augmentation. In CHI EA ’12, ACM, pp. 1453-1454. DOI=10.1145/2212776.2212482

Farwell, L. A., & Donchin, E. 1988. Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalography and clinical Neurophysiology 70(6), pp. 510-523. DOI=10.1016/0013-4694(88)90149-6

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. 1997. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies 1(4), pp. 208-217. DOI=10.1007/BF01682023

Gibert, G., Pruzinec, M., Schultz, T., & Stevens, C. 2009. Enhancement of human computer interaction with facial electromyographic sensors. In Proc. OZCHI ’09, ACM, pp. 421-424. DOI=10.1145/1738826.1738914
Heyes, A. D. 1984. The Sonic Pathfinder: A New Electronic Travel Aid. J Vis Impair Blind 78(5), 200-2.

Hosni, S. M., Gadallah, M. E., Bahgat, S. F., & AbdelWahab, M. S. 2007. Classification of EEG signals using different feature extraction techniques for mental-task BCI. In Proc. ICCES ’07, IEEE, pp. 220-226. DOI=10.1109/ICCES.2007.4447052

Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., … & Eiderbäck, B. 2003. Technology probes: inspiring design for and with families. In Proc. CHI ’14, ACM, pp. 17-24). DOI=10.1145/642611.642616
Jacob, R.J.K. 1990. What you look at is what you get: eye movement-based interaction techniques. In Proc. CHI ’90, ACM, pp. 11-18. DOI=10.1145/97243.97246

Johnson, L. A., & Higgins, C. M. 2006, August. A navigation aid for the blind using tactile-visual sensory substitution. In Proc. EMBS ’06, IEEE, pp. 6289-6292. DOI=10.1109/IEMBS.2006.259473

Kalcher, J., Flotzinger, D., & Pfurtscheller, G. 1993. Graz brain-computer interface: an EEG-based cursor control system. In Proc. EMBC ’93, IEEE, pp. 1264-1265. DOI=10.1109/IEMBS.1993.979127

Kanoh, S., Miyamoto, K., & Yoshinobu, T. 2011. A P300-based BCI system for controlling computer cursor movement. In Proc. EMBC ’11, IEEE, pp. 6405-6408. DOI=10.1109/IEMBS.2011.6091581

Kern, D., Marshall, P. & Schmidt, A. 2010. Gazemarks: gaze-based visual placeholders to ease attention switching. In Proc. CHI ’10, ACM, pp. 2093-2102. DOI=10.1145/1753326.1753646

Kruijff, E., Schmalstieg, D., & Beckhaus, S. 2006. Using neuromuscular electrical stimulation for pseudo-haptic feedback. In Proc VRST ’06. ACM, pp. 316-319. DOI=10.1145/1180495.1180558

Kunze, K., Iwamura, M., Kise, K., Uchida, S. & Omachi, S. 2013. Activity Recognition for the Mind: Toward a Cognitive “Quantified Self”. Computer 46(10), pp. 105-108. DOI= 10.1109/MC.2013.339

Majaranta, P. & Räihä, K.-J. Twenty years of eye typing: systems and design issues. In Proc. ETRA ‘02, ACM, pp. 15-22. DOI=10.1145/507072.507076

Mateevitsi, V., Haggadone, B., Leigh, J., Kunzer, B., & Kenyon, R. V. 2013. Sensing the environment through SpiderSense. In Proc. AH ’13, ACM, pp. 51-57. DOI=10.1145/2459236.2459246

Millan, J. d. R., Galan, F., Vanhooydonck, D., Lew, E., Philips, J., & Nuttin, M. 2009. Asynchronous non-invasive brain-actuated control of an intelligent wheelchair. In Proc. EMBC ’09, IEEE, pp. 3361-3364. DOI=10.1109/IEMBS.2009.5332828

Miniotas, D., Špakov, O., Tugoy, I., & MacKenzie, I.S. 2006. Speech-augmented eye gaze interaction with small closely spaced targets. In Proc. ETRA ’06, ACM, pp. 67-72. DOI=10.1145/1117309.1117345

Nagel, S. K., Carl, C., Kringe, T., Märtin, R., & König, P. 2005. Beyond sensory substitution—learning the sixth sense. J Neural Eng 2(4), R13. DOI=10.1088/1741-2560/2/4/R02

Narumi, T., Nishizaka, S., Kajinami, T., Tanikawa, T., & Hirose, M. 2011. Augmented reality flavors: gustatory display based on edible marker and cross-modal interaction. In Proc. CHI ’11, ACM, pp. 93-102. DOI=10.1145/1978942.1978957.

Perkusich, M. B., Rached, T. S., & Perkusich, A. 2011. Thinkcontacts: Use your mind to dial your phone. In Proc. ICCE ’11, IEEE, pp. 105-106. DOI=10.1109/ICCE.2011.5722486

Pinhanez, C. 2001. The everywhere displays projector: A device to create ubiquitous graphical interfaces. In Proc. Ubicomp ’01, Springer, pp. 315-331. DOI=10.1007/3-540-45427-6_27

Rebsamen, B., Teo, C. L., Zeng, Q., Ang, V. M. H., Burdet, E., Guan, C., … & Laugier, C. 2007. Controlling a wheelchair indoors using thought. IEEE Intelligent Systems 22(2), pp. 18-24. DOI=10.1109/MIS.2007.26

Sahami Shirazi, A., Hassib, M., Henze, N., Schmidt, A., & Kunze, K. 2014. What’s on your mind?: mental task awareness using single electrode brain computer interfaces. In Proc. AH ’14, ACM, pp. 45:1-45:4. DOI=10.1145/2582051.2582096

Samadi, H., & Daliri, M. R. 2014. Solve the Rubik’s cube with robot based on non-invasive brain computer interface. In Proc. ICIS ’14, IEEE, pp. 1-5. DOI=10.1109/IranianCIS.2014.6802558

Saponas, T .S., Tan, D. S., Morris, D., & Balakrishnan, R. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proc. CHI ’08, ACM, pp. 515-524. DOI=10.1145/1357054.1357138

Schmidt, A. 2000. Implicit human computer interaction through context. In Springer Personal Technologies, pp 191-199, DOI=10.1007/BF01324126.

Schmidt, A. 2013. Context-aware computing: context-awareness, context-aware user interfaces, and implicit interaction. The Encyclopedia of Human-Computer Interaction, 2nd Ed., The Interaction Design Foundation. https://www.interaction-design.org/encyclopedia/context-aware_computing.html

Schönauer, C., Vonach, E., Gerstweiler, G., & Kaufmann, H. 2013. 3D building reconstruction and thermal mapping in fire brigade operations. In Proc. AH ’13, ACM, pp. 202-205. DOI=10.1145/2459236.2459271

Serby, H., Yom-Tov, E., & Inbar, G. F. 2005. An improved P300-based brain-computer interface. IEEE Trans. Neural Syst. Rehab. Eng. 13(1), pp. 89-98. DOI=10.1109/TNSRE.2004.841878

Špakov, O. & Majaranta, P. 2012. Enhanced gaze interaction using simple head gestures. In Proc. UbiComp ’12, ACM, pp. 705-710. DOI=10.1145/2370216.2370369

Starner, T., Mann, S., Rhodes, B., Levine, J., Healey, J., Kirsch, D., … & Pentland, A. (1997). Augmented reality through wearable computing. Presence: Teleoperators and Virtual Environments 6(4), pp. 386-398.

Stellmach, S. & Dachselt, R. 2012. Look & touch: gaze-supported target acquisition. In Proc. CHI ’12, ACM, pp. 2981-2990. DOI=10.1145/2207676.2208709

Tamaki E., Miyaki, T., & Rekimoto, J. 2011. PossessedHand: techniques for controlling human hands using electrical muscles stimuli. In Proc. SIGCHI’11 ACM, pp. 543-552. DOI=10.1145/1978942.1979018

Tanaka, K., Matsunaga, K., & Wang, H. O. 2005. Electroencephalogram-based control of an electric wheelchair. IEEE Trans. Robotics 21(4), pp. 762-766. DOI=10.1109/TRO.2004.842350

Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M.G., & Raskar, R. 2012. Recovering three-dimensional shape around a corner using ultra-fast time-of-flight imaging. Nature Communications 3:745. DOI=10.1038/ncomms1747

Ward, J., & Meijer, P. 2010. Visual experiences in the blind induced by an auditory sensory substitution device. Consciousness and cognition 19(1), pp. 492-500. DOI= 0.1016/j.concog.2009.10.006

Ware C. & Mikaelian, H. H. 1987. An evaluation of an eye tracker as a device for computer input2. In Proc. CHI ’87, ACM, pp. 183-188. DOI=10.1145/29933.275627

Wijsman, J., Grundlehner, B., Penders, J., & Hermens, H. 2013. Trapezius muscle EMG as predictor of mental stress. ACM Trans. Embed. Comput. Syst. 12(4), 99:1-99:20. DOI=10.1145/2485984.2485987