Guest Lecture on “Augmented Reality as an Entry Point to Parallel
Consciousness” – Dr. Jason Orlosky from Osaka University
by Yannick Weiß

Why live in reality anymore? This was one of the existential questions Jason Orlosky asked us during his presentation, making the point that we all already live in an almost completely man-made world, just physically augmented instead of only visually. Jason discussed a lot of different projects and applications concerning Augmented Reality and especially how it could be an entry point for Parallel Consciousness.

Parallel Consciousness

Parallel Consciousness describes a utopian interaction between a human and a digital representation of cognition, basically a human consciousness inside a machine that can truly recognize your current thought processes, emotions and
needs and gives the user the right information at the right time. An Artificial Intelligence (AI) inside a User Interface (UI) that can understand humans the way we do or even better. We already see it in movies and TV shows all the time, for
example JARVIS – Tony Stark’s Artificial Intelligence in Iron Man, which could almost be considered a person.

Difficulties and Opportunities

But real AIs have a lot of difficulties thinking like humans and vice versa. Jason talked about some older attempts at smart UIs and AIs and how they failed without the means to sufficiently perceive the users cognitive state. “Clippy” from
Microsoft Office was one of the examples that mostly annoyed people instead of helping them, due to a lack of perception and understanding leading to contextually useless or unnecessary information.

But AI has gone a long way since then, now built into Smartphones and lately integrated in Virtual and Augmented Reality displays and Head-Mounted-Displays, which allow for great new ways to interact – actively or subconsciously – with AI. Augmented Reality has great potential for cognitive state recognition: AR already has knowledge of the user’s viewport through environment cameras used for tracking and mapping. It also enables easy access to the eyes of the user by eye trackers and can potentially even implement Brain-Computer-Interfaces effectively.

Eyes – “The mirror of the soul”

Eye tracking can give a lot of information about the user’s current thought process. It can detect reading, memorizing and searching, based of eye movements. Visual information can also be hidden or shown based on the pupil’s vergence and focus. A lot of Emotions can also be recognized solely by the pupil’s orientation and dilation. Jason presented some projects already utilizing a combination of AR, Eye Tracking and AIs to manage text positions, automate instruction manuals, show a user where he stopped reading, or overall predict his cognitive state or current activity.

Jason presented an interesting project of his for automated diagnosis of Parkinson’s disease through VR. He argues, that those kinds of illnesses and disorders are still very difficult to diagnose manually. One commonly used method to diagnose movement disorders is based on observing the eyes, which makes an eye tracker very valuable for automated diagnosis, especially in VR HMDs. Jason created an experimental interface where the participants had to perform different tasks in Virtual Reality, while their pupil dilation and the smoothness of their eye movements was tracked and analysed. The results showed significant differences in people diagnosed with Parkinson’s disease as opposed to the control group, showing a great potential for future use of eye tracking and VR/AR for diagnosis support.

How far can we go?

The access to the user’s viewport and computer vision also enables perception of context and environment, which can be used for memory modification e.g. for dementia patients. Even mind reading is not in the realm of the impossible, judging by an example showing an AI being able to roughly reconstruct seen clips by measured brain activity. But one important question always remains: Which aspects of cognition can we offload to Artificial Intelligence and which should stay? A lot of mundane tasks are already farmed out to computers and machines, e.g. calculators. Artificial Intelligence would allow us to remove even more of those tasks from our cognitive load, letting us focus on the important ones. Tasks with no or just short term memory requirements can also be offloaded without impairing human learning. Logical and complex long term memory tasks, as well as symbiotic tasks or joined learning, should in turn remain part of the human’s mental workload. Artificial Intelligence should always support cognitive tasks without impairing the user’s learning. In short: Make them smart but don’t make us dumb in the meantime.

Reinforcement Learning

But how can we train our virtual counterpart to better understand how we humans think and learn? Jason argues, that Augmented and Virtual Reality allows us to train both human brains and the interface simultaneously, for example by reinforcement learning. Machine Learning already uses reinforcement learning for training routines and humans generally learn by reinforcements: actions in the environment lead to a state change, which can reward or punish the user. In Mixed Reality – Augmented and Virtual Reality and the things in between – applications actions, environments and rewards can be generated virtually, making it a great choice for human and AI learning. This means that the user interface can train in parallel to the user to create better and more relevant augmentations for him.

But since the involvement and presence in Virtual and Augmented Reality can be a lot higher, than in regular displays, addiction could become a serious problem. Fast virtual rewards can become more important than the much harder achievements in reality. We already see this problem in gaming and gambling, and even a lot of relationships have become virtual, due to the advancement of Mixed Reality and Artificial Intelligence. Even the most simplistic AI with a low grade of immersion – a Tamagotchi – was perceived as a very real consciousness.

What does the future hold?

But what about the future, will there be an AI equal to our human consciousness? A User Interface that doesn’t require any manual interaction, magically seems to understand the user and allows us to offload a huge junk of our cognitive tasks? There are still a lot of difficulties to be overcome, but Jason argues that Augmented and Virtual Reality, especially in combination with additional tools like eye trackers, can definitely be used as a stepping stone towards such a smart User Interface.

MMI2 – WS18/19
1. Guest Lecture (24.10.18)
by Dr. Jason Orlosky
Written by:
Yannick Weiß
HCI Master – 2. Semester

 

 

Augmented Reality als Einstiegspunkt in das parallele Bewusstsein – Dr. Jason Orlosky
by Richard Schäffer

Wir kennen sie doch alle: Virtuelle Avatare, die uns eigentlich die Bedienung eines Programms erleichtern oder uns gar in unseren Aufgaben und unserem Alltag unterstützen sollen. Stattdessen empfinden wir sie teilweise sogar als störend oder als nicht sonderlich hilfreich. Doch worauf kommt es eigentlich an und wie können wir diese virtuellen Avatare heutzutage umsetzen? Was hat es überhaupt mit Virtual Reality und Augmented Reality zu tun und wie sehr fließen diese Technologien in unser reales Leben ein?

In dem Vortrag „Augmented Reality as an Entry Point to Parallel Consciousness“ von Jason Orlosky im Rahmen der Veranstaltung „Mensch-Maschine-Interaktion 2“ wurden mitunter diese Fragen angeschnitten.

 

Künstliche Intelligenz und Augmented Reality

Virtuelle Avatare, aber auch Roboter oder Programme, die mit uns kommunizieren sollen, verfügen über eine künstliche Intelligenz, die es ihnen ermöglichen soll, uns zu verstehen und mit uns zu interagieren. Dabei sollte die Kommunikation möglichst natürlich und authentisch ablaufen. Doch künstliche Intelligenz hat immer noch Probleme damit, den Menschen fehlerfrei zu verstehen. Dasselbe gilt für Anwendung mit Augmented Reality. Es wird jedoch bereits heutzutage schon davon gesprochen, dass virtuelle Avatare eines Tages mit Hilfe von Augmented Reality noch präsenter in unserem Leben werden. Etienne Koo spricht in ihrem Vortrag „How 3D Avatars will change your life“ darüber, dass solche Avatare nicht nur als Unterstützer und Begleiter in jeglicher Situation fungieren werden, sondern, dass sogar romantische Beziehungen mit ihnen geführt werden könnten, sobald sie über eine ausreichende künstliche Intelligenz verfügen. Überdies hinaus erzählt sie von einer Vision, in der Menschen nach ihrem Tod mit Hilfe eines virtuellen Avatars nach ihrem eigenen Abbild weiter an der Seite ihrer Familie und Freunde existieren könnten. Diese Konzepte klingen heute vielleicht noch unmöglich, sind laut Etienne Koo jedoch in einigen Jahren umsetzbar.

 

 

Es sind aber nicht nur virtuelle Avatare, die von Augmented Reality profitieren. Stichwort: Parallel Consciousness. Wir benutzen Computer bereits als Speicher für Informationen, Termine, Daten und alles, was wir uns nicht merken können oder wollen. Würden wir nun zusätzlich die Hilfe einer künstlichen Intelligenz hinzunehmen, die uns kognitive Aufgaben abnimmt, machen wir uns unser Leben sogar noch einfacher. Ob das gut oder schlecht ist, ist eine ganz andere Frage.

Geräte, die Augmented Reality erzeugen, greifen auf unsere Augen zu und können so nicht nur zusätzliche Informationen für uns einblenden, sondern direkt von unseren Augen ablesen, wie es uns gerade geht, wie wir uns fühlen oder worauf wir uns konzentrieren. Wir müssten nicht einmal mehr eigene Anweisungen geben, da die künstliche Intelligenz versteht, welche Informationen wir benötigen. Wenn wir uns verirrt haben, zeigt es uns den richtigen Weg. Wenn wir etwas vergessen haben, erinnert es uns daran und wenn wir etwas Wichtiges nicht bemerken, blendet es einen holographischen Hinweis für uns ein. Selbstverständlich benötigen wir für solche Programme gute Interfaces und eine starke künstliche Intelligenz. Außerdem muss nicht nur das Programm den Menschen verstehen, sondern auch der Mensch das Programm. Wir müssen der künstlichen Intelligenz vertrauen.

 

Vorteile von AR für den Menschen

Auch wenn man vielleicht nicht direkt daran denkt, birgt Technologie mit Augmented Reality sogar Vorteile für die Behandlung oder den Umgang mit so mancher Krankheiten. Eine demenzkranke Person könnte beispielsweise eine Brille tragen, die mit einer eingebauten Kamera in regelmäßigen Abständen Bilder vom Alltag der Person macht, welche sich die Person anschließend ansehen kann. Die Brille könnte solche Bilder mittels Augmented Reality zukünftig auch direkt einblenden, um den Träger daran zu erinnern wo er herkommt oder welche Gesichter er kennt. Eye-Tracker, welche die Bewegungen der Augen analysieren, könnten beispielsweise erkennen, ob der Träger an Parkinson leidet, und so eine frühe Diagnose ermöglichen. Menschen, die Farben nicht erkennen können, könnten davon profitieren, dass eine Kamera an der Brille Farben erkennt, und dem Träger mit Hilfe von Projektionen anzeigt, um welche Farbe es sich handelt, wodurch Warnsignale oder ähnliches einfacher erkannt oder von anderen Zeichen unterschieden werden könnten.

Nicht nur in solchen besonderen Fällen profitieren Menschen von Technologien mit AR, sondern auch in gewöhnlichen, alltäglichen Situationen wie in der Arbeit. Neue Mitarbeiter könnten mit AR-Brillen eingeschult werden, indem ihnen holographisch angezeigt wird, welche Arbeitsschritte sie als nächstes ausführen sollen oder worauf sie achten müssen. Selbst das Lesen von Texten würde sich einfacher gestalten, weil AR-Brillen Seiten scannen könnten und sich merken könnten, wo wir stehen geblieben sind oder an welchen Wörtern wir hängen geblieben sind und uns gegebenenfalls Übersetzungen oder Erklärungen dieser Worte direkt einblenden könnten. Mit einer sogenannten X-Ray Vision wäre es uns sogar möglich Dinge zu sehen, die für uns gar nicht sichtbar wären (wie zum Beispiel Farben im ultravioletten Bereich), solange die Geräte diese Informationen erkennen, umwandeln und uns in anderer Form anzeigen. Neben diesen Beispielen sind zahlreiche weitere Anwendungsmöglichkeiten von AR vorstellbar.

 

 

Wie „real“ ist unser Leben heutzutage noch?

Natürlich profitiert auch die Spieleindustrie von diesen Entwicklungen. Bei Immersive Gaming taucht der Spieler zum Teil oder sogar vollständig in die virtuelle Welt ein. Spiele wie „Pokémon Go“ nutzen Augmented Reality, um die Realität mit virtuellen Objekten zu erweitern, während hingegen Virtual Reality den Spieler in eine völlig virtuelle Welt zieht.

 

 

Hier stellt sich die Frage, wie die Zukunft aussehen wird. Ist eine Realität ohne virtuelle Objekte überhaupt noch vorstellbar? Unser Alltag wird von technischen Geräten regiert und oft spricht man von einer Down-Head-Generation, in der Menschen lieber über ihr Handy kommunizieren als direkt mit ihrem Gegenüber zu sprechen. Wie sehr verändert sich unser Alltag, wenn Augmented Reality und Virtual Reality für jeden zugänglich sind?

Jason Orlosky erwähnt in seinem Vortrag, dass man von Anfang an bedenken sollte, welche Aufgaben man auf Maschinen abwälzt und welche nicht.

Einfache Aufgaben, Kleinigkeiten, wie den Stundenplan oder Termine kann man getrost abgeben und von einer künstlichen Intelligenz organisieren lassen. Logische Aufgaben, die menschliches Lernen und komplexe Reaktionen erfordern, sollte man allerdings nicht so einfach von einer künstlichen Intelligenz erledigen lassen. Nicht nur, weil wir uns nicht sicher sein können, ob intelligente Maschinen diese komplexen Aufgaben richtig lösen würden, sondern auch, weil wir sonst selbst irgendwann nicht mehr damit zurechtkommen würden, wenn wir uns daran gewöhnen, dass Roboter für uns denken und wir unsere eigene kognitive Leistung gegen 0 sinken lassen. Auch wenn die erst so utopisch klingenden Vorstellungen und Visionen plötzlich greifbar wirken, sollten die Entwickler von AR- und VR-System nicht nur auf den eigenen Profit abzielen, sondern dabei auch die ethischen Aspekte beachten. Denn es ist vieles möglich, aber sicher nicht alles von Vorteil für die Menschheit.

MMI2 – WS18/19
1. Guest Lecture (24.10.18)
by Dr. Jason Orlosky
Written by:
Richard Schäffer

Guest Lecture: Participatory Prototyping to Inform the Development Of a Remote UX Design System in the Automotive Domain – Stavros Tasoudis
by Daniel Steidl

Stavros Tasoudis is Ph.D. Candidate in HCI and works on HMI/ HCI simulation studies. He is specialized in car simulating tests and for his guest lecture he introduced us in the topic Participatory Prototyping by presenting a user study he carried out. This blogpost is also filled with further information from the study’s related paper ( Participatory Prototyping to Inform the Development Of a Remote UX Design System in the Automotive Domain). In addition, the author added some information to expand the User Requirements Engineering theory.

Motivation and researching in automotive context

The latest technologies in the car (e.g. autonomous driving) effect a fundamental change in using a car. Secondary tasks like interacting with an entertainment system develop to a primary activity. That also changes the way to design an automotive cockpit.
This is why Mr. Tasoudis started researching in this area. User Research combines techniques like Contextual Inquiry, Naturalistic Observation, (Job) Shadowing, Convert Observation and Participant Observation. If you implement new prototypes, you can use Usability Testing to test them.

A pilot study is a shift between the driver and the researcher. The idea is to develop a remote UX Design system to support a (UX) researcher in this shift. For that Tasoudis and his team tried to identify explicit, observable, tacit and latent needs from UX Design Professionals. They recruited twelve professionals with the age from 18 to 34. The participants consisted of eight academic UX-Designers and four from the industry.

A study in the car has it Žs own challenges, which are important to note in automotive context. The researcher can distort the driver’s experience by her or his presence. Other possible challenges are Intrusiveness (because of the limited space) and the concern for privacy and safety in the car. The researcher has to make sure, that the driver would not be harmed while the interaction takes place. Driving itself has a cognitive effort for the participant, which could also affect a study. Taking notes in an automobile, the researcher should pay attention on motion sickness. In the end the research effort in automotive context is much higher than on traditional methods.

Epistemology-pragmatism

If you work with the user to create or to improve a product, you have to be careful. The relationship between researcher and the consumer is bi-directional. In most cases the user doesn’t know what is really important to get the perfect product for a specific need.

A good example, what happens if you ask your user too directly, is the Atari Lynx (see the book Designed for use, page 21). In the early 1990’s Atari wanted to become market leader for handheld consoles, which was owned by Nintendo GameBoy. The responsible researcher invited users to ask them, what they would prefer. In the end, the Atari Lynx got a big sized display and enormous performance inside.

The product failed extremely, because the consumer (a kid in most cases) wasn’t able to put it into school bag to carry it around. It was too big for a handheld device. Also the Lynx’s battery last only a few hours because of the high performance.

Henry Ford is quoted to have said “If I had asked people what they wanted, they would have said faster horses.” Whether he did really say that, it has a true core.

To avoid collecting a wrong requirement you can use job shadowing or pay attention during the dialogs with the participants. It is import to observe the driver’s reactions. Tasoudis mentioned that requirements are co-constructed in the ongoing dialog between the user and designer.

Nevertheless, ethnographic research is unusual in automotive environment and researchers only slowly catch this field. Mr. Tasoudis told us, that Nissan made a few studies to explore sociocultural issues in the driving experiment. Also, BMW is using early remote open innovations and co-creation paradigms by involving users in the design phase.

Prototyping workshops and their important need

To develop a well-designed system, the researchers did some prototyping iterations combined with agile methods like User-Centered Design and Design Thinking. The prototyping process includes construction and testing it in collaboration with the prospective participants. After the testing iteration, they collected the data and what the test persons said, did and made. With this method they identify the values, which need to be fulfilled by the designer. The result is creating design goals of all these findings.

The following sentences are from Tasoudis related paper, which described the initial workshop session very well: „The first part of the toolkit consists of cards of low-fidelity paper illustrations of components […] derived from the guidelines of previous work. These cards both serve as stimuli for discussion (i.e., what the users say and do) about the design and application of the system in context, the acceptance of the technologies used, the system‘s architecture and interactivity, and lay the foundations as a tool for prototyping (what the users make).
The paper cards represent technologies or affordances previously used to help in identifying the context of the vehicle, supporting the remote communication, and presentation. We base the selection and design of our cards on previous work on remote communication, automotive domain deficiencies and UX Design process guidelines.”

Exploit the results

The next step is to analyze the workshop. That was made by visualizing all information at once on a glass wall. Patterns were identified. Important issues were clustered. Personas and scenarios were created. After sketching user-flows the first interactive prototype app were implemented.

Interacting in context – critique session and results

In a final iteration, Stavous Tasoudis made a co-designed session with the UX Professionals. The event was also used to get some feedback and critique about the prototype.
All this impressions resulted in updating the proposed system and an automotive domain. The research team could capture the results in ten Participant Needs and seven UX Goals. Following needs were developed:

• Privacy (Avoidance of intrusiveness)
• Efficiency (Quick interaction)
• Effectiveness (Map participant’s feelings)
• Engagement (More in-depth insights)
• Naturalness (Natural to use)
• Ease of Use (Feel comfortable)
• Information retrieval (Assist in avoiding information loss)
• Self-Image awareness (Support different levels of self-presentation)
• Politeness (Interaction only at the right time)
• Flexibility (Adapt needs based on the situation of use)

And finally following UX-Goals were shaped out of the study:

• Support the contextual understanding (Empathy and Effectiveness)
• Reduce the intrusion of people’s lives (Privacy)
• Support long-term understanding of behavioral patterns (Effectiveness)
• Operate even when the problem is ill-defined (Flexibility and Effectiveness)
• Avoid redundant information at any interaction (Efficiency, Ease of Use, and Flexibility)
• Avoid anxiety about uncertainty (Privacy, Engagement, Naturalness, and Self-image awareness)
• Feeling of intelligent interactivity (Efficiency, Ease of Use, and Information retrieval)

MMI2 – WS18/19
Guest Lecture (07.11.18)
by Stavros Tasoudis
Written by:
Daniel Steidl

Guest Lecture on “Avatars and Social Interactions – Engineering Social Realism” – Dr. Stephan Streuber
by Yannick Weiß

Stephan Streuber is an Assistant Professor at the University of Konstanz and is very experienced in the field of neuroscience, where he made his PhD, as well as computer science. He was invited to talk to us about his research into social perception and social avatars, especially in conjunction with Virtual Reality.

Why use Virtual Reality in researching psychology?

Researching psychology and social interactions is accompanied by lots of obstacles, because typical lab experiments are not able to reproduce actual real-life situations to a satisfying degree. Virtual Reality allows us to close or at least narrow this gap between the lab environment and reality. This is due to the high degree of immersion a Virtual Reality system can provide, resulting in an increased illusion of presence. This further allows the simulation of situations that otherwise could not be experimented in, e.g. situations that pose a danger to the participant. Experimental Control is another important issue in experiments where VR can be a great benefit, since it allows you to control most of the environment. Conclusively, VR offers a lot of advantages to regular experiments, while being relatively cheap, portable and scalable.

Triggering Emotions

Virtual Reality fairs well when it comes to triggering different emotions like aggression, anxiety or stress, due to the high perceived presence inside the virtual world. Stephan told us how he achieved these emotions in his work: To make participants anxious, they had to step into a cage. They then got lifted high above a city until a sudden explosion made them fall. Stress could be created similarly, by putting pressure on the participant. Stephan showed us an experiment where the participants had to solve math questions under time pressure on a floor, which tiles exploded and fell down, when a wrong answer was given. Or to generate aggression, participants were pitted against another player (actually an AI) in a game of hot wire, where both players were able to give each other electroshocks to disrupt the other player.

Studying Social Interactions

The coordination of social interactions with other people is something that comes to us naturally. We are very good at it, without really knowing why, which makes it very difficult to study. Even the most mundane looking situations – e.g. getting handed a coffee from a barista – requires analysis and deduction of the interaction partner’s mental state, the action itself, the objects being interacted with and the social context of the situation. A lot of times an action fails, it is due to one interaction partner misinterpreting the intentions of the other.

One possible way to study natural social interaction is to occlude certain visual information, e.g. black out faces, in the tested situations. Virtual Reality allows us to do that rather easily, while still being able to keep the rest of the perceivable environment steady. Artificial Intelligence further allows us to erase the need for a second person to interact with, negating the influence the two people would have on each other. Stephan showed us an experiment about visual cues, where participants were confronted with another person – controlled by AI, visualized and animated as a person – in a table tennis match. He observed, whether and what visual cues where used to predict the next shot. It showed that prediction through visual cues is used in table tennis a lot more often than assumed and that the context can play a major role, for example when playing in competition or cooperation. How much the prediction is dependent on the human counterpart was examined in another study in Virtual Reality conducted by Stephan, where participants had to give a High-Five to virtual avatars – a human and a robot-arm. Both times the exact same animation played, but there were significant differences in the trajectories of the user’s hands, supporting the claim that context matters for those kinds of actions.

Self Avatar

So far, we’ve only talked about exchanging the virtual counterpart, but we can also replace the visual representation of one’s own body in Virtual Reality. Stephan showed us how this could be used to study body perception for example of people suffering from anorexia nervosa – an eating disorder characterized by the desire to be thin and often leading to food restriction. The study used a 3D-Scanner to accurately remodel the user’s body, then changed specific properties to make them look fatter or thinner. The resulting virtual bodies – approximately ranging from -20% to 20% to the actual BMI of the user – were presented to the patients and a control group, with the task of finding their actual body. Surprisingly, both groups did very well, although when asked about the desired body shape, the anorexic patients chose a significant thinner body type than the control group.

Constructing Body Shapes

But how thick or thin a body is, is only a small fracture of how we describe bodies. Using special types of body models – skinned multi-person linear models (SMPL); see this video and paper – it is possible to generate a virtually unlimited amount of different body shapes. Stephan and his team used these models together with a machine learning algorithm to understand exactly what vocal descriptions (e.g. pear shaped, short, attractive, feminine) correspond to what body parameters. From this they were able to reconstruct bodies with a reconstruction error similar to 3D-Scanners, which allowed them to ultimately generate bodies based on descriptions or even just a single word. Since they can translate the descriptions of bodies into actual models, it is also possible to reconstruct characters of books based on their description or to just browse through a huge database of body models by applying keyword filters.

But people not just only relate specific vocal descriptions to body shapes, we go way further: A study, that Stephan conducted, showed that many people associate body types with personality traits. These social biases were found on group level as well as subgroup and individual level. This allowed him to reconstruct bodies from described personality traits (e.g. trustworthy, independent, powerful, democrat or republican), similar to the previous construction through body descriptions. This is a great way to study social biases, for example different associations for male and female bodies, e.g. the difference in body types for an aggressive woman and aggressive man.

Try it out for yourself

If you want to build a body yourself, you can try out and play around with the following tool, provided by the Max Planck Institute for Intelligent Systems. It allows you to generate body models according to different keywords and scales: BodyTalk .

MMI2 – WS18/19
3. Guest Lecture (24.10.18)
by Dr. Stephan Streuber
Written by:
Yannick Weiß
HCI Master – 2. Semester

Avatars and Social Interactions:
Using Virtual Reality as a Tool to Study Social Perception and Interaction – Dr. Stefan Streuber
by Christina Scheckenhofer

In the last decades the way of studying psychological behavior looked the same: Researcher tried to observe natural interaction in a
typical laboratory experiment. But is this the right way to explore natural interaction and behavior? Even if such experiments are
controlled and have a high degree of internal validity, in most cases, a laboratory environment does not reflect the real world and
the results may not match with the real-life situation. Therefore, the external validity of laboratory experiments is very low in
contrast to field studies. Is there a way to improve the validity?

With the technological advance of the last years new kinds of experiments appeared. One might be Virtual Reality (VR), which can be
used as a research tool for psychology. In VR, researcher can control the experiment the same like they would do in a labor
experiment. The difference is, that people feel present in the real-life world, it seems like a realistic situation and looks like
a natural interaction. The environment is everywhere and the participants are immersed in the virtual world. For example, people
may walk on a plank in real life, which is just on the floor. But with VR glasses they seem to be on top of skyscrapers and walk
along this plank. The situation feels very realistic for them.

This experience and other VR experiments can also be made at the exhibition “Thrill of Deception”, which is currently in
Munich.
Therefore, experiments in VR may have big advantages. It is cheaper than a “normal” laboratory experiment and portable.
Due to the fact that each participant experiences the same conditions and gets the same feedback from the panel,
external disruptive factors are almost eliminated and the experiment is scalable.

Coordination of Social Interaction

But how can social interactions in a VR experiment be coordinated? Humans know what to do in a social interaction intuitively,
which is facilitated by the fact that they do not only react, but also participate in a certain situation. For example, to buy
a coffee in a coffee shop, the buyer uses several information sources to know what to do next. As the buyer can not look into
the head from the seller, he derives information for instance from the mental states, the body language of the seller and the
social context. This information is integrated to know how to interact social. But how can researcher study such natural social
interactions? And which exact information are needed? Furthermore, how can mutual influence be prevented? One solution might be,
to perform a controlled experiment with VR. This was for example tested with table tennis. While the participant played table
tennis with an VR avatar, information about the opponent was changed. As a result, it is important to see the opponent. This
visual information about the avatar was used to predict the direction of the shot ahead of its realization. Therefore, in a
competition the information about the body is used the most.

Visual Appearance of Avatars

As already mentioned, in VR avatars are used to show other people in the virtual room so that the participants of the experiment
can interact with them. This visual appearance of the avatar can be changed with VR. Researchers explored that it does not matter
if the participant interacts with a humanoid or a robot avatar. Not only interaction partners can be shown via an avatar, also
self-avatars are possible in VR. This means that the person really sees his or her own body in the virtual room. This avatar and
thus, the participants body can also be modified, which can be used for different kind of experiments. One case is the body
perception in anorexia nervosa. Patients with this disease misperceive their bodies. The researcher provided in an experiment
different versions of avatars, which might be thicker or thinner than their actual body, and presented them to the patients.
Afterwards the patients had to estimate which one is their actual body and which is their desired body. Surprisingly, both groups
of participants, the patients and the control group, estimated a smaller body shape than they really had. This new technological
progress can help to understand the patient’s disease better and may offer new ways to heal such diseases.

Statistical Body Models for Psychology

But these 3D human avatars are also useful for different kind of studies, like for games, shopping or also for health applications.
So, the question arises how such avatars can be developed? As these body models are costly to design, the guest speaker Stephan
Streuber created crowdsourcing to generate attribute ratings of 3D body shapes corresponding to standard linguistic descriptions of
shapes. With this new process named “crowdshaping”, 3D body shapes can be automatically reconstructed by how people or avatars are
described. Furthermore, personality traits can be derived from the body shape. Therefore, social biases associated with body shape
can be uncovered. At the Bodytalk Website from the Max Planck Institute for Intelligent Systems you can test which characteristic
can be concluded from which body shape.

Provoking and measuring emotions like anxiety, stress and aggression in VR

Another topic for VR use cases is the psychological field of research and thus, feelings like anxiety, stress and aggression are
also explored. Researcher tried to provoke these emotions with VR and measure it. As already mentioned, the real world is
uncontrollable and situations that you want to practice may not occur when you need them. In VR everything can be planned and
researcher know exactly what can happen. Another advantage is that there may be a trainer next to the participant, who gives
feedback. Therefore, situations where people get anxious, stressed or aggressive can be trained. For example, how do people react
when someone cuts the line in the super market? With an interactive social scenario people have the opportunity to learn how to
deal with such a situation. Also, people with psychiatric problems can learn with VR how to reduce their stress level and how to
relax.

Conclusion

To sum it up, VR has lots of advantages and use cases for research about the social perception and interaction. With VR, situations
can be controlled and the participants in the experiments are not influenced by the real world. With the mentioned 3D body models,
which can be used in a virtual room, there arise new use cases. For instance, people can have an own avatar and buy online fitting
clothes. There may also appear communication platforms which you can really visit with your perfect body model of yourself and talk
to other people’s avatars.

MMI2 – WS18/19
Guest Lecture (14.11.18)
by Stephan Streuber
Written by:
Christina Scheckenhofer
MMT Master – 3. Semester

 

 

Guest Lecture: Avatars and Social Interactions/Engineering social realism – Dr. Stefan Streuber
by Daniel Steidl

Stefan Streuber is Assistant Professor in the Department of Computer and Information Science at the University of Konstanz. His main interest is the collective behavior of human and animals. He held a lecture about his work about social perception and social avatars in Virtual Reality with the Max Planck Institute Tübingen and presented some projects of this collaboration.

Virtual Reality as a research tool?

Laboratory experiments can feel very artificial for a test person. They stand in contrast to real life situations. But a natural interaction would be important to get as realistic results as possible. Because of that Stefan Streuber uses Virtual Reality.

Using Virtual Reality leads to a few advantages. It is possible to imitate a real situation; the participant has the feeling of being there. This phenomenon is called presence and results in the test person doing real body reactions.
Another positive effect is the flexibility that arises with Virtual Reality in experiments. The researcher is able to replace and insert different objects in the same situation. That brings a better control of the research environment. Furthermore, it is possible to examine dangerous situations without harming a participant.
In common, working with VR is cheap, portable and scalable.

Project coordination

The first project, Mr. Streuber showed us, was about human coordination. In a common interaction it’s not only important what a person says. The most human interaction occurs from facial expression, body talk and similar behaviors.
His hypothesis: A person not only reacts on another human, but also on natural social interaction. There should be a participant’s reaction on social context, mental states or objects with which the test person interacts.

But how get mental thoughts? And is there a present natural influence? For that he and his team used VR to get control over visual information. They simulated a table tennis match and let randomly disappear important objects for this interaction: Either the partner, the table tennis racket or the ball wasn’t shown. The researchers looked on the participant’s reactions. Is the reaction a natural one? And which of the objects affects more performance in table tennis than the others?

The results were impressive. As well the own body as the paddle are not important for a realistic and natural reaction. In opposite of that the partner’s body is important for realistic reaction and an affective performance in playing table tennis.
These results are different compared with previous studies without VR.

Project social context

In the next project the research team compared between VR avatars and robotic avatars. The exercise was to make high five with them. Again the focus was on looking at the natural reaction. And again the context matters as the movement reactions vary whether it was the human or the robot.

Project Self-Avatar

In another experiment the research team used Virtual Reality to visualize an avatar of the participant in a virtual world. In this case, the test persons had a distorted body perception, called anorexia nervosa. They were shown different body shapes of which they had to find out their own body size. A second group without distorted body perception was used for control. They were also asked about what body shape they would desire.
The results show surprisingly that the participants mostly did a right guess about their own body shape. Nevertheless, the anorexic patients desire a smaller body than the control group.

Project Anxiety, stress & aggression

In this project the participants were exposed to three different situations in Virtual Reality to trigger different emotions. Firstly they get the feeling of falling down trigger fear. The test persons were lead into a cage. This cage moved into a significantly height, stopped and immediately fell down.

Then they were virtually placed into a room with squared plates. They had to calculate mathematic tasks in a short duration. Every time they calculated wrong one plate fell into the deep. This should trigger stress feelings.
In a third experiment the participants had a ring and their task was to pull that ring trough a rail. If they touched the rail, they failed and had to start again. After every round the ring was shrinking to increase the exercise’s difficulty.

Body Talk Project

Traits from Body Shape

In a last project participants were shown human faces with different emotions. They also were prompted to choose bodies, that place to specific words from the US-Electives. It turned out, that more corpulent bodies were connected with republican politicians and smaller and thinner bodies with democratic politicians. Furthermore, personally traits of a person could affect the interpretation of a whole group. This also includes social traits like kindness, power and attraction. Even the gender difference could influence the body shape interpretation.
In summary, the body talk project is very interesting and can be tested by yourself on the project’s homepage.

MMI2 – WS18/19
Guest Lecture (14.11.18)
by Stephan Streuber
Written by:
Daniel Steidl

 

 

Guest Lecture: Peripheral Interaction Design – Dr. Saskia Bakker
by Yannick Weiß

Guest lecture held by Saskia Bakker, Assistant Professor at the Industrial Design Department of Eindhoven University of Technology (TU/e).

Much the same as “Love” in the hit single “Love is all around” by Wet Wet Wet, digital information is all around you and everywhere you go. We already interact with it ubiquitously and extensively. But despite the rise of fast mobile internet and IoT (Internet of Things), one thing seems to remain static: If we want to interact with this digital information, we still must switch our entire focus to it. Saskia Bakker spoke to us about this problem and how we can design peripheral interactions to accommodate it.

Information in the real world

In our daily environment we already perceive a lot of information simultaneously without the need of our undivided attention. In a simple conversation, we perceive and conclude data not only about the other person’s state of mind and intentions, but also about everything that is going on in the environment: the temperature, weather and wind, the birds in the trees and the cars honking on the next street. So, we are capable of perceiving and interacting with our periphery, but when it comes to digital information, we still focus all our attention on a single display, i.e. a smartphone or desktop screen, to interact with it.

What is peripheral interaction?

Interactions can be placed in one of three categories: Focused Interactions – which require the full attention of the user Implicit Interactions – which are completely outside the attention field and don’t require any explicit input by the user Peripheral Interaction – which describes the spectrum in between the fully focused and completely implicit interactions. While all three types can be observed in our environment, the interactions with digital information are usually done through either focused interaction – i.e. sending or receiving messages on your smartphone – or completely implicitly through smart gadgets or artificial intelligence – i.e. smart thermostats. To show us how displaying digital information in the periphery can be used effectively to lower cognitive load, Saskia introduced us to four different projects/products she worked on.

Fireflies – Peripheral Interaction in the classroom

Fireflies is a project that tries to lower the cognitive burden of teachers during class. Displaying information about the students is usually not done at all or through digital dashboards with graphs, which require the teacher’s attention. By equipping each desk with a small lamp, it is possible to convey virtual data in the physical world through light. This enables a lot of possible use cases in combination with different information tracking. One application would be to track the geographic location of the teacher throughout a course, and indicate by light, which desks the teacher already visited to help or observe a pupil. This makes it easier for a teacher to give each student the attention he or she requires, without giving up some of the focus on the lecture.

Physikit – Peripheral Interaction in the home

Physikits are small colored cubes consisting of different sensors and actuators. They can observe different environmental conditions like temperature and humidity and indicate these information to the user by lighting up, vibrating, rotating or acting like a fan – depending on the cube. The user can freely decide what the cube observes and how and when it should react. You can for example map the humidity of the air onto the rotation of the cube or make a cube light up if the room is getting too hot. That makes them a great alternative to alarms and alerts, which are meant to draw your full attention.

Beepless – Peripheral Interaction in the hospital

The same principle, offloading the information from alarms to peripheral displays to reduce stress and facilitate multitasking, is used in Beepless. This project introduces peripheral interaction into hospitals to help nurses in their daily routines. Let’s imagine a situation often encountered by nurses: A baby is hooked up to several machines to monitor its health (e.g. by measuring temperature and heartrate) or administer small doses of medicine. The nurse gets pulled away by something urgent. When he or she comes back, there are several visual and auditive alerts going off, since for example the medicine is almost running out. These alerts can induce further stress and make it harder to distinguish between more and less important or urgent tasks. The solution to this were ambient lights on each machine that slowly switch or fade into different colors, making it easier to accurately understand the machine’s status without the need for alarms.

Musico – Peripheral Interaction in the Home

The last project Saskia talked about was Musico, a context aware music player. A lot of people listen to different Spotify playlists, which they set up for specific occasions, e.g. a playlist for lazy Sunday mornings or a family dinner. But controlling or selecting the playlists still requires the user’s full attention. Musico makes controlling the music more mundane and peripheral by automatically choosing suitable tracks based on specific environmental context. Through data like the current time and location or the noise level, which lets it approximate the amount of people in the room, Musico can dynamically adjust the music to the user’s current situation. But it doesn’t take away the control of the music completely, since the user still can interact with the player e.g. pausing, skipping or volume control.

Peripheral Interaction in the Future

Saskia Bakker showed us a lot of excellent approaches to designing and integrating peripheral interactions into different situations and use cases. Conveying digital information more peripheral to let us focus on the most important aspects and interact with the rest more incidentally is an idea, that will probably become even more relevant with the rising amount of digital data we have to or want to interact with. The hardware and software requirements, as shown in the projects, are not that complex and expensive, making peripheral interactions an easy alternative to staring at different computer screens and a great tool for information visualization.

MMI2 – WS18/19
Guest Lecture (11.12.18)
by Dr. Saskia Bakker
Written by:
Yannick Weiß
HCI Master – 2. Semester

 

 

Guest Lecture: Peripheral Interaction Design – Dr. Saskia Bakker
by Nina Augustin

Saskia Bakker is an assistant professor at the Industrial Design department of the Eindhoven University of Technology. In 2013, she obtained her PhD, on her dissertation entitled “Design for Peripheral Interaction”, from the Eindhoven University of Technology.

Thinking outside the Attention Box

When we interact with our smartphone we seem to be involved in a monologue: we shift our full attention to the phone in order to receive the information we need. And then it is only us and our phone. But appearance is deceptive – reality seems to be more than that. We are surrounded by lots of information sources, we concurrently interact with day by day. Be it the sun that is shining and causes a happy mood, the sound of music in the background or people surrounding us. Peripheral information occur everywhere outside of our focus of attention. And even if they are not in focus, they are subconsciously processed and provide meaningful insights into our surroundings.

Interaction devices are usually designed to be in the focus of attention and rather exclude our peripheral environment. But why give away the potential that lies outside our attentional field? How could we exploit our attention in the periphery?

These are the questions Saskia Bakker asked herself and which motivated her to study and design human attention and interactive devices that operate outside the direct attentional field of the user.

Filling a Gap in Interaction Design

Based on the Interaction-Attention Continuum, which she established together with a colleague in 2016, she explained that there exist various levels of human attention in interaction design ranging from fully focused attention to interaction completely outside of the human attentional field. In the center of attention happens our focused interactions . They occur consciously, intentional and we have a direct precise control. In contrast to this, the implicit interactions take place outside of this attentional field. Thus, interactions happen subcounsciusly, untintentional and we cannot control them directly. According a continnum, the states between these two extremes, are described as peripheral interactions . They happen in the periphery of our attention. Just as implicit interactions they are subconscious, however they are intentional and can be controlled in an imprecise way. By focusing on peripheral interaction, Bakker intends to fill a gap that usually exists in interaction design; which however becomes more and more important in times of Ubiquitous Computing and the Internet of Things, when more and more devices surround us, becoming smaller and more invisible.

In the following she introduced four projects she worked on, which constitute examples of how to design peripheral interactions in three different environments: a classroom, at home and in a hospital.

Fireflies

Over time, classrooms become more and more enriched by different kinds of technology. However, monitoring devices for teachers often demand too much attention and cognition and therefore make the job of teachers even harder. Fireflies is a simple solution which helps teachers to give each student equal attention and assistance. The concept contains of several small devices which are placed in front of each student on the desks. A system tracks the position of the teacher and leads him/her to students, who did not get attention yet or might need help, by switching on different colored lights in the devices. With fireflies, the classroom is visually augmented with (beforehand) invisible information, which can be recognized and processed with low cognitive load.

Physikit

The information we receive about our environment help us making decisions in our everyday lives and make us more aware of our current situation. However much of this information is not directly accessible to us – such as temperature, humidity, air quality or noise level. Physikit is a visualization platform which make sensor data tangible and give us a better feel of our environment changing over time. 4 little colorful cubes make it possible to convert sensor information into either light, motion, air flow and vibrations. Through a web interface, sensors can be integrated, and the output be configurated.

Beepless

The premature baby unit of a hospital is an extremely demanding and critical work environment. Nurses must constantly monitor the baby’s status, make fast decision and handle huge responsibility. As nurses often act under stress and pressure, cognitive resources and time are limited. Thus, devices in their work environment should efficiently convey information without being distractive. Beepless is a concept, including an implementation of lights in the environment, which color-code the status of the respective patient. With this offload of information into the environment, nurses can receive information by their periphery and help their patients much faster. Moreover, this concept includes a foot paddle silencing the alarms when pressed. Without auditory distractions, stress for both – the patient and the nurse – can be reduced and the nurse can focus her full attention on the baby.

Musico

This conical device moves us back to our homes and daily live situations. Many of us know the struggle of creating and adapting the perfect playlist for different occasions and locations. One for getting up in the morning, another for a romantic dinner for two or a game night with many friends… Musico is a context-sensible device, which automates the adjustment of music played around us according not only our preferences, but also the location, the number of people in the room and the day time. Additionally, it is dynamically learning during its use when songs are skipped or replayed.

Interaction Design for the Future

Saskia Bakker allowed us an interesting insight into her design-research work and impressively demonstrated the potential of designing things outside of our attention focus. Big part of her job, so she says, contains of studying the human in natural environments. Methods like for instance “shadowing” and observing a nurse for one day or conducting qualitative interviews with teachers and students provide her instructive insights.
Corresponding the thought of ubiquitous computing, devices are put into the background (i.e. the periphery) and get more and more invisible, as we users get more into focus. Our attention and cognitive capacity are limited resources in an increasingly complex world. By conveying information tangibly through the periphery, we can reduce cognitive load and stay focused on important tasks.

MMI2 – WS18/19
Guest Lecture (11.12.18)
by Dr. Saskia Bakker
Written by:
Nina Augustin
HCI Master – 2. Semester

 

 

Devices that overlap with the human body – Pedro Lopes
by Nina Augustin

How can interactive devices can connect with users in the most immediate and personal way?

During the last decades, mobile devices became smaller and got closer and closer to our human body. In the 90s, mobile devices moved computing into our pockets and recently, wearables brought computing into constant physical contact with our skin (see illustration). Some researchers argue that the next generation of interactive devices will move past the user's skin and be directly implanted inside the user's body. However, Pedro Lopes follows another approach:

"What we need is to get closer to the user's senses and, as I argue, to the actuators as well."

Lopes aims to design and create interactive systems, which intentionally borrow the user's body for input and output, rather than adding more technology to the body – Devices that overlap with the user's muscles and intentionally triggers muscle movements.
Most of his invented systems integrate EMS (Electrical Muscle Stimulation). Some people might already have experienced this technology as "EMS-Trainings" are increasingly offered at gyms as a way to efficiently working out and losing weight. Pedro Lopes however uses this technology in a more accurate and sophisticated way in order to study the scope of EMS and to contribute to the scientific field of Human-Computer-Interaction.
Following, Lopes offers us an insight into the projects he has worked on with different research groups. As he creates devices that are – to a certain degree – taking over actions for us, important considerations arise: Who is in control? Does the overlap change the user's perceived agency?

These are questions that he addresses towards the end of his talk.

Experience: An alternative to mechanical actuators

An exoskeleton is a device, which allow for limb movement in a mechanical way, actuating on motors. Exoskeletons, which are for instance employed in military or medical settings, appear bulky and restrict the user in his movements. Further disadvantages of this technology are the high weight and the reliance on big battery packs that the user needs to carry. This is where EMS comes into play: This technology does not rely on big batteries as the limb movements result from muscle contractions, which are triggered by the electrical stimulation. Thus, the motion energy comes from our muscles themselves – namely from carbs. Instead of carrying a heavy battery set – eating your morning cereal should do the job. The EMS-skeleton is further small and fits – attached to the arm – perfectly under a t-shirt sleeve. These are some of the huge advantages that makes EMS attractive to research.
An obvious application of EMS are Virtual Reality environments as haptic sensations can augment the visual and auditory experience and eventually create more realism. With a sensation of haptics, it is possible to simulate obstacles like walls or adding the experience of gravity when carrying a virtual object. One of Lopes' past projects is called Impacto . The hybrid bracelet combines electrical muscle stimulation and small solenoids, which exert a mechanical force on the user's skin. It is used in VR settings to simulate for instance realistic punches with force feedback when the user hits his opponent or gets hit by him. An evaluation study indicated that a combination of both techniques result in a higher perceived realism level than the technologies applied separately. The arm moving back (caused by EMS) is a reaction to a virtual punch, explained by the solenoid tap. This event however is too brief for the human to rationalize and thus the impact is attributed to a moving mass against the body, which feels realistic to the user.

Information access: a new I/O modality (proprioception)

After exploring the output side of the EMS-technology, Pedro Lopes moves on with a new input modality for his interactive systems: Proprioception. In contrast to stimuli like sounds or images that we receive externally, proprioception origins internally and reflects the pose of our own body. The counterpart of the EMS is the EMG (electromyogram), which measures electric muscle activity and derives proprioception. Proprioceptive interactions are usually conceptualized in relationship between the device and the human – the following project however examined the application to the user's physical environment.
The usability term affordance was established by Donald Norman and describes the interaction possibilities an object offers to the user by its visual and haptic appearance. Affordance++ is a project, which takes this definition beyond these aspects and adds an action component to it. It is hard to design timely actions visually, but with the help of Affordance++, a cup of hot water can "tell" you to use the handle instead of touching the burning surface and a spray can does not "allow" you to forget shaking it before usage. You wonder how this should work?
Watch this video to see how Pedro Lopes and his colleagues implemented a way that objects could communicate directly and immediately with us in a whole new way:

In contrast to pictures or music, muscle memory is nothing you can recall or express easily. Try it once to explain your muscle movements while drawing a car – not easy, right? Another project that Lopes presents in his talk is the muscle-plotter, which records muscle movements and allows for outputting recorded drawings by actuating the user's wrist via electrical muscle stimulation.

Who is in control?

In this talk, Lopes presented many exciting projects. The possibilities of EMS- and EMG-technology are amazing and seem endless. However, the thought arises, where this technology finally could lead to as we allow systems to take over muscle functions and generally – agency. A sense of agency implies the experience of controlling one's own body, as well as having the ability to change the external environment. Instead of completely taking over the control of the user (as illustrated in the Art Project Ad Infinitum), the goal should be to maintain the users's (perceived) agency to a certain degree.
If you want to get more insights into Pedro Lopes' research and see more of his projects, check out his website.

MMI2 – WS18/19
Guest Lecture (12.12.18)
by Pedro Lopes, University of Chicago (assist. Prof.)
Written by:
Nina Augustin
HCI Master – 2. Semester

 

 

Guest Lecture: 10 Principles for Gaze Interaction Design – Professor Hans Gellersen
by Christina Scheckenhofer

Hans Gellersen is a professor at the Lancaster University in the United Kingdom where he focuses on Human-Computer
Interaction and Ubiquitous Design. Apparently, he works on pervasive eye tracking and novel interaction methods based
on eye movement. Therefore, Hans Gellersen gave a lecture about 10 Principles for Gaze Interaction Design.

1. Gaze signals interest

One of the problems in Human-Computer Interaction Design is, that you cannot exactly comprehend where the user focuses on.
Especially when you have lot of information coming in, users and also researchers cannot process how to focus on the most
important information. But gaze is very intuitive and may be a solution for this problem. In the midst of impressions and
events, the best way of directing attention is vision, specifically, where you are looking at. Based on this idea, the
gaze automatically tells the researcher or also the computer, where the users are looking at and what they are interested
in. So, attention is the mechanism for filtering information. For example, there is visual information which is caught by
moving the eyes around and reveals what the user is focusing on in his or her mind. Considering these aspects, the
professor talked about letting the eyes do the talking. The question arises how general or specific the interest of the
user is, which means if the user is interested in the detail he or she is looking at, the whole object or the entire
scene. So there arises an example with a drawing of the little prince. If users are asked where the little prince lives,
they look at the whole planet. On the other side, if asked about the hobbies of the little prince, users look only look
at some special points that indicate the hobbies. If you want to see where users look mostly, you can collect gaze data
on how people looked at objects and build up a picture of user’s interest. But the problem arises, that we cannot look at
everything at once. To demonstrate this attention dilemma, some researchers around Hans Gellersen designed a game, where
users had to look at some characters to put them with the gaze into a safe zone, while they can also decide to look at
enemies in order to destroy them. Therefore, the users are in a dilemma where they have to decide where to look at and how
to manage the situation and attention. To sum this first principle up, gaze naturally selects focus and context.

2. Gaze is a natural pointer

As gaze gives away a lot of information, we are able to point with our eyes. But can we also use these for interaction?
The idea is, that what you are looking at is what you get. So, instead of mapping to an object with the mouse, you map
the cursor with your gaze. Thus, an action can be selected implicitly. But there arises the problem, that we may not want
to get everything we are looking at. This is called the midas touch problem, where our eyes have evolved over millions of
years to view the entertainment and not to manipulate it. This issue can be addressed by combining gaze with other modalities
like touch. For instance, the eyes select the object and the hand does the manipulation.

3. Gaze can be different

The gaze can be meaningful when it is different. This means we can choose to move our eyes in ways that are different
from how our eyes move when we ‘just look’. For example, with a long fixation of an object, we can use this dwell
time for selection. With anti-saccades, objects can be selected by looking away. Another instance are saccadic
sequences, where gestures with our eyes are performed.

4. Gaze is fuzzy

Gaze selects a region and might not pick a single point. Hence, eye tracking involves inherent uncertainty and unambiguity.
To compensate for inaccuracy there are a wide range of techniques for resolving ambiguity and detail. For using gaze
representation that reflect uncertainty, the need for giving feedback arises. For example, with a cursor which shows a
circle around the gaze. But, also trajectories, highlights or spotlights are possible. These examples can be found
here.

5. Gaze precedes action

Another principle is, that the gaze precedes action. For instance, a basketball player does not look at the ball he is going to
throw, but he is looking at the basket. At an experiment with block stacking, the point of gaze went a little earlier to the
point where the participant wanted to put the block. If someone else stacks the blocks and therefore the participant does not
act, his or her gaze is exactly at the block and not looking ahead. If the blocks moved alone, the gaze of the participants just
followed the blocks. Thus, we can extract meaning from whether the eyes are where the action is. Another example is that gaze
shifts manual input, and a pen does the fine work on drawing a car, whereas the point is guided by the gaze. So, the gaze sets
the content.

6. Gaze follows movement

When we look at an object that moves, our eyes can couple to it and perform the same smooth movement. The gaze interaction may
be possible without having to estimate the direction of the gaze. For example, in a game, a frog captures flies while you are
looking at them. This shows that eyes are very good for following motion and picking out motion in comparison to hands which
would not be fast enough. Another advantage of eye tracking is that it cannot be faked. Eye motion cannot happen accidentally.
But first the calibration of gaze has to be made. This can also happen unaware. By asking participants to fix something, a
mapping is calculated. With smooth pursuit of an object, which the participants should follow, the eye tracking device can be
calibrated implicitly. This smooth pursuit may not only be helpful for calibrating, but also targets can be selected. For
example, picking a button by surrounding a circle. This interface may be appropriate for smartwatches.

7. Gaze guides perception

Before eye tracking was possible, people used saliency maps to show where the most information at a picture is and where the
attention is paid to. Knowing where people are likely to look at can support the research process. With the knowledge of gaze
details can be suppressed that would otherwise not be noticed. This may be especially interesting for virtual reality with
head-mounted displays. Because within this window your eyes will just move a little bit, whereas if something is more outside
of the window, there appears a mixture of head and eye movement. Therefore, eye tracking can modulate perception. For instance,
the perception of the color of a cube can be changed due to manipulation of the surrounding.

8. Gaze reflects context

Another important principle is that gaze reflects the context. First, the eye movement depends on the task. For example,
in a study a painting is shown to participants but with different tasks like figuring out the age or the
income level of the people. Participants will look at the picture differently and the eye movement is driven by what they are
looking for. Second, eye movement depends on the activity. And finally, the eye movement depends on the situation and incurring
contextual cues from eye movement.

9. Gaze is social

It is also said that gaze is social. For instance, with our gaze we give signals to other people. Shared attention is also a
characteristic of this principle like looking at the same things, for example in the lecture to slides or in a cinema. Gaze
also acts as a non-verbal communication. The professor Hans Gellersen and his team developed a
game to propose social gaze
interaction as a game mechanic to enhance user interactions with virtual characters.
So, this kind of gaze interaction is also possible for VR.

10. Gaze is ubiquitous

The last principle of professor Hans Gellersen is that gaze is ubiquitous. Detecting gaze is possible with ambient devices.
There were also some field studies ‘in the wild’. But this kind of interaction was unusual to the people and they tried to
interact in another way with the screen like with hand waving or head moving.

Conclusion

To sum it up, eye tracking can be used for a new interaction design and has some principles. Gaze signals interest, which is
useful as an implicit input. Gaze is also a natural pointer which can be used explicitly. Gaze can be different and enables
gaze-only interfaces. Gaze is fuzzy which has to be accepted. Gaze precedes action and is therefore appropriate for multimodal
interaction. Gaze follows movement, whereas motion serves as design material. Gaze guides perception which is a bottom-up attention
and also reflects context, which is a top-down attention. Gaze is social and has therefore a deeper meaning than (x,y). And finally,
gaze is ubiquitous and hence occurs to anyone, at anytime and anywhere.

MMI2 – WS18/19
Guest Lecture (19.12.2018)
by Hans Gellersen
Written by:
Christina Scheckenhofer
MMT Master – 3. Semester

 

 

Guest Lecture: Gaze interfaces w/o calibration – Professor Hans Gellersen from Lancaster University
by Niklas Müller

Modern thinking needs new interface methods. Either for research, the gaming industrie or to create new interface possibilities for
different devices. One of these modern research areas are gaze interactions and eye tracking. Hans Gellersen, professor at
Lancaster University, works mostly on Human-Computer Interaction and Ubiquitous Computing. In his guest lecture at
the LMU Munich, Hans Gellersen talked about the 10 Principles for Gaze Interaction Design.

1. Gaze signals interest

In Human-Computer Interaction there are two big problems for a research team. First of all it is very difficult to determine
where the participant or the user focuses on, just by observing and talking to him. On top of that there is a huge amount
of data and information. To sort this information and pick the most important things out can be a difficult task.
The gaze handels these problems very well because humans can’t look at everything at once. So, what the gaze does
is the following. It simply indicates what the person has in mind at this exact moment. If you are talking to a person
for example as long as you listen to him or her, your gaze will mostly be focused on the person. To track this eye movements
can be pretty simple with modern technology and delivers different findings. How general/specific is the interest? Is the user
interested in a detail, a whole object or even in the whole scene? In other words, the gaze naturally selects focus and context.
Therefore it is a perfect source to get informations about what people are focused on.

2. Gaze is a natural pointer

Because people are mostly looking at what they want to interact with, gaze can be used as a natural pointer.
What you look at is what you get.
When it comes to interaction and not just looking, the question arises if you really want to get
or manipulate everything you look at. This problem is well known as the midas touch problem. The human eye
has evolved over million of years to look at the environment and not to manipulate it. An approach to solve
this problem might the combination of different interaction methods – the gaze selects things and e.g. touching
manipulates them.

3. Gaze can be different

Gaze movements can be different and thus meaningful. This gives us more possibilities to use the gaze as an interaction possibility.
When fixating an object for a long time it is for example possible to use the dwell time for selection. Another idea is to use
anti-saccades by looking away to select an object. On the other hand saccadig sequences are also possible. Several different
gestures can be performed with the eyes and can be used as different interaction possibilities.

4. Gaze is fuzzy

One problem with gaze is that it selects a region and not only a single point. Because of this eye tracking always involves
inherent uncertainity and ambiguity. Thus a wide field of techniques are required to reflect uncertainity and ambiguity.
This can be accomplished through different gaze interactions. Possible methods could be a cursor in form of a circle around the gaze,
trajectories, highlighting or through spotlights.

5. Gaze precedes action

The 5th principle points out that gaze precedes action. Before you carry out the action, you always take a look where the action should end.
To explain this Hans Gellersen showed an example of block stacking. When the participant stacked a block on another, his gaze went
to the place where he wanted to put the block before he carried out the action. When someone else stacked a block, the gaze of the person
who were just watching, stayed just on the block. The same happened when the block moved alone. This experiment showed that you can
put a meaning into the gaze, depending on where it exactly is.

6. Gaze follows movement

The gaze can couple with moving objects to perform the same smooth movement. A small game where you play a frog and have to catch flies
by looking at them showed that eyes can perform smooth movements very easily. On top of that these movements can be performed faster
than hand movements. Also eye tracking can not be faked because gaze movement always happens intentional. To ensure this, first a
calibration have to be made, either unaware or aware. This mostly happens by asking the participants to follow a particular path with
their eyes. This technique is not only used for calibrating the users gaze but also for new interaction methods, e.g. to select
a button by following a circle around it.

7. Gaze guides perception

Saliency maps were used to determine where the information on a picture is and where most people pay attention to. Knowing or have
a intimation where people are likely to look at can support research or the highlighting of details. This could be especially important
for virtual reality systems with head mounted displays. In head mounted displays there could be little eye movements for everything
inside the display but a combination of eye and head movements for everything outside the display area.

8. Gaze reflects context

Another principle is that gaze reflects context. The eye movement can depend on the task, the activity or the situation in which the person
has to look at something.
In a small study people were asked to look at a picture showing a traditional family in the 19th / early 20th century. The gaze of the
participant were tracked throughout the whole study. At first they were asked to guess the age of each family member. After that the participant
was asked to determine how poor or rich the family was. For the first question the gaze of the people were mostly focusing on the faces of the family
members. When asked on the wealth of the family the participants focused on the whole scene, e.g. on the interior and the clothes. This shows that
the gaze of the people is based on the context.

9. Gaze is social

Another feature you need to focus on is that gaze can be/is social. Gaze can give signals in form of non verbal communication to other people or can cause a
shared attention, for example in a lecture or the cinema. Hans Gellersen et al. developed a game in which the NPCs behave different, wether you look at them or dont look at them.

10. Gaze is ubiquitous

The last and also quite obvious principle is that gaze is ubiquitous. The challenge here is that you have to make it obvious for the people how to use the device.
In a small field study for example, people had difficulties to controll the device because they didn`t thought of using gaze as a interaction method and tried
to interact with it trough hand gestures.

MMI2 – WS18/19
Guest Lecture (19.12.18)
by Hans Gellersen
Written by:
Niklas Müller

 

 

Guest Lecture: 10 Principles for Gaze Interaction Design – Professor Hans Gellersen
by Lisa Bauschke

In this guest lecture, Hans Gellersen from the University of Lancaster presented the topic gaze interaction. His studies focus on Human-Computer Interaction and Ubiquitous Computing as well as eye tracking and new interaction methods with the basis on eye movement. Furthermore, Hans Gellersen has a special interest in spontaneous interaction across devices, more precisely the interaction with mobile devices on shared surfaces as well as public displays.
During his guest lecture, Hans Gellersen presented the 10 principals of gaze interaction design, which will be explained in the following blog article one by one.

1. Gaze Signals Interest

“In the midst of impressions and events, the mode par excellence of attention is vision specifically where you are looking.” This claim summarizes very well, that the eyes are focusing on things of interest and have to be selective since there are too many attractions out there. But how can the eye identify what should be in focus? This question was examined during a study and the result was that gaze tells automatically the computer what we are interested in. Where the people exactly look at, show what they have in mind. This information can be used to dissolve interaction in natural gaze behavior. By using zoom gesture, the focus of the eyes can be found out. Gaze naturally selects focus and context. Self-centering displays can be used to combine gaze and users’ interest since where ever the user looks at, it will be transferred to the display.

2. Gaze is a natural pointer

Using gaze as a pointer is possible and provides certain information on the computer. The idea is whatever you look at is what you get. During an experiment you can select a button or an option on the menu with gaze by asking the user specifically to look at the choice. But one problem appears. People look at things they don’t automatically want. Therefore, the computer needs to know when people want something and when they are just looking at it. To find this out, the combination of different interaction methods like touch and gaze is necessary. For example: Using gaze to pick up a new color and by putting the color down with the hand, the chosen color is confirmed.

3. Gaze can be different

People can choose to move their eyes in ways that are different from the ways when they are just looking at something. We have the differentiation of a) fixating longer means using dwell time for selection, b) anti-saccades means looking away for selection and c) saccadic sequences means performing gestures with our eyes (e.g. drawing with the eyes or making unusual movements).

4. Gaze is fuzzy

Gaze selects a region and not a single point. Whenever eye tracking is used, it involves inherent uncertainty and ambiguity. But how can be represented where the user is looking at on for example a display or screen and how can we give feedback to the user? Certain options to give feedback are the usage of cursor, trajectory, highlighter or spotlight.

5. Gaze precedes action

Eye can participate action. One of action from eyes can be observation. Action can be followed by eyes even if we don’t do it on purpose by ourselves. We can extract meaning from whether the eyes are pointing where the actual action is. For example: A pen can be used as an input for the gaze. The pen is used by the gaze to draw at the exact position. An indirect pen makes the cursor appear at the user’s gaze with which the user edits the lines. In this case the pen can be a remote controller where the user doesn’t have to look at directly but still does the task.

6. Gaze follows movement

Looking at an object that moves creates a smooth movement of the eyes (e.g. movement from one point to another by following an object). Smooth pursuit is like a fixation. To detect what people are looking at from an ocular pursuit by comparing the motion pattern of the display and the eye movement.
Gaze interaction can appear without having to estimate gaze direction. This can appear when it is a) user independent (no user calibration procedure), b) target-size independent (no problem with small targets) and c) tracker independent (e.g. wearable instead of remote).
In general, one cannot find out one hundred percent whether the eye movement happens or not. The system just knows when the user is looking at something on the display or not. But unaware calibration can become a huge problem in eye tracking systems. To overcome this problem could be to match eye tracking with a moving object on the screen.

7. Gaze guides perception

People have to make the decision where to use the visual resources regarding the selection of different stimuli like colors. With the knowledge of gaze, eye tracking can provide insights where to place important information and how to draw peoples’ attention. This knowledge is important for creating a virtual environment with a new field of vision. Eye tracking can be used to manipulate the perception of e.g. color or certain objects.

8. Gaze reflects context

Gaze reflects what people are doing because eye movement depends on the task, activity and situation. The activity can be spotted based on characteristics of eve movement. Machine learning features recognize different patterns and features.

9. Gaze is social

With gaze people can draw signals to other people as a non-verbal conversation by just looking into someones eyes. But how can we use social gaze? Eye Trackers are designed for individual people. But currently, there is not much research on how to use it for interaction between multiple people.
In game design, shared attention is very important. Users are seeing several things on the screen simultaneously and have to react to them. The reaction various between users, depending where they are looking at. In game design, it is important to include natural behavior of peoples eye movement.

10.Gaze is ubiquitous

As a future outlook, using gaze control of ambient device like sound systems, TV etc. by “just looking at it” and receiving a direct response of the system.

To sum it up one can say: Gaze is anyone, anytime, anywhere!

MMI2 – WS18/19
Guest Lecture (09.01.2019)
by Professor Hans Gellersen
Written by:
Lisa Bauschke
MMT Master – 3. Semester

 

 

Guest Lecture on “Gaze interfaces w/o calibration”

– Professor Hans Gellersen from Lancaster University

by Niklas Müller

Modern thinking needs new interface methods. Either for research, the gaming industry or to create new interface possibilities for
different devices. One of these modern research areas are gaze interactions and eye tracking. Hans Gellersen, professor at
Lancaster University, works mostly on Human-Computer Interaction and Ubiquitous Computing. In his guest lecture at
the LMU Munich, Hans Gellersen talked about the 10 Principles for Gaze Interaction Design.

1. Gaze signals interest

In Human-Computer Interaction there are two big problems for a research team. First of all it is very difficult to determine
where the participant or the user focuses on, just by observing and talking to him. On top of that there is a huge amount
of data and information. To sort this information and pick the most important things out can be a difficult task.
The gaze handels these problems very well because humans can’t look at everything at once. So, what the gaze does
is the following. It simply indicates what the person has in mind at this exact moment. If you are talking to a person
for example as long as you listen to him or her, your gaze will mostly be focused on the person. To track this eye movements
can be pretty simple with modern technology and delivers different findings. How general/specific is the interest? Is the user
interested in a detail, a whole object or even in the whole scene? In other words, the gaze naturally selects focus and context.
Therefore it is a perfect source to get information about what people are focused on.

2. Gaze is a natural pointer

Because people are mostly looking at what they want to interact with, gaze can be used as a natural pointer.
What you look at is what you get.
When it comes to interaction and not just looking, the question arises if you really want to get
or manipulate everything you look at. This problem is well known as the midas touch problem. The human eye
has evolved over million of years to look at the environment and not to manipulate it. An approach to solve
this problem might the combination of different interaction methods – the gaze selects things and e.g. touching
manipulates them.

3. Gaze can be different

Gaze movements can be different and thus meaningful. This gives us more possibilities to use the gaze as an interaction possibility.
When fixating an object for a long time it is for example possible to use the dwell time for selection. Another idea is to use
anti-saccades by looking away to select an object. On the other hand saccadig sequences are also possible. Several different
gestures can be performed with the eyes and can be used as different interaction possibilities.

4. Gaze is fuzzy

One problem with gaze is that it selects a region and not only a single point. Because of this eye tracking always involves
inherent uncertainty and ambiguity. Thus a wide field of techniques are required to reflect uncertainty and ambiguity.
This can be accomplished through different gaze interactions. Possible methods could be a cursor in form of a circle around the gaze,
trajectories, highlighting or through spotlights.

5. Gaze precedes action

The 5th principle points out that gaze precedes action. Before you carry out the action, you always take a look where the action should end.
To explain this Hans Gellersen showed an example of block stacking. When the participant stacked a block on another, his gaze went
to the place where he wanted to put the block before he carried out the action. When someone else stacked a block, the gaze of the person
who were just watching, stayed just on the block. The same happened when the block moved alone. This experiment showed that you can
put a meaning into the gaze, depending on where it exactly is.

6. Gaze follows movement

The gaze can couple with moving objects to perform the same smooth movement. A small game where you play a frog and have to catch flies
by looking at them showed that eyes can perform smooth movements very easily. On top of that these movements can be performed faster
than hand movements. Also eye tracking can not be faked because gaze movement always happens intentional. To ensure this, first a
calibration have to be made, either unaware or aware. This mostly happens by asking the participants to follow a particular path with
their eyes. This technique is not only used for calibrating the users gaze but also for new interaction methods, e.g. to select
a button by following a circle around it.

7. Gaze guides perception

Saliency maps were used to determine where the information on a picture is and where most people pay attention to. Knowing or have
a intimation where people are likely to look at can support research or the highlighting of details. This could be especially important
for virtual reality systems with head mounted displays. In head mounted displays there could be little eye movements for everything
inside the display but a combination of eye and head movements for everything outside the display area.

8. Gaze reflects context

Another principle is that gaze reflects context. The eye movement can depend on the task, the activity or the situation in which the person
has to look at something.
In a small study people were asked to look at a picture showing a traditional family in the 19th / early 20th century. The gaze of the
participant were tracked throughout the whole study. At first they were asked to guess the age of each family member. After that the participant
was asked to determine how poor or rich the family was. For the first question the gaze of the people were mostly focusing on the faces of the family
members. When asked on the wealth of the family the participants focused on the whole scene, e.g. on the interior and the clothes. This shows that
the gaze of the people is based on the context.

9. Gaze is social

Another feature you need to focus on is that gaze can be/is social. Gaze can give signals in form of non verbal communication to other people or can cause a
shared attention, for example in a lecture or the cinema. Hans Gellersen et al. developed a game in which the NPCs behave different, whether you look at them or don’t look at them.

10. Gaze is ubiquitous

The last and also quite obvious principle is that gaze is ubiquitous. The challenge here is that you have to make it obvious for the people how to use the device.
In a small field study for example, people had difficulties to control the device because they didn`t thought of using gaze as a interaction method and tried
to interact with it trough hand gestures.

MMI2 – WS18/19
Guest Lecture (19.12.18)
by Hans Gellersen
Written by:
Niklas Müller

 

 

Guest Lecture: User Experience in Automated Contexts: From Factory to Automobiles – Alexander Meschtscherjakov
by Christina Scheckenhofer

Alexander Meschtscherjakov is an assistant professor at the Center for Human-Computer Interaction and
Computer Science Department of Salzburg University. There he does some research about persuasive
technology, user experience, automotive user interfaces and effects of automation on UX and interaction
design. Apparently, he is working on some projects, like the Digibus Austria and GeFaBe which is a city design process.
Therefore, Alexander Meschtscherjakov gave a guest lecture about “User Experience in
Automated Contexts – From Factory to Automobiles”.

Transitions

In the last decades some transitions took place. First of all, vehicles changed. Not only the placement and
shape of the steering wheel altered, but also the whole design of a car changed from the past to
automated cars in the future. But also, the industry is changing, for instance through the Industrial
Revolution. There are different kind of industry transitions, like the industry 1.0 where mechanization
and steam power were introduced. In industry 2.0 mass production, the assembly line and electrical
energy became popular. Afterwards, in industry 3.0 automation, computer and electronics were initiated,
whereas nowadays, in industry 4.0, the focus is for example on cyber physical systems. Therefore, also
the usage of technology, like the Human-Computer Interaction, is changing to being part of an ecosystem
where the focus has to be on Human-System Collaboration.

Usability and User Experience

Due to these transitions, the Usability and User Experience of systems and machines have to be regarded.
Usability is the ease of use and learnability of an object created by a human. User Experience means a
person’s perceptions and responses resulting from the use and/or anticipated use of a product, system
or service. With transition of the technology, the Usability and User Experience are also changing,
as you may see in the video where a woman is driving a self-driving car for the first time.



Hence, we have to rethink these factors as the interaction changes in an automated car and in general
in the dawn of ubiquitous computing, tangible interaction, physical computing and natural interaction.

Direct and Indirect Manipulation

Direct manipulation means that whatever we do with the system, the system reacts to our action. But
what does it mean in a world, where we are connected to a lot of systems? How can we manipulate
directly in such a connected system? In practice, direct manipulation of all objects may not be
feasible. For example, it may be difficult for people to learn and remember all the necessary
operations and actions, and hence, it is better to use indirect manipulation. The question also
arises, what does it mean to User Experience? Therefore, we also have to reconsider the concept
of (1) Attention, (2) Perception, (3) Memory, (4) Mental Model, (5) Physical Effort and
(6) Cost/Benefit.

New Form of Interaction Needed

Due to these changes, new forms of interactions are needed, for example the social interaction.
Currently, the human-robot interaction arises. If the system is autonomously acting, having its
own agency, the user does not know the outcome and an open-ended interaction arises. Furthermore,
room for misinterpretation comes up. Thus, in a world of robots, we have to interact in a
different way.

Where lays the “intelligence”?

An artefact is perceived as being dumb, as maybe a robot is only built for a limited context and not
capable to do many other things. The intelligence lays in the backend, and is renewed by updates.
In the future, it may happen, that you go into your car and due to updates the User Experience, but
also the interaction may be fundamentally different. Hence, new mental models arise.

Too fast? Too slow?

Furthermore, timing issues arise in the context of automation. For example, a machine may be too
fast and the assembly line worker might get stressed or might be bored if the machine works too slow.

Automation Levels

With these transitions different automation levels arise. A very highly autonomous system decides
everything, acts autonomously and ignores the human. On the other hand, the lowest level, where
rather no automation happens, the computer offers no assistance and the human must take all
decisions and actions. So, for example in the car industry, there exists
5 levels of autonomous
driving. In level 1 the driver assistance systems support the driver, but do not
take control. In level 2 there is a partly automated driving system which can take
control, but the driver remains responsible for operating the vehicle. A highly
automated driving experience happens in level 3, where in certain situations, the
driver can disengage from driving for extended periods of time. Whereas in level 4
fully automated driving is possible, where the vehicles drive independently most of the
time, but the driver must be aware to take over the control. Only in level 5, the full
automation, the driver does not have to interfere and the vehicle assumes all driving
functions. It seems that nowadays the human is the assistant to the machine. But so far,
it is not clear how to fully automate it, because the car has to react to different
problems, solutions and outputs.

Furthermore, regarding the industry, it may be monotonous and boring to interact with a robot.
But also, in a self-driving car, after a few minutes people get bored without anything to do.
How can the driver be kept in the loop? Another complexity occurs from connectedness. For example,
at an assembly line, the screwdrivers are exactly at the place where the workers need them. If the
system changes, at the area you might not need screwdrivers anymore. Thus, also humans have to
adapt to the different kind of situations.

Errors

In such automated systems lots of errors may arise. The first question should be, where the error
is. It may be in the system itself, in the backend or you have to ask if there is an error at all.
For example, an error may be initiated via a time-out of a system. But how does the system know
that there is an error and how should this time-out be mentioned? If it is too short, it may have
not been an error. But if it is too long, the error might become bigger. Worse situations evolve,
when people did not react fast enough. Therefore, this problem has to be tackled.

Loss of perceived control

If asked anyone, the loss of perceived control might be the biggest problem of autonomous driving
(and also similar technologies). For example, what happens, if there exists a problem with steering
the wheel or breaking in autonomous driving. Hence, the question arises how much context information
does the driver need to be aware of in order to be able to take over control? A sequence of the (1)
situation awareness (perception, comprehension, projection), followed by the (2) decision and (3)
action arises, which loops via the state of the environment back to the situation awareness. For
example, the perception of the same time changes based on the context. If you have fun, 10 seconds
are short, whereas 10 seconds seems to be long if you are bored.

Ethics

Due to automation, ethical issues arise. One of the most popular dilemmas is the trolley dilemma,
which is a thought experiment in ethics. If a train or trolley is moving and stays on course it
will hit five people lying on the tracks, if you change the course it will hit one person. How do you
decide? A dilemma is called a dilemma, because it does not have a solution. As long as you
have to answer this decision, such systems should not be built. Therefore, an automated system
should be designed in a way that you never get to such a dilemma.

MMI2 – WS18/19
Guest Lecture (09.01.2019)
by Alexander Meschtscherjakov
Written by:
Christina Scheckenhofer
MMT Master – 3. Semester

 

 

Guest Lecture: UX in automated Contexts – From Factory to Automobiles – Alexander Meschtscherjakov
by Susann Warneke

Transitions

“Nothing is more constant than change”. Change has always existed. However, one could argue that the current changes in the handling and use of machines and new technologies are particularly groundbreaking. Being currently in the fourth phase of the dital revolution, cyber physical systems, new technologies such as IoT and generally entities talking to each other become part of the every-day life. So, it is all about transitions; this blog post concentrates mostly on transitions in vehicles and transition in industry.

Transitions in vehicles means the shift from non-automated or just partially automated cars to a highly and fully automated driving experiences (referring to automation levels). The future vision of driving a car relates to the fact that there is a transition of tasks performed while driving. Usually, the task of driving the car, that means holding the steering wheel in the hand and looking concentrative at the street, is the primary assignment for a driver. Within the context of current changes and technological developments, the original primary task for a car driver disappears and is controlled by the car system itself, so that the driver can focus on previous secondary tasks such as reading or even sleeping. Therefore, the attention for the driving act itself is somehow dissolved.

Also in the industry sector, we currently experience a vast change regarding the usage of machines and based on that the interaction of machines and people. It is not just about getting more and more machines integrated into the production processes of huge factories, but also about an increased intensity of human-machine-interactions – it is becoming a more personal interaction. We move from using the technologies to “being part of an ecosystem”. So, one could additionaly say that it is not about human-computer-interaction anymore, but about human-system-collaboration.

Usability and User Experience

There is a huge variety of different usability and user experience (UX) definitions. Two short definitions should function as a basis for the understanding of the following paragraphs: “Usability refers to the extent to which a product, system, or service can be used by certain users in a particular application context to achieve specific goals effectively, efficiently, and satisfactorily” and “User Experience extends the term usability to include aesthetic and emotional factors such as appealing, “desirable” design, aspects of trust-building or enjoyment of use (joy of use).” So regarding the current transitions in the area of human-system-collaborations and developments such as ubiquitous computing, we need to rethink factors concerning the usability and UX of a system, products etc.

(In-)Direct Manipulation

Oftentimes, people talk about the manipulative impact machines could have on humans in terms of their personal behavior and perception while interacting with each other. But what does direct manipulation means in a world we don’t interact directly with the systems anymore? What is the interaction here; what is the interaction with systems or technologies e.g. based on sensors? What does that mean to UX? Can you use a system that you are only perceiving? There is no direct manipulation anymore, but an indirect one.

Talking specifically about the interaction of humans and machines, it is always influenced and led by user limitations such as attention, perception, memory, individual mental models, physical efforts and cost/benefit considerations. One thing is clear in this context: The status quo requires a new form of interaction which will automatically have an impact on the user limitations; it will be more of a social interaction. Interacting with autonomous systems such as the fully automated car, it is important to think about the system as giving an open-ended and not-known outcome (like in a social interaction). That means, in interacting with autonomous new systems, we need to keep space for the own system’s outcomes.

Where lays the “intelligence”? People nowadays oftentimes think that robots or automated systems are rather dumb. But one need to focus on the fact that humans are just experiencing the frontend of such systems and technologies. Of course, robots are mostly not capable of doing a diverse set of actions and therefore seem to be rather limited to a low intelligence. But this is not true; the real intelligence of these systems lays behind the frontend, at the backend.

People and machines need to be well aligned. Since the interactions are getting more and more intense and connected, e.g. timing issues are of huge importance. Thinking about production processes at an assembly line for example, it is inevitable that the timing of the robots perfectly matches to the tasks accomplished by the humans. Otherwise, the humans will quickly feel stressed or bored. Another fact regarding the connectiveness of humans and machines is that detailed investigations need to be done to figure out where to position the humans in the complex network of human-system-collaboration.

Errors and loss of perceived control

With relation to the increased connectiveness and therefore the increase in complexity, it is also always a question of who is responsible for errors. How does the system knows when there is an error? How to manage the timing of the timeouts in production processes? Especially the question of who is in control is very important to think about. You as a human need to know what the system does and what kind of information you need in order to be able to take over control. It is also oftentimes a question of ethics whether the human or the system should be in control of a certain situation.

Irony of Automation

The blog post ends with an interesting term called “Irony of Automation” which refers to the ironic situation that the more advanced a control system, the more crucial may be the contribution of the human operator. That means e.g. while using autonomous driving you need to take over in exactly the situations where you need help. So, there are still a lot of open spaces which we need to think about in the new kind of human-system-collaboration.

MMI2 – WS18/19
Guest Lecture (09.01.18)
Written by:
Susann Warneke
MMT Master – 3. Semester

 

 

Guest Lecture: User Experience in Automated Contexts: From Factory to Automobiles – Alexander Meschtscherjakov
by Lisa Bauschke

In the guest lecture with the title “UX in automated contexts – from factory to automobiles”, Alexander Meschtscherjakov presented the state–of–the–art research projects in the context between user experience and automation.
Prof. Dr. Alexander Meschtscherjakov is an Assistant Professor at the Center for Human-Computer-Interaction and Computer Science Department of Salzburg University. Additionally, he is leading the Car Interaction Lab and is the deputy director of the Center for HCI. At the moment, his field of study is dealing with the persuasive interaction technologies, contextual user experience, challenges of automation for HCI as well as automotive interfaces.

Transitions

In the last decades, the transition in the automotive industry, especially of vehicles, went through great changes. From the early stages of cars and vehicles to nowadays vision of future car driving like autonomous driving. But not only the car itself has changed, also the manufactory processes and the development of factory machines. The current manufacturing processes are moving from a human building process within to the direction of an interaction of machines. Looking back to the industrial revolution and the process which took place, we went through several development stages. Starting at the “industry 1.0” we are currently in the “industry 3.0” and already speaking about the “industry 4.0” which will be the complete and full digitalization of the industrial production. The idea is that people, machines, logistics and products communicate and cooperate directly with each other. It becomes a human computer collaboration.

Usability and User Experience

Currently, there are many definitions of user experience out there. But in general speaking, user experience involves several aspects of the interaction between a user and a product, service, environment, IT system, software, etc. It comprises the extent to which a product can be used. This holistic approach encompasses the entire user experience received when using a product. Users should not only reach their goal quickly and smoothly but also can create a positive output when using the product.
Additionally, usability is the way a product, system or service can be used by users in a certain application context to achieve specific goals effectively, efficiently and satisfactorily.
Through the transition which were just describe above the user’s interaction and usability with a certain product develops as well.

Direct and Indirect Manipulation

In human-computer-interaction, direct manipulation occurs when the system interacts with the user and the user immediately react to it. But there can also be a direct manipulation when you don’t actively interact with the system.
What does exactly mean manipulation in a world where you don’t interact directly with a system? Nowadays, people are surrounded by system and technology. They are using them automatically and in some cases without active perception. Therefore, the question appears is in-direct manipulation necessary in order to keep up with the development? Humans have among their strengths also limitations which needs to be taken under consideration.

New Forms of Interaction

With the changes which are currently taken place and will further develop in the future prospective, new forms of interactions are needed. One of them is the social interaction which can be classified in general as an open-ended interaction. In contrast, an interaction with a robot would be the interaction with an interactive system which just response to my input. But the system should have autonomous interaction in order to keep space open for the potential outcome as well as room for misinterpretation.

Where lays the “intelligence”?

In general, an artifact is perceived as being dump. But the intelligence of a system lays in the backend. The backend can be updated in order to perceive a different interaction. It might create a completely new user experience. The more “dump” part of the system could be called the interactive artefact. This needs to be differentiated from the interaction with the user itself.
When interacting with a system the timing issue can appear. When people are working on an assembly line, they are working in collaboration with the machine. But in such a process, the timing needs to be precise in order to be able to keep up with the speed of the manufacturing process.

Levels of Automation

The single levels of automation are diveded between high and low. There are in total 10 levels. The following two (10 and 9) emphasizing the highest levels whereas the number 1 and 2 are presenting the lowest ones.
“10. The computer decides everything, acts autonomously, ignoring the human.”
“9. Informs the human only if it the computer decides to.”
“2. The computer offers a complete set of decisions / action of alternatives.”
“1. The computer offers no assistance: Human must take all decisions and actions.”
When thinking about autonomous driving cars, one day the human is becoming an assistance of the machines, who will just take over the control for helping the system if needed. But at some point, driving an autonomous car could become boring since the actual tasks like driving, paying attention to speed limits, reacting the signs etc. will be done by the car. But at that stage, people are not allowed to do something else while driving, since they might have to take over the weal and be fully aware of the surrounding. The question appears, how to keep the driver in the loop? The complexity arises and at some point, the user needs to become part of the network.

Errors

The overall question in human-machine-interaction could become where is the error? Is it the system, the background or the human? Or is it the increased complexity with the increased connectiveness? How does the system and the user know there is an error? Most errors occur if there is no pressure to act and human are waiting too long until they are fixing the system.

Irony Of Autonomous

At some point, people could face the loss of perceived control in the autonomous driving context since the car is taking over. But what could be the role of a human being in the driving situation? One way could be that the user becomes to an assistance of the machine. This leads to an ambivalent situation since the development progress in terms of technology is of great importance but the driver itself will be deskilled and the manual control will be replaced by the system. The irony is the more advanced a control system is the more crucial may be the contribution of human operators. The human will be more in the role of a planner and problem solver as well as laying the ground for responsibility and documentation since the law is an important factor too.

MMI2 – WS18/19
Guest Lecture (09.01.2019)
by Alexander Meschtscherjakov
Written by:
Lisa Bauschke
MMT Master – 3. Semester

 

 

Guest Lecture: Instant Interaction – Florian Echtler

by Anna Hubert

Florian Echtler is a junior professor for computer science and media at Bauhaus-Universität Weimar, where he is part of the Mobile Media Group. His field of research covers the “ad-hoc creation of ubiquitous computing environments”.

When Mark Weiser formed the term “ubiquitous computing” thirty years ago, he envisioned a world where computing is integrated in all parts of every day life, making interactions with computers common, implicit and independent from devices. We seem to be close to this vision, most of us carrying our smartphone as a powerful mini-computer with us at all times. Still, the simple task of sending a picture to the person next to us requires multiple steps including using a cloud service in most cases. Despite its ease of use this fragmentation leads to data safety as well as performance issues.

In his research Florian Echtler is working on solving these issues with the concept of “instant interaction”. It uses all kinds of sensors – most of them already present in arbitrary environments – to connect devices in order to be independent from a backend or client-server communication. The only things needed are physical proximity and data about the environment. Several research projects using this concept are presented in the following sections.

PhoneProxies

This project addresses a technique for sharing content between mobile devices via interactive surfaces. Custom NFC (near-field communication) tags in combination with optical markers are placed on an interactive surface. Users place their mobile device on top of the tags which enables the user to view content on the surface or to drag and drop content across the surface to another device, allowing the two interactions “pickup” and “share”. In contrast to many other content sharing systems no time-consuming setup is needed.

MoBat

The idea of this project is to use ultrasound to localize multiple unmodified mobile devices. The first setup included a regular table with microphones in each corner of the table. The mobile devices emit barely audible sound pulses received by the microphones. By analyzing these audio streams the mobile devices can be located.

Ongoing research tries to improve that system by placing stationary ultrasound emitter in the corners of a room without connection between or knowledge about each other. Similar to the first scenario the mobile device sends a synchronization broadcast, the emitters respond with an ultrasonic pulse and the device calculates its position using the TDOAs (time difference of arrival). Some challenges for this project to work on include the Bluetooth synchronization or the fact that smartphones aren’t really designed to pick up ultrasound.

M³ (Massive Mobile Multiuser)

The M³ framework allows simultaneous interaction of multiple persons with public displays through their personal mobile devices. Main goal was to create a publicly presentable, playful tool that is easy to join. Thus players don’t need fix controllers but join the game with their own mobile device instead which shouldn’t require more than 30 seconds. As a trade-off between avoiding to install a program and reducing the latency to below 100ms to go unnoticeable, players need to join an extra WIFI network forwarding them to the webpage of the game. The game itself is implemented with pure HTML5.

ShakeCast

This project was inspired by the thought of simplifying exchanging business cards. As a new approach contact information is exchanged during a normal handshake from smartdevice to smartdevice carried on hand or wrist. The device e.g. a smartwatch detects the physical handshake and triggers setup-free information transfer between the users’ personal smartphones using Bluetooth LE broadcasts. However, many challenges accompany this concept, starting with the obvious cultural problem of people shaking hands with their right hand but usually carrying their watch on their left wrist. Another problem is the lack of accuracy during the identification of different kind of handshakes where currently many errors occur.

One possible solution could be the use of siamese networks. Two neuronal networks get the same task – in this case detecting and measuring the handshake -, but receive different signals. From the difference between these signals the networks learn where to measure the handshakes most accurately and improve the system.

Conclusion

The large amount of available technologies offers many ways to implement the different facets of “instant interaction”. The next huge step would be to go outside and test the described methods in an unpredictable environment outside the lab. In an unpredictable environment the used techniques need to prove their utilizability. Research will need to answer not only questions about how to gather and handle correct data but also questions concerning human factors such as the comfort zone of users. All in all there is still lots of room for developing novel interactions in arbitrary environments.

MMI2 – WS18/19
Guest Lecture (16.01.19)
by Florian Echtler
Written by:
Anna Hubert

 

 

Guest Lecture “Instant Interaction” – Florian Echtler
by Lisa Bauschke

Instant Interaction

This guest lecture was held by Jun. Prof. Dr. Florian Echtler from the Bauhaus University Weimar who is since 2014 Junior Professor for Mobile Media. During his guest lecture, Florian Echtler presented the concept of instant interaction which has the aim to enable ad-hoc interaction between various people and their smartphones with a fixed infrastructure but without the need of an intermediary. This blog article shows multiple studies in the field of instant interaction, which were presented by Florian Echtler.

The topic ubiquitous computing was first introduced in 1991 by Mark Weiser but has still nowadays immense importance due to the constant availability of mobile devices. The original idea related to the question how everyday devices can become connected and interactive.
The term instant interaction can be explained as a form of simple, immediate and accessible ad-hoc interaction which doesn’t require any form of central infrastructure, signing up or login or an app installation. It just requires data about the local environment. The idea is that if people in one room would like to exchange e.g. data they should not use intermediaries like cloud providers etc. in order to do so. Through the use of Bluetooth, cameras, sounds etc. an interaction without an intermediary would be possible since most mobile device are having these functionalities.
In the following part of the blog article, current studies in the field of instant interaction will be presented.

1. PhoneProxies – Sharing content (2014)

This study focuses in the possibility to use a mobile device as well as an interactive table in order to exchange data. Through the connection of the mobile device with the table through a tag, data can be exchanged immediately without the need to waste time while setting up a complicated connection. The tag (QR code) can be scanned by the smartphone and a website will automatically be open where pictures can be uploaded which are than shown on the table. You just have a direct connection between the table and the phone without intermediary. With that token you have the advantages that a much larger number of pictures can be shown in the table in a much shorter time.

2. CalendarCast – free/busy sharing (2016)

This study picks up on the issue to find a collaborative appointment. During the scenario in a meeting situation when people try to get an appointment with other people without seeing the content or exchanging calendar details. The idea is as soon as you start your own calendar, it will automatically broadcast the information to the others when you are free or busy. A free slot for everyone will be shown in grey or colored space. The personal calendar details are hidden and none of the other people need direct access to your own calendar. CalendarCast is a display which is receiving data from all other calendars and matches it on one screen. It is a rather specific scenario how we can use peer-to-peer communication through the usage of Bluetooth. One issue appears is that sending and receiving simultaneously is very difficult. Not all mobile devices are capable of this functionality. But one solution is mentioned to send data packages by default 10 times a second. If you lose data it is no problem since they repeatedly end the same packages all over again after a few seconds.

4. M3 – Massive Mobile Multiuser

This study researches on the simultaneous usage of mobile phones as a controller. The aim is that the user is doing something on the phone and seeing the immediate reaction on a second screen above. In order to achieve that the technology should try to stay below 100ms otherwise it might feel a bit sluggish for the user. To make this work, the user just has to connect the mobile phone through wifi with the network. The important part is to deliver the data as soon as possible to the server.

5. ShakeCast – Context exchange

This study is working on a peer-to-peer approach. The initial situation is that when people are shaking hands on a e.g. fair / exhibition they usually also exchange business cards with personal data on it. The idea behind this process is that the exchange of personal data through the paper business card should be automated through the recording of the hand shake. One common problem might be that people in western Europe are shaking hands with their right hand but wearing their smart watches (incl. the signals) on the left hand. Furthermore, it is difficult to find out when two signals belong together. The recording does not start at the same time and you do not want to look up at the Bluetooth ID even before you shake someones hand. The aim is to find out the correlation of data which belongs together and which does not. One future approach could be to look at the two directions (an even more exploratory approach). Instead of use e.g. a smartwatch the transfer data, a sensor ring and a wrist mounted sensor which are picking up data from several locations. Another approach could lead to hand-crafted features like Siamese networks by using machine learning (neuro learning). Networks can be designed to condense two signals. You just have to look at the output of the two signals since it should have the identical weight, etc. because differences could lead to further errors.

Future outlook

All presented projects have the aim to enable people to interact without the essential need of e.g. cloud-server infrastructure. Future experiments could move beyond the lab into unpredictable environments. Through further experiments these questions could be examined more closely.
How to trigger interaction (without annoying the average user)? How to create a shared local context from unaligned, noisy environment data? How to enable novel interaction methods (tangible interaction, AR/VR, etc.) in arbitrary environments?

MMI2 – WS18/19
Guest Lecture (16.01.19)
Written by:
Lisa Bauschke
MMT Master – 3. Semester

 

 

Guest Lecture on “HCI and Sports” – Florian Daiber

by Niklas Müller

Today sport training isn’t anymore about a trainer who is shouting at you to run faster. New technology has reached the field of sport. Through sensors and smart wearables it is possible to track everything you need to analyze your training and keep getting better. Florian Daiber is a post-doctoral researcher at the German Research Center for Artificial Intelligence in Saarbrücken who is focusing his research in the field of human computer interaction. At the 23.01.2019 he had a guest lecture on the topic “HCI and Sports” at the LMU Munich.

Examples of HCI in sports

Before the late 20th and 21th sport equipment had less to do with a better performance of the sportsman. After that a lot of research were done to support the athletes to get better or to provide a more comfortable performance. Famous examples are a swimsuit developed by Speedo and Nasa to provide a better performance of the swimmers or little motors for bikes, mostly determined as motor doping. Eventually a new science were created, the sport science. More and more guidlines were developed like “Daniels Running Formula” by Jack Daniels which provided tipps to increase your performance. But also concepts like regaining strength with ice baths were developed. Also in the first time in history in 1968 Plagenhof used a computer to calculate the force of the athletes foot while running. All of these aspects led to the modern sport science and the following possibilities to combine sports with human computer interaction systems.

Physical education

A short but important point is the physical education. HCI systems should not only support tracking and feedback, but should also educate the athletes and trainers. This could be useful so the sportsman can draw the conclusions of the data on their own and also understand what is happening in the human body or rather know what to change.

Entertainment

HCI systems can not just be used to support the athletes but also to support the spectators in the form of entertainment. Have you ever wondered were exactly the runner is in an marathon? Through for example tracking devices it is possible to track the runners and tell the audience in real time were they are. It is also possible to support both professionals and amateurs through gamification. When combining sport exercises with gaming elements it can be a good way for professionals to keep going with the training, for example in the winter when its hard to make sport exercises outside. For amateurs it could be a good way to get used to sport activities. A good example is Zwift, where you combine serious bike training within an gaming environment.

Wearable Computing

Because of todays technologie and the ubiquitous appearance of computers it is possible to find new ways of interacting with them. Especially in sports. What we are talking about is wearable computing. So computers that can be worn on the body like watches, glasses or sensors. Wearables provides some advantages. The computer is easy accesible and the user has new ways of interacting with the wearable computer. Also it is easier to track the user and his environment and to share the data with other wearables that are connected through a network.

Designing Wearables:

When designing wearables a few guidelines has to be kept in mind.

• Human users should be integrated in the design process as soon as possible
• Corpus collection of human data
• Data collection is challenging (synchronize different sensors)
• Ethical aspects have to be covered
• Lab studies are often not suitable

At the moment wearable computing is a large and interesting research topic. Nevertheless there are far more prototyps than commercial products. Some are not usable now in todays demands but there are also not widely accepted. Marc Weiser once said:”The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” – when the wearables fullfil this challenge they surely will be widely accepted in public.

Sports & activity tracking

At the end of the lecture Florian Daiber showed us some examples of sports & activity tracking. He pointed out that tracking methods are good for running or swimming but can be challenging for sports like climbing. In his studies, Florian Daiber is focusing on new possibilites for tracking and aid devices by connecting different sensors and different modern technology.

ClimbVis

Rock climbing involves complex movements and therefore requires guidance when acquiring a new technique. This project was developed to guide the users through climbing movements. Three different display options were therefore researched through a study. One display option was a lifesize model which was projected directly at the wall to show the climber the movements. Another option was just a video which was projected at the wall and the third option was a projection which was projected inside the google glasses.

Slackliner

It can be a difficult task to explain the slackliner what he is doing wrong and how he has to do the exercices the right way. This is why the project “Slackliner: Using Whole-body Gestures for Interactive Slackline Training” was developed. Slackliner is an interactive slackline training assistant which features life-size projection, skeleton tracking, and real-time feedback.

Footstriker

A problem with running especially for beginners can be the so called heel striking. After lifting the foot the heel is the first thing that touches the ground again. This can cause several injuries or at least leads to higher risks for injuries. A better possibility would be to put the toes to the ground first. Because it is hard for humans to change the behaviour after years of habit Florian Daiber and his team developed a possibility to change this heal striking behaviour. This wearable detects heel striking with a pressure-sensitive insole. If so the calf muscle is contracted by applying electrical muscle stimulation on it.

Conclusion

So why should be use human computer interaction in sports? The main point is the understanding of humans and athletes. Through the gathered data it is much easier to analyze the data and help the athlete to adapt the training to it. Trough entertainment systems it is also more likely that beginners can adapt to sports more easily. In the future there will be likely more hardware and software which is a wearable, medical approved and smart!

MMI2 – WS18/19
Guest Lecture (23.01.19)
by Florian Daiber
Written by:
Niklas Müller

 

 

Guest Lecture “Physiological Computing” – Stephen Fairclough
by Susann Warneke

Physiological Computing

According to Stephen Fairclough, physiological computing is a term that describes any technological system that incorporates physiological data from humans into its functionality or displays these data at the interface. That means, it is about computer systems that are able to react to humans in terms of e.g. changes of a specific interface through the use of specific human body measurements and the recording of quantitative data from the human central nervous system (including e.g. brain activity, patterns of muscle activation, outputs from the cardiovascular or respiratory systems and differences in the electrical conductivity of the human skin). This leads us to the range of different physiological interfaces that lead from overt intentional to covert unintentional, while covert and rather subtle, indirect forms of human-computer interaction (HCI) are of major importance in this regard.

There are two big things that need to be thought of while characterizing physiological computing. Firstly, there is the need of a close interaction between system and human. Secondly, it is about enabling explicit (reactive, direct feedback) and/or implicit interaction.

Basic component of Physiological Computing: Biocybernetic loop

The biocybernetic loop is the underlying mechanism behind physiological interactive systems describing the fact of information transfer between humans and machines. The loop can be divided up into 3 different main stages. The first stage of the loop describes the recalling of human data or signals (nowadays, there is a rising number of measurement variables that explain the physiological characteristics of people in a more accurate and extensive way). In stage 2, there is a so-called psycho-physiological inference, that means it is about a representation of the psychological state of the user to a computerized monitor – a transfer of psychological trackings to physiological systems. In stage 3, an adaptive controller of the system functions as data translator. It transforms the collected data into a system’s reaction used for enabling the software interaction adaptation.

There is a real design challenge of how to create these HCIs of different interfaces and for different use cases?

An example: Adaptive gaming – Tetris

In the research area of gaming and motivation, the so-called motivation-intensity model is frequently discussed and mentioned. It describes the relation between effort and demand and divides the users perceptions and motivational stages into 4 segments: (low effort/low demand = boredom, medium effort/medium demand = engagement, high effort/medium-high demand = zone, too high demand = overload). The third stage “zone” is oftentimes compared with a feeling of “flow”, in which a person forgets the feeling of room and time. It is declared as the “perfect” relation of demand and effort which results in the maximum degree of users (gaming) engagement.

The example in the specific field of adaptive gaming focuses on the overall goal to create a system that pushes the people into the “zone” stage. With the help of electroencephalography, EEG, signals of the human brain surface were measured and evaluated. So, neuroadaptive Tetris means changing (increasing vs. decreasing) the speed of falling blocks based on live EEG measurements. This should increase the fun and state of “flow” while playing. For making it as accurate as possible and for adapting to the changing psychological states of the test person, the system makes a decision whether to de- or increase the speed of falling blocks every 5 seconds. Important to mention here is the fact that in a situation like this, the system tends to have somehow a mind of its own, because they have information about a person, he or she doesn’t deliberately know of; an aspect that should be kept in mind.

Another example: Implicit tagging of media

Also in the case of media consumption, physiological computing opens up an interesting field of research and experimenting. Monitoring a person’s data while consuming media, more precisely while looking at a specific picture, could lead to information about a person’s interests, whether he or she likes or dislikes a single picture. With this regard and also in other scenarios, there is always one important question coming up: What to measure and how? With regard to the specific case of media consumption: How to measure a humans interest? Which human physiological attributes could provide information about satisfactory and dissatisfactory media consumption? Machine learning could be an answer in order to classify an interesting picture from an uninteresting one.

Where does it go in the future?

Overall Benefits

What could be achieved with the help of physiological computing? Could machines help people making life easier, more enjoyable, more engaging …? Looking into the future, the specific case of physiological computing has the potential benefit of increasing general entertainment, enhance engagement and social quality. In addition, it could also help with regard to human frustration, enhanced safety and it could lead to personalized technology.

Technological Trajectories

Concerning the specifics of the underlying technologies of physiological computing and the special characteristics of HCI, important and decisive aspects are complexity, diversity, freedom, evolvability, mutualism and sentience. So, as a future prospect, humans somehow have to learn to live with technology and to give technologies partly control over the mutual interaction. What will remain a critical issue in the next few years and will alsways be a topic of discussion is the question of data protection and privacy issues in giving complex data of oneself to a rather unknown system.

People are flexible but technology is always a one-size-fits-all solution, so, technologies should become more flexible and adaptive to individuals. It is the question of becoming a digital self and getting a dynamic digital shadow that adapts automatically to specifics of the individual physiological characteristics.

MMI2 – WS18/19
Guest Lecture (30.01.18)
Written by:
Susann Warneke
MMT Master – 3. Semester