Most of UX design (UXD) right now is for the screen, like mobile phones and desktops. It may have some auditory elements—like when a swipe on the interface is accompanied by a whoosh—but the discipline is skewed toward visual design.
Many practitioners as well as futurists predict that the field will see more designers creating multi-sensory experiences to engage with users. And with “new” stimuli come new reactions and interactions.
We’re already seeing folks using awesome technology to create “off-screen” experiences, but they’re mainly for games. So, I wanted to discuss how we’re using technology to create richer UX, especially outside games.
First off: What constitutes “beyond the screen UX”?
Let’s count any experience that engages other senses—sound, touch, motion, smell, and taste.
Many experiences are “visual first”, and they are enriched by designing other aspects of the user’s sensory experiences. But there are also completely non-visual experiences and interactions. Both were discussed during the session.
Key Themes Discussed
Voice UI (VUI)
The challenge in designing VUI is about simulating a more “human” interaction—allowing people to speak with a device without forcing a Spock-like language, be understood by this device, and receive a unique but reasonable response that doesn’t sound coldly formal and robotic.
Digital assistants like Cortana and Siri enabled us to learn extensively about “off-screen” human-computer interactions.
- When do users to talk their devices? A VUI seems to be the optimal method for communicating with users who cannot be overwhelmed with visual stimuli, such as drivers. We also uncovered a curious interaction issue: Do you think it’s weird to use Siri in public?
- How do users speak to digital assistants? Users tend to talk to digital assistants formally—sometimes using an awkwardly formulaic syntax—thinking that a more natural language is too complicated for the device to process. This is both a technical and UX issue: It’s not just about refining the technology that picks up the user’s input, but also about creating a VUI character that encourages users to speak using their natural language.
- Finally, how do users react when a digital assistant responds to them? Getting a predictable and formally stated response from a digital assistant reinforces user tendencies to interact “less naturally”. Have you tried having fun with Siri? You’ve probably realized that her antics can get old pretty fast. What does the fox say, Siri?
Sound Design in Storytelling & Journalism
With the resurgence of long-form journalism, publications are finding ways to make articles of several thousand words come to life. Apart from incorporating large hero photos and cinemagraphs, they’re also integrating sound into their storytelling to transport readers into different environments and to evoke certain moods. A good mixture of sight and sound drives the emotional elements of storytelling.
Out here, no one can hear you scream: http://highline.huffingtonpost.com/articles/en/park-rangers/
The Internet of Things
The smartest appliances (will probably) have a no UI approach. Instead of asking a million things about your personal habits through a UI, it’s an air conditioner that learns your microclimate preferences, your comings and goings simply by observing your for a week, and then configuring itself from there.
But is it enough for appliances to connect online? Designing smart homes has been tricky, because smart devices from different brands don’t often talk to each other. All devices considered individually may be smart with beautiful UIs, but it’s cumbersome to have to interact with them separately each time. For example: Searching for an address in your phone, and then having to key in that destination in your car’s GPS. It’s repetitive and can be improved by having a phone that connects to the car.
Google Glass & Augmented Reality
Google glass may have “failed”, but AR is bound to catch on. Existing use cases for Google Glass turned out to be more of a novelty. But we know the potential of AR in many industries and for many professionals—from surgeons and interior designers to construction workers. UXD for AR is visual, but unlike screens (2D), this is complicated by depth (3D) and motion.
UX beyond the screen steps into the realm of service design, especially when we talk about non-gaming applications. Designers need to consider the context in which certain technologies are used. They also need to orchestrate their presence. For example, how would you design a queueing system for a telco shop versus say, a hawker centre? Depending on established cultural norms and user expectations, designers can encounter elements of an experience that they can choose not to automate. If they do, extensive efforts arise to hide or disguise them in favor of a less industrial or more nostalgic vibe.
Design for Real Life & Compassionate Design
Beyond our sensory perceptions, design above all affects our emotions, our psychological states. At the same time, we also interact with technology and digital interfaces in various emotional states.
The way typical user tests are designed, there is an implicit assumption that people are mostly calm, cool, and collected when interacting with products and services. But that’s not always the case, and how users feel ultimately affect their ability to use a product. And sometimes, certain users appear only in specific circumstances and emotional contexts.
For example: Have you tried asking Siri to call the police or the ambulance in a state of panic?
Design for Real Life: https://medium.com/@sara_ann_marie/design-for-real-life-9c1bdd672d28#.8lu9if5q1
Building Empathy & Virtual Reality
People are exploring how we can use technology to build empathy or interrogate social issues. See: Gender Swapping using VR* and Design Fiction. Visa’s SocialSwipe transforms the act of swiping a credit card into a symbolic gesture of feeding the hungry.
Note: There was a quick discussion on whether VR is considered “beyond the screen”. We can argue that it is beyond the screen, because it engages kinesthesia—our sense of motion.
Augmented (hyper)Reality – Domestic Robocop: https://vimeo.com/8569187
Closing Note: Good UXD on and off screen
To some extent, the UX you design can only be as good as the technology you use. All the work is done in the backend so that users are presented with a simple screen that takes the minimum manual input to produce astounding output. With off-screen interactions, it’s much harder to make the technology “invisible” so that users have an organic experience that generates delight instead of frustration.
This article inspired this discussion topic: The Most Important Design Jobs of the Future — http://www.fastcodesign.com/3054433/design-moves/the-most-important-design-jobs-of-the-future