exp_04

Item Vision records and modifies the webcam image of the Virtual Reality (VR) headset HTC Vive and plays it back to human testers’ eyes in real time. By emphasizing highly saturated colors in the imagery and darkening less intense colors, household objects stand out while backgrounds, such as walls, lose their contours. Since colors are not only strengthened but also pixelated, the experiment applies visuals from digital games on objects and bodies that are physically present in the test room.

a flower that looks like out of a game world

research questions

  1. How do testers of the experiment interact with objects and architectures that appear visually different to them?

  2. When linking the aesthetics of digital games with tangible objects and spaces, will testers experience their gazes or their surroundings as digitized?

development

  1. The Virtual Reality (VR) headset HTC Vive is equipped with a webcam. At first, we researched how this webcam is usually used to get an idea on how we could reuse it. Additionally, we learned how objects in games are indicated as suitable for interaction.
When using games in what producers like HTC promote as VR, players sometimes lose their feel for the timings and workings of their non-virtual surroundings. The HTC Vive’s internal webcam can be used during gaming sessions to check back with the ‘real’ room by pressing a button on the HTC Vive’s surface, for example when players hear a sound from outside the game they cannot classify.
Many game developers make their games’ interface communicate clearly what the player is supposed to do. Typically, those objects that players can interact with stand out visually compared to objects that belong to the scenery or are purely decorative. Objects players can use in some way are called items. To distinguish them from everything else, they are more colorful and/or have a different artstyle and/or a different resolution. Light may be used to highlight items. Most games allow players to pick up items and move them around, in some games they can be stored and used at a later point.
  1. Next, we accessed the webcam of the HTC Vive by writing a simple piece of code in the game engine Unity, and applied it to a 3D plane. This allowed us to see a live image of our workspace on the plane in Unity.
  1. By writing another script that analyzes every pixel in the live webcam image, an algorithm figures out every pixel’s saturation. We used the outcomes of the pixel analysis to send a modified image back to the HTC Vive. Because we wanted colorful objects to stand out even more, we emphasized the colors in the live webcam image that were already strong. The image below shows a person holding a flower in front of blue curtains.
Webcam image with a pixelated flower but non pixelated room in the background
  1. To carve out the impression of seeing items and not just everyday objects in more color we darkened the original webcam image until it was black.

try or catch

Tester 01 took a red broom and turned it around in front of the VR headset for three minutes. He told us that the exclusivity of colorful objects in Item Vision enhanced his interest in them immensely. The most fun to him was that he felt like being inside a game in which he could touch objects.

Filip watching a broom in Item Vision the broom in VR
leo looking at a bathtub duck leo looking at a bathtub duck in vr
leo looking at glass cleaner leo looking at glass cleaner in vr

Tester 02 was very interested in seeing his body and the bodies of other people in the room pixelated. He moved his hands in front of the headset and asked other testers to walk around or towards him.

Tester 03 made us adjust the resolution of the pixels so that the live image she received consisted of a few really big pixels only. She said she liked to watch changes in light over a longer period of time. The following image shows light falling through three windows in a higher resolution.

light falling through three windows

During the days in which we developed the experiment, we brought colorful objects to our workspace just to see them with Item Vision. Our favorite object was a blue flower.

jasper watching a flower with item vision

conclusion

Most people are not very interested in everyday objects like brooms, cups, package materials or even houseplants. These objects are often very colorful, possibly to please the eye or to be easily found among other objects. Item Vision awoke an interest in these objects as long as they were visually modified – as soon as testers took off the VR headset, they did not care about the objects more or less than before the experiment.

The testers’ responses stressed what we had assumed in exp_02 - All is surface: Even though games are supposed to be immersive, three-dimensional images of the player’s self are weirdly excluded from today’s game worlds.

Testers said that the experiment made them part of a game-like world instead of saying that it made game aesthetics a part of their world. We can relate to that impression: When using Item Vision we knew that we were looking at a screen in a headset that displayed the room and its contents in a slightly different way, and nevertheless it felt like we had switched to another layer of possible perceptions.

Maybe it is easier to ascribe the modified perception caused by the experiment to a virtual or digitized reality, instead of including new and other perceptions into what one calls reality. In upcoming experiments, we would like to find out more about why we are so prone to build up a variety of realities with different namings instead of accepting digitized perceptions as part of reality.