exp_05

A synchrony establishes a real time connection between a physical and a virtual space that look deceptively similar. The setup consists of a gray box with a wooden cube inside that can be altered from two realms: human actors and algorithms.

the virtual box on fire

research questions

  1. How do delays, skips and glitches effect the plausibility of a space?

  2. How do humans cope with an interface that acts on its own will?

development

  1. A potentiometer is a simple analog component that can be rotated to adjust a voltage. It usually acts as an interface between human and machine, for instance as a knob for adjusting an audio volume. We decided to use this component to enable test persons to interact with a virtual space by converting their kinetic force into digital values. First, we soldered our potentiometer to an Arduino board, which we then connected to our computer via USB.

  2. With the Arduino software, we uploaded a bit of code onto the Arduino board. When powered with five volts, the board constantly measures the potentiometer’s current rotation angle. By establishing a programmed interface between the game engine Unity and the Arduino software, we received the measurements in Unity and could work with them.

Arduino sending the potentiometer’s values:
void setup() {
  Serial.begin(9600); // the connection with Unity via the Serial port is established
}

void loop() {
  val = analogRead(potPin); // the current potentiometer value is read
  Serial.println(val); // the value is sent to Unity
  delay(20); // wait for 20ms until sending the next update
}
Unity receiving the potentiometer’s values:
void Start() {
    serial = new SerialPort(SerialPort.GetPortNames()[0], 9600, Parity.None, 8, StopBits.One);
    serial.Open();
}

void Update() {
    serial.BaseStream.ReadTimeout = 64;

    try { // try to receive a new potentiometer value that was sent by the Arduino
        string s = serial.ReadLine();
        int old = potentiometerValue;
        int.TryParse(s, out potentiometerValue);
        Debug.Log(Mathf.Abs(old - potentiometerValue));
    }
    catch {} // if there was no new value, do nothing

    // update the rotation of the wooden cube according to the last potentiometer value
    transform.localEulerAngles = new Vector3(-90f, potentiometerSpeed * (float)potentiometerValue + offset, 0f);
}
  1. We then created a default cube in Unity and applied the measured rotation values of the potentiometer to the rotation values of the digital cube. Even though rotating the potentiometer translated into a rotation of the cube in Unity, the feeling was more as if using a game controller or a remote. To deepen the immersion, we decided to make the physical space look as similiar to the virtual one as possible. We decided to built an opaque box, because it is a shape that can be easily crafted as well as 3D modeled.
  1. We bought the wood that we needed to build the physical box, glued it together, drilled holes for the Arduino’s USB cable and the potentiometer, and cut a thick cardboard into shades. Later, we covered the object with gray and black spray paint to make it look untextured, similar to a raw 3D model.
building the box
  1. The university’s wood workshop had a nice hand-sized wooden cube in the leftovers. By using a technique called photogrammetry we took high resolution images of the wooden cube with a digital camera and created a 3D model in Blender. We drilled a hole in the non-digital cube and attached it to the potentiometer. The virtual wooden cube was imported to Unity to replace the default cube we had used before.
photographing the wooden cube from all sides
wireframe textured wooden cube
the wooden cube as a rotating 3D model
  1. Since we already calculated all the dimensions of the physical box when we had the wood cut, it was no problem to quickly create a three-dimensional counterpart.
the box around the cube rendered in the 3D program Blender
  1. Having all virtual resources imported to Unity, we were finally able to light both spaces. We picked a spot in our workspace that had very little ambient light and pointed a single bulb lamp in the physical box’s direction. Then we did the same for the virtual box, which was easier because we were independend from physical materials and their quirks. After we applied simulated Ambient Occlusion, reflections, bloom and some noise, there were only minor visual differences between both cubes.
the virtual setup of the installation the physical setup of the installation
  1. As a last touch we wanted to give back some control to the software, to balance power between human and machine. In the final installation, the physical piece and it’s virtual counterpart are totally in sync while the interface works just as expected most of the time: rotating the physical cube results in an identical rotation of the virtual cube. We implemented occasional glitches to make the two spaces go async or behave completely different without warning.
little light dots flying around the virtual cube

try or catch

When presented right next to each other in an exhibition-like situation, the physical space on a pedestal with a camera-like contraption on a tripod in front of it and the virtual space visible on a screen, visitors studied the installation from a distance first. We prompted them to touch the physical wooden cube and to rotate it to their liking. When moving their hand closer, they expected the hand to be captured by the camera and displayed on the screen, which did not happen. Their reaction was to hesitate and pull their hand back a little.

A moment later though, they overcame their disorientation and grabbed the wodden cube, played around with it, seeing their doing mirrored on the screen.

Some visitors flinched when the virtual cube on the screen burst into flames but felt soon entertained with the introduction of fire, lights and other randomly appearing glitches caused by the Non-Player Character. When the glitches were gone again, the standard wooden cube became trivial and the visitors tried to replicate the glitched states.

conclusion

The testers were quick at processing if a situation is safe and can be investigated further. Even after noticing a fault or discrepancy between the spaces, the testers continued their haptic interaction. They were playful even though they were largely in the dark on how the installation worked. The connection between the physical and the virtual wooden cubes was a given to them.

Only the missing of physical body parts in the virtual scene, combined with a occasional jerks and sub-second delays of the virtual cube’s rotation, created a kind of itchy, subconscious discomfort.