Interactive Architecture Lab | : Design for Performance and Interaction
The Interactive Architecture Lab is a multi-disciplinary studio interested in the Behaviour & Interaction of Things, Environments & their Inhabitants. We design, build and experiment with Responsive Environments, Robotics and Kinetic Structures, Multi-Sensory Interfaces, Wearable Computing & Prosthetics.This website shares our portfolio of work and daily research through our blog
The Interactive Architecture Lab is organising the first Bartlett Creative Coding Summer School 9-20 September 2019. The course introduces participants to computer programming as a creative discipline for expression. Whether you are an artist, designer, or architect, creative coding can offer up completely new ways of thinking and making.
A ten-day coding course for students, artists and professionals aged 18 or over.
This course has been specially tailored to teach coding in the context of the visual arts and design. Participants learn to use programming to create visuals and experiences from scratch, gain a better understanding of how programming can inform design, and learn how to integrate programming into your own creative practice.
Participants receive an introduction to coding concepts, including data-types, variables, functions, object-oriented programming and simple interface interactions. They also learn to implement simple math, geometry and motion principles into code. By the end of the course participants will be comfortable with writing code within the processing environment for creative design explorations.
Current and incoming Bartlett students: £750
Other institution / Concession: £1000
General admission: £1750
Drawing visuals with code
Pattern creation and Generative art
UCL at Here East
Queen Elizabeth Olympic Park
Is it for me?
This course is suitable for students, artists, designers and other professionals in the creative industry who want to bring their practice to a new level. No prior programming experience is needed.
Participants will need to bring the following materials with them to the course:
A desktop or laptop computer
Processing software, available to download online. Processing is a programming environment and language which has been used in the creative industry for over 20 years. Being open source, it is free to use and has been adopted as a tool of choice by creative people around the world
During an architectural experience, the architectural space and the human mind infinitely interact. Spatial experience is a non-ending interactive loop where the perceptual space changes by the mind, and the human mind changes during the current experience. The thesis aims to illuminate the brain processes that occur during an interaction between a user and an architectural space. The question of ‘how could we design a physical interaction where space has memories of all the individual experiences through time’ underpins the investigation.
Perceptual space is a product of our consciousness, a subjective representation of what an architectural user experiences; it is the outcome of a process of thought (Damasio, 2010). The experienced physical space is subjectively perceived as a unique model of the created mental space. When subjected to a perceptual process, an architectural space is constantly understood in a unique and different way. Chapter two explains the perceptual process of an architectural space that leads to the creation of a subjective inner reality. In addition to this, the role of memory and consciousness in this process is also described.
Neuroscientific studies have justified that the human mind changes during an architectural experience. The entity of mind, thoughts, the sense of self and the personality are shaped through life experiences (Begley, 2009). In the third chapter, the research focuses on the way that experience affects and changes the human brain.
The theoretical investigation led to the design of the project ‘íchni’ which is illustrated in the fourth chapter of the thesis.
Experiencing an architectural space is an endless interactive relationship between the architectural user and the designed space. Individuals are in constant negotiation with the surrounding environment. The negotiation happens through their bodies, since the body is their interface with the outer world. During the interaction, the human brain constantly tries to understand what the body is being involved in. According to the neuroscientist Antonio Damasio, the external world is ‘mapped in the brain and mind’ (Damasio, 2010, p. 91) through the interaction between the body and the environment. In this continuity of space-body-mind, unique mental representations are created in the user’s mind, which constitute the result of a spatial experience (Neisser, 1967).
This paper seeks to outline the interaction between an architectural environment and its user. More specifically, the main research objectives of this investigation were the brain processes that occur during a spatial experience and which eventually result in the creation of inner mental representations. The question that drove this investigation was how we could design a physical interaction where space has memories of all the individual experiences through time.
An architectural environment is not interpreted by the users in a general and objective way. On the contrary, the architectural debate, that our interaction is, happens from a relative and not fixed point of view, given the fact that humans, as subjective beings, are specified by their personal emotions, thoughts and life experiences (Kosslyn, 2005). The process of perception is described in the first part of the paper in order to highlight how the subjective inner reality is created during a spatial experience.
Following that, the role of memory during the perceptual processes is analysed. During perception, mental representations will be created so that what the user perceives can be understood. At the same time, those images will be recorded through memory in the brain and they will influence future interactions/ perceptions. In addition to that, memory also filters information that will be restored and used during a perceptual process (Blakemore, 2018). For this reason, a description of memory types is made in this part of the thesis. Memory types will be analysed in three different categories. First, types of memory according to the amount of time they keep the data stored. Secondly, types of different consciousness range during data storage. Thirdly, a description of different memory types during recall information will be made.
Subsequently, consciousness’ contribution to making us aware of a perceptual process is explained. Space is a product of thought produced by the user’s subjective self (Damasio, 2010); the notion of self that is introduced by the concept of consciousness is going to be presented. The extent and scope of consciousness will be described as well. This scope refers to the ability of consciousness to make us aware of both present and past perceptions. Additionally, its role in the body-mind continuity will be illustrated.
This paper also refers to neuroscientific theories which have justified that brain and accordingly the mind change through our life experiences (Begley, 2009). The user’s experiences shape the entity of mind, thoughts, the sense of self and the personality. Therefore, the conductor of the inner reality, which the human brain is, is affected by experiences. By introducing modern neuroscience concepts, the case of generating a personal view and a subjective reality as a high-level and complex human brain process is described. So that these ideas can be communicated, the manner of brain function and information transmission are explained in this part as well.
Eventually, according to the philosophical, cognitive and neuroscientific research, the design project ‘Choreographed Traces’ which is currently in progress, is described. This project aims to create a virtual environment which is constantly being sculpted by the subjective and different way in which every visitor understands and interacts with the physical designed space. Its purpose is to visualize something intangible, but real; to highlight this sense of flux between the user and space. Not only that, but the suggested environment has memory, and captures every user’s experiences. This environment will constitute a collective library shaped by users’ experience traces.
2. BACKGROUND | PERCEPTION OF SPACE
2.1.1 The perception Cycle
The perceptual process defines the relationship between the user and the physical space. During perceptual experience, the human mind constantly processes information that it receives from the environmental stimuli in order to understand and interpret where the body is being involved (Neisser, 1967). Every stimulus -whether visual, auditory, gustatory, olfactory or tactile – undergoes a brain process in order to create internal representations of the environment. This train of information processes is continually never-ending and ongoing; indeed it ‘is a cyclic interaction with the world’ (Neisser, 1978, p. 104).
At the same time, the way that we perceive our surrounding environment is selective, thus subjective (Gibson, 1950). The user of an architectural space is characterised by unique emotions, thoughts, memories, and personality traits. Moreover, our life experiences shape our different view of the world. It would be therefore impossible to assume that different people would share the same perception of a space. Even if the architectural space is the same, the process of perception is always different.
As a result, in addition to the existing physical space, another one, a unique one, is formed in our minds. The information that is collected by our sensory system, as well as the unique perception that every individual creates, determines the inner space that is formed. Given that information processing is different for every person, this it leads to unique mental models for every individual.
2.1.2 The visual buffer
The human body is the interface with the outer world. Environmental stimuli are captured by different bodily devices, for example, the eyes, the ears, the skin, etc. Those signals of received information will then be transmitted and processed in the brain. Brain processes will try to make sense of what the body is experiencing, by creating and storing information in the form of the inner representations.
As vision is the greatest channel of stimuli and information to the mind, visual perception is one of the basic functions of the human brain (Gibson, 1950). Without any doubt, visual perception is a fundamental brain function occurring when we are experiencing an architectural space. According to neuroscientist Stephen Kosslyn (2005) visual perception is divided into two systems: the “bottom-up” system, which operates with real-time received information, and the “top-down” system (fig. 1). The top-down system, in contrast with the bottom-up one, is driven from stored information which is obtained by our memory, our knowledge, our thoughts, etc. Accordingly, our perceptual outcome, the imageries, is never based solely upon the real time incoming signals.
The fundamental organ of vision is the eye, one of our perception devices. The human eye has the capability to convert light energy (the incoming image) into a form of electrical signals which can be recognised by neural cells. According to Kosslyn’s bottom-up system, the incoming signals, after their recognition from neuron cells they are transmitted to the groups of neurons which form the visual pathway. The incoming images are deconstructed into different qualities that define form, movement, colour and are then advanced to different neural processing sets (Kosslyn and Osherson, 1995).
The aforementioned groups of nerve cells opt for the visual information they are interested in, while completely ignoring others. This function is what Kosslyn defined as the ‘attention window’ (Kosslyn, 2005). Attention window is the selective process of specific information from our ‘visual buffer’ (Kosslyn, 2005), i.e. all the available visual information. The selected information will subsequently be thoroughly processed. Perception, therefore, is governed by expediency and selectivity and vision is a key feature of active selectivity. The active and discriminative nature of the visual process contributes to the subjective understanding of an experience.
All the stimuli that come into mind contain a wealth of information that the brain requires to process, understand, classify and store in its different memory areas. In order for the users to comprehend and interpret a visual image, they must also compare the visual impressions that are perceived each time with previous, stored ones. According to Kosslyn’s top-down system (Kosslyn, 2005), incoming information is associatively connected with existing brain information. If incoming information was solely picked up and not processed by the brain, as for example Gibson suggests (Gibson, 1950), it would be useless.
Fig. 1. Diagram of the perception cycle (2018)
2.1.3 The Minds’ Eye | Information processing in the brain
Perceiving is a ‘constructive’ process rather than a passive one. Ulrich Neisser explains through his model of ‘mental imagery and perception cycle’ (Neisser, 1978) that perception is a continuous process between the perceiver and the architectural space. An architectural user is constantly processing incoming information, and his perceptions are in constant interaction with the environment. Internal information structures are constantly created, and the repetitive nature of perception makes him foresee more information from the surrounding space. Neisser, with his model of perception cycle, wanted to highlight the perceiver’s contribution to this process.
According to Neisser (1978), the process of visual perception must include along with collection, the prediction of stimuli as well. ‘Anticipatory schemata’ (Neisser, 1978), a term that he defined, are internal structures helping the brain to anticipate information. Schemata operate as 2d plans constituted of real-time and recorded information. Schemata are not percepts themselves, neither are capable of producing one. They are a phase of the perception cycle occurring when a user interacts with an environment.
The role of schemata is to obtain more information, and along with pre-existing brain information determine which information is going to be collected and which will be foreseen (fig. 1). Therefore, they determine the perceptual observations of a user, and they will influence what will be seen, as well. The user constantly anticipates specific information becoming available and collecting it. When the user collects them, the schema is going to transform and then he/she will anticipate more information. Hence, the perception becomes a creative process of constructing plans where the perceiver’s contribution becomes crucial. In fact, the importance of perceiver’s contribution enhances the concept that subjectivity governs the perceptual process.
According to Neisser, a schema is an inner structure that ‘changes by experience, and in a way by the “object” of observation’ (Neisser,1978). The ‘schemata’ are not passive. They direct the movements, they receive information when it is available, and they release information which will be subsequently processed. Not only that, but schemata operate on pre-existing schemata. Therefore, past experiences will determine the process of the present ones, since users acquire information according to past schemata.
During the construction of schemata, chunks of information are being processed. As claimed by the parallel processing model (Neisser, 1967), the human brain can process multiple information at the same time. Information from different sources is subject to simultaneous processing by being distributed in different brain areas.
In general, the perception cycle is an interactive process; an interaction between ‘schemata and available information’ (Neisser, 1978). At a specific point of time information will create a schema; then this schema will become available and it will become part of a bigger schema in order to anticipate information. The anticipation will change it, since the world itself changes, and it will lead the user to acquire more information.
2.1.4 Mental imagery
The last stage of the perception cycle is the creation of mental imagery. Mental images refer to visual mental representations, namely the mental images of visual information. They contain information from stimuli that are not available at that time, but they are stored in our memory. Mental imagery does not constitute a new influx of information but the “perception” of stored information (Kosslyn and Osherson, 1995).
Mental images are nothing more than representations of what is occurring in our minds; a projection of the human mind that tries to understand the outer world. “A mental image is not an accurate representation of the world, but a map from which the subject receives information, which is not yet recognized and analysed. It is the inner side of spatial prediction “ (Neisser,1978). Imagery constitutes a larger and more complex schema (fig. 1).
Under those circumstances, in every spatial interaction a unique representation is created in the user’s mind and stored in memory for future use and reference. Identical stimuli, identical spaces, always result in unique representations in human’s minds.
Owing to the fact that mental images are created by the current stimuli and the memories of an individual, they are a reflection of the individual’s self. They enclose all the physical, emotional and cognitive attributes of a human being acquired in their life until their current experiences.
2.1.5 Mental Space
As mentioned earlier, perception is not a passive representation of the experienced environment. On the contrary, it is a creative process of an inner mental space. Additionally, it determines the predictions that make us capable of receiving any kind of information, when they become available. As Ulrich Neisser claimed, our eyes see what the mind has predicted (Neisser, 1967).
The mental space is defined as the specific space that does not contain faithful representations of the real world but is formed through the idealized cognitive models of the subject. These cognitive models are a way of representing knowledge in a semantic frame. The mental space is built through the representations of the mind and the knowledge of the subject (fig. 1). Therefore, it does not rely on the reality of the external environment in which the individuals reside, but rather in the attempt to rewrite it through the processes that occur in the individuals’ brain.
John Locke gives another name for the mental world. It defines it as: “a pure space, that is, as an objective reality that corresponds equally to a universal representation of intellect. It is a simple idea of the mind, completely distinct from the idea of solidity that accompanies the physicality of the bodies” (Terzoglou, 2009).
Despite not being a real representation of the outer world, mental space is a real world itself. As Henry Bergson stated (Bergson, 2016) the imagery that we create during a perceptual process is nothing more than reality. Anything that appears in our consciousness is nothing more than reality.
2.1.6 The Element of Time
As discussed in the previous chapters, perception is a constant creative process. Three-dimensional structures are created and continuously modified by experience. The critical element that defines the creative aspect of perception, is time. The user of an architectural space perceives an environment constantly in subsequent time contours (Neisser, 1978). Space perception is a chronology of events and time is consecutive and progressive. As a result, nothing can be stable and fixed under its realm.
From one time section to its subsequent, mental models and the mental space created by the user during a spatial experience are altered. In addition, from one fixation to another, the information that is going to be processed by the schemata has been updated. Even the more solid environment, the same architectural space, changes from one time section to another according to the different fixations. Countless images are created for the same environment.
As a consequence, we never perceive the same space, particularly in different time sections. Our mental space, as a result, is in constant negotiation between the interactive process of perception and the surrounding space.
2.2.1 From Representations to Memory
Through the individual’s perceptual abilities, internal mental images are composed in (architectural) users’ brains. An entity that reflects reality is created. These reconstituted images are the users’ attempt to understand reality. Human minds are instantly creating images of anything that happens inside and around them, in order to make sense of it.
The construction of those images requires the processing of information; incoming and new, as well as preexisting ones. Not only that, but the incoming information interacts with stored ones, during the imagery process. Eventually, the images will be stored in the brain for future use. Consequently, memory plays a crucial role during the perceptual process. Henry Bergson mentions in his book ‘Mind Energy’ that ‘on the side of perception there is always memory’ (Bergson, 2016).
Additionally, every individual has different memories obtained from personal life experiences. The ways that memory will select how to store and/or recall information vary between users under the same circumstances. The result will be the enhancement of the subjective understanding of an architectural space.
2.2.2 Encoding and Storing | Memory Function Models
‘During the formation of mental imagery, the brain is encoding, recording and storing the respective patterns and must retain a path to retrieve the patterns for the attempted reproduction to work’ (Damasio, 2010, p. 131). Neuropsychology develops various evolutionary memory function models, categorizing them according to their function and time duration of storing data that they hold.
The Atkinson and Shiffrin (Sternberg, 2011) model represents memory as a repository and storage space for information and events. The aforementioned model essentially categorizes memory structure. Their proposal separates memory into three memory systems of information restraint: sensory memory, short-term memory, and long-term memory.
Sensory memory is considered the first storage space for a large amount of information, which will then be gradually transferred to the short and long-term memory.
Several researchers talk about the existence of a virtual memory (Sternberg, 2011), namely the existence of a distinct optical stimuli recording stage, which is thought to hold information in image formats. So, visual information is advanced to memory via virtual memory which holds them for a very short period of time. Under normal conditions, this information will be forwarded to next mnemonic systems or will be erased forever if they are overwritten by new ones. This results in the subject having little or no access to the sensory memory content
Consequently, some information moves to the next memory system that is short-term memory (fig. 2). This system not only holds the information, but also contains some control procedures which are responsible for the flow of information to and from the long-term memory system. The information remains in the short-term memory for about 30 seconds, unless the subject uses internal repetition to hold it longer (Blakemore, 2018). According to George Miller (Sternberg, 2011) the system, at any given time and with a wide variety of information, can hold seven elements with deviation ±2.
The last storage system is the long-term memory. Its function is to store information for a long time, perhaps forever. Its limits are inconspicuous, as there is no fixed number of information it can contain, nor a specific amount of time that keeps the data stored. In accordance with Stephen Kosslyn (2005), long-term memory representation is multimodal, because it can encode a vast amount of information, deriving from different stimuli. In general, long-term memory is a womb of long-durative information which helps to identify an input during an encoding process.
“The objects which surround my body reflect its possible action upon them.” (Bergson, 1988, p. 21)
As humans, we would like to believe in the autonomy we hold over our actions. However, we have to concede that we do not act entirely on our own accord. As Bergson writes, the environing matter participates in the production of our thoughts and actions by suggesting to us how to act upon them. Humans are not “free willed” as we would like to think we are. We are not accustomed to the concept of non-human entities possessing agency, and that our agency is co-constitutive of both human and non-human influences. By equating inanimate matter with their corresponding action possibilities, James Gibson’s ecological perspective on affordance (1979) illustrates the non-disassociation between the human animal and the material environment. It also grants us a glimpse into the plenitude of potential actions offered to us by our environment, although we often find ourselves looping through the same motions. This is due to our environment disposing us to act in specific manners, which are then drilled, over time, into our muscle memory; our bodies are preprogrammed to perform the same habitual actions when confronted with similar situations.
French novelist, Georges Perec (1997), in questioning the habitual, bemoans that our conditioned way of living has desensitised us, stating that “this is no longer even conditioning, it’s anaesthesia. We sleep through our lives in a dreamless sleep” (p. 210). We cruise through our everyday spaces in routine choreography – waking, eating, moving, sleeping – ambling through life like a somnambulist. It is time to wake up.
This paper investigates how architectural devices can provoke the body into better and more actions. The title of this paper is a wordplay, using parentheses to tease the word ‘habit’ out from ‘habitat’. It was George Teyssot (1996) who identified the etymological association between the two words, stating that “habitations are actually places for long habits” (p. 53). This connection between the two words reveals the normative architectural notion that a house should be designed to reinforce habits. This paper, on the other hand, challenges conventional design objectives of reinforcing our conditioned way of living and advocates for the contrary; architecture should disrupt the habitual.
The etymological relation between the two words also resonates with the enactivist model of cognition that our behaviours arise from the dynamic interaction between us, as the organism, and the environment. It unveils the situatedness of our actions within a particular context, and as Teyssot (1996, p. 53) continues to define, habitats are “places where habits may be inscribed in a space that awaits them”. This paper adopts the definition of the body as an ecological site in which our being is defined by its relation to the world. The body exists not as a distinct and autonomous entity, and the various mechanisms of our thoughts, actions, behaviours, experience and perception manifest through the dynamic and complex interweaving with the environment. As our agency is constituted by both human and non-human forces, there is the potential for physical devices to nudge humans into alternative actions. Conventional designs often disregard the suggestive influence inanimate objects have on our behaviour. Therefore, designers need to understand their role in the production of the material environment and the capacity of the things they create to affect people’s behaviours. What actions await in the spaces we create?
2. Architecture’s Action Form
2.1 Architecture as Verbs
We have grown accustomed to viewing the world as solids, and to the formal appreciation of architecture being its primary design language. Can we conceive of an alternative architectural vocabulary that consists not only of nominative terms that limit spatial appreciation to forms, surfaces, lines and curves? An architectural language in which action is the form? Pallasmaa (2005) proposes that architectural images have less of a noun than a verb form. Rather than thinking of a door as a panel with a knob, we can think of it in its suggested act of entering and exiting; instead of thinking of a chair as a surface propped by four legs, we can think of it in its implicit action of sitting. Architecture then can be conceptualised in terms of its solicited actions instead of its physical attributes. Spatial structures thus function as a condition and a facilitator for human life, or as Pallasmaa (2011, p. 123) beautifully describes, “promises and invitations” for actions.
In his paper discussing the sociology of a door-closer, Latour (1988) anthropomorphises the nonhuman, technical mechanism of the hinge entrusted with the responsibility of ensuring that the door is kept closed. He humorously equates the engineered combination of the hydraulic piston and spring with that of a “well-trained butler” (p. 302). This mechanical butler has the ability to prescribe behaviours of users passing through and to discriminate between the types of users who could enter or exit. His choice of words presents the nonhuman door-closer as an entity with an agency. For example, in an earlier design without the hydraulic mechanism to ensure the slow closing of the door, the door-closer hustles users to move through quickly or risk getting slammed in the face; a “very rude, uneducated porter” (p. 301) as opposed to a polite butler. Latour’s anthropomorphic analogies enable us to perceive nonhuman entities beyond their object form. Similarly, Gregory Bateson (1979, p. 109) described the switch as “the thing that is not except at the moments of its change of setting, and the concept ‘switch’ has thus a special relation to time. It is related to the notion ‘change’ rather than to the notion ‘object’”.
According to Gibson (1979), affordances are “action possibilities” offered to an animal by its environment. In his seminal paper, he distinguishes between the physical properties of an object from its implicitly suggested actions – noun versus verb form. The suggested actions of material surfaces can be what we, the animal, perceive as “climb-on-able or fall-off-able or get-underneath-able or bump-into-able”. By adopting an affordance-based definition of architecture, we perceive spatial elements not by the physical delineation of their surfaces but as cues and lures. Architecture then becomes the encounter of the physical body meeting space.
Conceiving the environing matter in their “verb-essence” (Pallasmaa, 2005, p. 64) unveils the performative nature of objects. Thus, it might be useful to look into theatre studies to help us in shifting our understanding of architecture from a noun to a verb form. After all, the “construction of action” (Easterling, 2012) is the basis of theatre. A good performer does not play “being the mother” – noun form – but instead portrays “smothering a child” – verb form (Easterling, 2012). In immersive theatre productions from companies like Punchdrunk and Secret Cinema, the audience is free to roam the set and craft their own narrative based on whichever course they choose to embark. Actors are not confined to an elevated stage but instead, are planted throughout the entire space where they would enact their scenes before moving on to the next place. These planted actors function as a performative instrument, cueing and influencing the course of actions taken by each audience. As such, an audience may attend the same theatre production multiple times but generate a different narrative each time. Architectural objects should operate like planted actors, triggering users into actions while at the same time, offering a myriad of action possibilities – performative, but not prescriptive.
2.2 Non-disassociation between Body and Environment
Latour’s paper mentioned above, titled Mixing Humans and Nonhumans Together, proposed a new sociology vocabulary as a critique against the then accepted study of humans without any association with nonhumans. He concludes the paper by claiming that “studying social relations without the nonhuman is impossible” (Latour, 1988, p. 310). Adopting the theory of affordances ensures that the environment is always relative to the human animal, and vice versa – an absolute non-disassociation between the body and the material environment. It also acknowledges nonhuman entities as co-constitutive of human agency. We sit on a chair because the surface of the seat allows, and sometimes coaxes, us to sit.
Shusaka Arakawa and Madeline Gins (2002) developed a concept of an ‘architectural body’, where they view the body-environment as a continuous assemblage with no distinction between where the body ends and where the environment starts. The environment is to be a second, third, and even ninth (or so they have counted) skin of the body. They articulate the concept of embodiment with their theory of ‘landing sites’ which maps out the body’s sited affectivity relative to its environment. The body is intrinsically sited within a space and the links between the body and environment are manifold, so that it is impossible to abstract the body out of its surrounds. The intention to study a person would prove futile unless one develops a “person architectonic” and examine the sited organism as a whole (Arakawa and Gins, 2002, p. 6) – the body as an ecological site.
Fig. 1 Visualisation of Architectural Landing Sites by Arakawa and Gins (1997)
Fig. 2 Visualisation of Architectural Landing Sites by Arakawa and Gins (1983-85)
Arakawa and Gins classify the body’s sited affectivity into three categories: perceptual, imaging and dimensionalising. The perceptual landing site describes environmental cues that managed to enter the conscious awareness of perception. Many of the stimuli in a person’s surrounding fall below the threshold of active awareness, outside the focal areas of the perceptual landing sites (Arakawa and Gins, 2002), leading to the misconception in some that they have sovereignty over their own agency. The active, on-going participation of the environment on our agency is simply imperceptible. William Connolly (2011, p. 150) defines affect as “an electrical-chemical charge that jolts or nudges you towards positive or negative action before it reaches the threshold of feeling or awareness”. By defining persons as sited entities, the notion of having absolute autonomy over our actions starts to disintegrate. Humans are actually “hackable”, malleable beings. This is evident not only in our physical interactions but also in the digital realm, where clickbait has proven successful in influencing our internet activity and the information we consume. Enactivism is a model of cognition proposed by Varela et al. (1991) as a criticism against the traditional opinion that postulates the mind as a disembodied, passive, information-processing entity that functions independently of the environment. The enactive model highlights the intricate and dynamic connectedness between the organism and environment, and thus debunks the myth of free will, illustrating how “outside environments fold into the fabric of our bodies” (Dewsbury, 2012, p. 79).
There exist a rich diversity of bodies; each of different dimensionality, of varying shapes and sizes; each embodying different genders and cultures; each with its different set of skills, abilities and disabilities. Yet contemporary architecture often assumes a stereotypical, Neufertian identity of the body. Disabilities studies expose the discriminatory reality of architecture, in which conventionally designed environments are conducive for the development of habits in those who “exhibit normative embodiment” but hinders habit development in those who do not, i.e. people with disabilities. (Engman and Cranford, 2016, p. 28). Conclusions from these studies not only demonstrate the influence the material environment has over the development and enactment of our actions, but also highlights the lack of diversity anticipated by the built landscape. This cookie-cutter approach toward architecture leads to narrowed spatial experiences.
As a humorous critique against the standardisation imposed on our bodies, Thomas Carpentier proposes an add-on to Neufert’s data reference book that caters towards different types of bodies: real, fictitious, grotesque, amputated, peculiar, odd, oversized. For example, he depicts an elevated bedroom design for a legless dancer to create a uniform level where the bodies of his wife and him can “interact and mingle without difference” (Carpentier, 2017, p. 7). As affordance theory prioritises the body in the conception of an environment, adopting an affordance-based approach in architecture will open up a host of action possibilities that celebrates anomalies and accommodates a diversity of bodies.
Fig. 3 Subverting the standardised body in The Measure(s) of Man: Architects’ Data Add-on by Thomas Carpentier (2011)
Fig. 4 The Measure(s) of Man: Architects’ Data Add-on by Thomas Carpentier (2011)
The Fallacy of Habits
3.1 Habits as Building Blocks
Human beings are creatures of habit. Habits operate as an optimisation tool to ensure that the mind does not go into cognitive overdrive as we go about our daily tasks. Habits are defined as “self-valorising repetitive behaviours that can be performed with minimal conscious effort on the part of the subject” (Engman and Cranford, 2016, p. 28). This definition highlights two essential aspects of habits.
First, habits are repetitive actions that instinctively react to situations encountered regularly. Jean Piaget (1952) introduced the concept of schema in his cognitive development theory, which he defines as: “a cohesive, repeatable action sequence possessing component actions that are tightly interconnected and governed by a core meaning.” (p. 240) It is a cognitive framework that organises and interprets information; a system which pigeonholes experiences into mental categories. Therefore, it allows for mental shortcuts in which we tap on past experiences to discern the appropriate action to take in situations that we often encounter. Second, habits are actions we enact without active awareness. Our habituated actions are performed at the periphery of consciousness to free up cognitive processing power that can then be directed towards new and more complex problems (Joas, 1996). As our actions become habituated, “the brain stops fully participating in decision making” (Duhigg, 2012, p. 20). As a result, the body automatically falls back on its muscle memory and tend towards enacting familiar, repeatable actions. Habits form the building blocks for the development of human cognition; however, we are also losing our bodies to the automatic.
3.2 The Filter of Habits
Although habits relieve us of a cognitive burden, they also desensitise us. Michael Pollan (2018, p. 15) laments that habits are the “muscles of attention atrophy”. He writes, “The good thing is I’m seldom surprised. The bad thing is I’m seldom surprised.”
The act of ‘seeing’ should be distinguished from ‘recognising’. Seeing is the act of perceiving something for the first time; recognising, on the other hand, is the act of identifying something that we have already established. Victor Schlovsky (1991, p. 9) writes, “A phenomenon, perceived many times, and no longer perceivable, or rather, the method of such dimmed perception is what I call ‘recognition’ as opposed to ‘seeing’.” Our enactment of habit falls under the category of ‘recognising’. Through repeated reliance on our habits, we have become conditioned and constricted in our actions – fixed in our ways. Infants, on the other hand, continuously exhibit the phenomenon of seeing anew, or neophilia – the love of novelty, the new and the unfamiliar. We, with our dulled and conditioned minds, should learn from infants their delight, surprise and wonder that arises from their virginal seeing.
Our proclivity towards our habits makes us unable to access the full potential afforded to us by the environment. The material environment suggests to us how we can act on them. The word can, instead of must or should, is used because objects are in fact open-ended in their action possibilities offered to the human animal. An object presents multiple ways of acting upon it, but we often choose to act in a singular, repeated manner. Therefore, the actual affordance of an object is distinguished from its potential affordance (Rietveld and Kiverstein, 2014). Factors that limit the actual actions produced by the human animal include the lack of appropriate skills by the animal to act accordingly and socio-cultural norms that dictate ‘proper’ or ‘acceptable’ actions (Rietveld and Kiverstein, 2014). Costall (2015, p. 51) terms the “single, definitive” use of an object its “canonical affordance”. Frequently resorting to our habituated actions also contributes to the object’s canonical affordance, eliminating all other options offered by that object. Our habits act as a filter, sifting out alternative actions and thus reducing the potential for a variety of interactions; this results in what Arakawa and Gins (2002) describe as an “insufficiently procedural” environment, which demands little from the body.
3.3 In situ habits
The enactment of habitual actions is grounded in our innate sensorimotor faculties, and thus predominantly embodied, revealing the material environment’s complicity in the triggering of our habits. As Dewsbury (2012, p. 74) writes, “Material dispositions are..
Acknowledging and reflecting upon how the 21st century discourses that shape the notion of body extension, this design thesis presents a quantitative documented work in which a woman experiments with the possibility of amplifying her autobiographical memories by interposing her daily body movements and postures with external olfactory stimuli.
The main question of ‘How she exposes herself  to amplify her memories through a specific sensory modality,’ kickstarted a series of design explorations and experimentations on how design is used as a tool to heighten bodily self-consciousness (BSC)  with the environment. Primarily, the cross sensory relationship between body movements, postures  and environmental context using bio-sensing has been explored followed by the statement raised that “Everyone perceives differently – experiences and memories are something very personalised.” This thesis imbues scientific strategies with personal documented narratives and emotional expressions.
The meaning of our body movements and postures are stored in our body unconsciously  in relation to our memories (learning history) and past experiences. Feedback loops from body movement and posture in the environment to herself has been explored (eg. the rapid adaptations of herself in the environment on memory for life events) followed by the question: “Which sensory modality is the most powerful one to achieve the goal?” and “How to create new associations between body movement, posture and a specific sensory modality through bio-sensing?” The study of the role of sensory extension can modify one’s perception and possibilities of transferring new learning into long term memory has been discussed in this thesis.
Sliding back and forth with the aim of making the data collected more deep-rooted and the unknowable quantifiable, this design thesis operates at the tension between her emotional and rational self. By investigating her body as a site, this design thesis raises questions on how she can create an immersive experience for herself by employing multiple sensory inputs and a wider range of feedback. A series of small design projects are created alongside this thesis writing and has also provided a series of speculative platform on future wearable and body extensions.
It is hypothesised that through the effect of Neoteny as a new body extension, Neoteny is able to amplify her memories by exposing herself to external olfactory stimuli with the built environment as a result. By creating a work that is simultaneously technological, functional and symbolically potent, it seeks to expand our notions of what is possible.
 Refers to a user’s body senses.
 Refers to awareness of own body and self understanding (Freud, 1915)
 Refers nonverbal communication which are used to convey information.
 All our past experiences and memories sit in the unconscious mind (Freud, 1915).
I have been systematically logging and documenting my daily life with and without Neoteny for the past 3 months. Through this period of time I have amplified my memories by expanding my capacity of remembering, which has been increased through the daily exercise with the device Neoteny. I have been documenting my daily experiences at the end of each day and allowing my data to be recorded and analysed during the experiment. The main idea of doing this is to compare my memories and experiences of each day with the written results of my mental images with my physiological data obtained through Electromyography (EMG) signals. Upon completion, I study the relationship between the two avenue of descriptions and representations. Neoteny aims to compare what my documented emotional expressions and narratives reveal, with what the machine reveals through its output. These two representations will eventually complete the bridge with one another after the analysis.
(Hui Sim Chan, 2018)
We move and sense through our body and that reflects who we are.
Eventually, our body is a collection of our past memories and experiences. 
Knowing that our body has long been a site of our life history, where our past memories and experiences were formed additionally through the interactions and connections of our five senses with environmental stimuli. If we were meant to start having experiences and memories since we were born, how would they shape us into who we are today? How much awareness do we still have or own to ourselves?
Human beings are known for taking their given five sense for granted. It is well into the 21st century while writing this design thesis, where human-technology relationship has been explored. The human of this century has not solely been exposed to body-related technologies . The human-machine interactions (HMI)  have very much raised the possibility of having an external device as part of the extended cognition  (Clark and Chalmers, 1998).
Our body has become a vessel where systems and ideas are explored and manifested, it’s no longer just a body. 
Body extensions or prosthesis  with the integration of high-tech functions has been used as a means to amplify the senses to our body by enhancing a user’s physical ability or senses by accommodating the way humans already perform. This thesis will explore the use of human body as a site for future technologies by studying the existing use of prosthesis and its relation to the human perception whilst questioning the future possibility of introducing new body extensions in respect to the human needs (Antonelli, 2008).
So is it possible to design a body extension that are able to amplify human memories and experiences by challenging our natural five human senses? How do we design and recreate a new body extension that is able to provide a translucent layer of information over to the human senses while at the same time not interrupting the user’s daily activities? To what extent can a user have a subtle communication between the body and the environment based on this layer of information? It is speculated the human senses and reality will be challenged by the next generation of body extension or prosthesis.
Through the following pages, I:
1. present the discussion and arguments on the future of body extension and the technological context that motivated the creation of this work;
2. tracing the theoretical and philosophical frameworks to which this design work contributes;
3. explaining the design project technologically, and
4. describing the aesthetic decisions involved, together with the public acceptance (social aspects) in communicating this project as a new body extension.
Philosophically, I will be tracing technologies of bodily control as a main mechanism for this design thesis, referencing Foucault’s notion of bodies  whilst at the same time referring to Cyborg Manifesto by Donna Haraway, to envision our body constantly being extended by devices and technology by positioning our body as an evolving entity (Haraway, n.d.). In the very famous read, The Extended Mind by Andy Clark and David Chalmers, they claimed that body extension are part of our extended mind (Clark and Chalmers, 1998).
In the second chapter of said book, the possibility of searching for autobiographical memory retrieval has been experimented and documented through a series of visual cues and a brief study of the relationship between human perception and body senses, with the effect of an external attachment during cognition processes (olfactory stimuli) followed by the exploration of the quantified data and digital information (EMG signals) recorded.
Technologically speaking, this project implements a human machine interface (HMI), where the electromyography signals (EMG) detected through muscle tensions are then translated into a micro controller to control the release of smells. This will be further discussed in the fourth chapter, alongside the quantified explorations and experiments that have been recorded while I am documenting my daily activities with the designed wearable, including how my experiences changes each day. This has demonstrated some of the emerging possibilities and consequences of the impact of Neoteny on me in relation to the environment, which leads to a speculative investigation of the future of the human body with body extensions. The primary objective of Neoteny is to compare personal perception  of smells related to my autobiographical memory subjectively with a quantitative analysis through Electromyography (EMG) signals in attempt to identify a bridge between these two dimensions.
The observations and experiences from these interactive set of work helps building up a better understanding and guideline managing the design logic of series of Neoteny. Neoteny has demonstrated the potential changes in the relationship between user and the environment in relations to such technology and has been used to speculate a future where attaching such devices become common. The design details will not be shown in this paper as it is still ongoing at the time when this thesis paper was submitted.
 Our body remembers who we are and how we are located in the environment. Our five senses help us to fuse our perception of self with the experience of the environment (Pallasmaa, 2014).
 Refers to digital information that has been stored, produced or projected through body-related technologies (eg: smart watch).
 Refers to design of technology, in particular, the interactions between human (users) and the machine.
 Cognition process can be extended by having an external attachment to human mind and body (Clark and Chalmers, 1998).
 In this context, our body is referred as a site of memories, experiences and the environment.
 Refers to attachment to the body.
 Foucault’s later works mentioned that in order for norms to be effective, they must first be experienced and embodied, whether this is a self-taught behaviour or discipline enforced upon oneself (Foucault, 1984).
2.0 Where does the mind stop & the rest of the world begin?
(Clark and Chalmers, 1998)
2.1: The current attachment to the body, body extension
Both temporary or permanent attachment or addition to the body are defined as a body extension . As much as we humans have been taking our five senses for granted, we are still born with insufficient capabilities, Le Corbusier argued (Corbusier, 1987) . For example, the fact that we are not able to sense or experience distant environment  from ourselves has made our body as a site of extended body related experimentations and explorations by scientists and artists. Body extensions or attachment has been proven to be able to expose the human body to more opportunities on the usage of it.
Artist Rebecca Horn demonstrated a new way of sensing the environment in her piece Finger Gloves[Figure 2.1]. During her performance, she uses her enormously long finger extensions to scratch both sides of the room at the same time. She challenged the limitation of original human body functionality and augment her body movements through extensions to re-experience the room. Finger Gloves is the extension of her body – it extended her abilities and allowed her to experience what she had never experience before.
On January 2018 in Japan, Kaiji Moriyama  collaborated with Yamaha Corporation in an Artificial intelligence performance. Yamaha has provided an Artificial Intelligence (AI) system which enabled him to perform a musical piece through his body movements [Figure 2.2]. He has fully demonstrated the body-technology relationship by performing his personalised form of expression that fuses his body movements, posture and music (musical instrument: piano). Muscle tensions produced by his body during his dance performance has turned into a medium for him to extend his artistic expression and the technology adopted in the system becomes the focus during the performance. Signals generated by the different type of sensors attached to his back are able to identify his movement in real time through AI system. As he moves around, the music produced modifies according to the movement and muscle tension of his back. Muscle groups on his back were introduced as his medium for applying this intelligent layer of data of his body.
Our muscle tension often arises from unconscious muscle contraction, which makes it interesting to usher designers and artists in a new way of bio sensing (like Neoteny piece). Muscle tensions are able to introduce a new way of understanding the physiological data of our body through the way we move and position ourselves in the environment. Muscle groups in different parts of our body help us to sense and respond to the environment.
These attachments or body extensions made the user behave and move in weird and strange ways, but at the same time experiencing a heightened awareness of their bodies and the environment.
Figure 2.1 Captured photograph of the artist Rebecca Horn wearing her enormously long finger extensions – Finger Gloves, 1974.
Figure 2.2 Captured photograph of the dancer Kaiji Moriyama performing dance expressions in Japan, 2018.
 Body or sensory extension occurs when we extend the reach of our embodied mind beyond our own limitations (McLuhan, 1964).
 Humans do not have the natural ability to fight natural predators, withstand harsh weathers, and hunt or fight for food (Corbusier, 1987).
 Refers to built environment, the tangible or intangible such as physical objects, experiences and etc.
 “Just like the railroad opened up a new world, a new object can open up an uncharted territory. Let people inspire and mesmerise. Or like the telephone and television are extensions of ear, voice or touch (McLuhan, 1964).
 Refers to a Japanese dancer.
2.2: The Quantified Self
When I was in primary school, my teachers in school told me to write diaries at least once a week.
(Hui Sim Chan, 2018)
I am brought up in a family where everyone write diaries or journals daily, or at least once a week. This habit is embodied . In today’s world, we come into contact with a range of quick evolving technologies and digital devices. Humans have slowly shifted from writing on paper to digital. We have learnt to adapt to new tools, to enable ourselves to do things and achieve complex goals with more effective and efficient ways. With that in mind, what about putting the notion of body extension into a technological context in this digital age?
It is well into the 21st century at the writing of this thesis, the amount of affordable consumer digital devices on the market has slowly pushes human to start using them. We have all learnt to make use of them as part of ourselves as our extension and attachment to our body. These digital devices are able to expand our incapabilities and allowed us to work upon ourselves (Lupton, 2013), leading us to walk into another new world with a better self. However, Merleau-Ponty argues that any external artefacts can become an extension of a live body to such a degree that the user has a heightened awareness of its existence as it was part of the body (Merleau-Ponty, 1945). Humans have learnt how to quantify self-movement with digital devices such as smart watches , and smart phones.
Digital devices allow wearers to keep track of inaccessible visible information such as body physiological data or even as part of our external memories to store photos, contact numbers and other relevant information . Wearable devices have been changing the way we move and the way we think about our lifestyle; for example, smart watches are slowly changing the way I exercise. The data stored are information that we may not necessarily need but retrievable at any point of time. As Andy Clark and David Chalmers commented, body extensions are part of our extended mind, and have become part of our identities (Clark and Chalmers, 1998). Technology is our new body part and data is our new perception.
Pekko Vehvilainen has been quantifying himself with 11 different digital devices and platforms in the past, see [Figure 2.3]. He used them to record his physiological data such as body fats, heart rate, sleep patterns and his daily physical activities . Digital devices or wearables help give us awareness and insights into the invisible aspects of our life by heightening our bodily self consciousness (BSC). As Lupton commented, all of us want to be a better self each day and these technological devices are tools to mediate our perception of our body and the environment (Lupton, 2013).
Figure 2.3: The most quantified man in Finland, Pokko Vehvilainen, 2014.
2.2.1: Communication between Body Movements, Postures and the Environment
It’s interesting enough to have a look at these body extensions and digital devices discussed in previous chapters; how they create communication with our body. Haptic responses are the most widely used tactile feedback on digital devices (eg: vibrations). We have been exposed to haptic responses on daily life without realising it. For example, peeping into the closest portable digital devices around us like mobile phones, they are all designed with haptic responses. We get different vibration patterns and rhythms when we receive different notifications on our smart phones . It is usually activated when our phone is on silent, or when our sights are away from our phone, strange enough that we are able to process the information. According to Wigley’s statement in 2010, devices are able to affect our perceptions before, during and after the usage (Wigley, 2010).
I went on an exploration in a group by quantifying my own heart rate walking past places in London, my heart rate changes when the environment changes, I wonder why? Is that the only data I could’ve gotten from my body? Or is my body trying to convey messages to me in a different way through different physiological data?
(ref. Sim’s Diary, Neoteny, Day 06)
I have had my heart rate recorded at every different spot on the streets and in different environments. Tracing back to a video recorded while I was walking, I realised that my speed of walking and the way I oriented my body differed when the environment changes. I walked faster passing through alleys, I don’t stand comfortably next to a very busy street, I can’t walk slower passing through crowds and I tend to walk slower in a place with less cars and more open spaces – these have all got to do with my body movement and posture.
This paper aims at exploring the potential of movement-based interactive experiences in XR to train mindfulness, and more specifically four of its main facets: Awareness towards inner experiences, Awareness towards outer experiences, Relativity of thoughts and beliefs, and Openness to experience. It proposes to focus on balance as a central node between sensations of the body, perception of the world, emotions, and their respective stabilities.
The research question is explored through a literature reviewing the current practices of mindfulness, the role of balance in sensations of the body and perception of the world, possible techniques in bringing one to attend to the sensations of the body and perception of the world through balance, and the notion of relativity of thoughts and beliefs in balance. This review is then used to guide the design of five prototypes of (Un)Balance, a movement-based interactive mindfulness experience in XR where participants are invited to explore the limits of balance through movement. Through a thematic analyses of the response of participants, four main themes emerge which are an openness to experience facilitated by VR and play, the interaction of different elements of XR experiences in allowing a flow of attention, the importance of balancing different sensory inputs in allowing general awareness, and the role of unpredictable patterns of response in facilitating sustained attention.
Those findings are finally applied in a proposal of design development of the experience (Un)Balance, its physical and virtual layers, as well as its experience protocol.
The last 10 years have seen a rise of general interest in mindfulness practices, such as body scan meditation, yoga or Tai Chi. Also used in clinical contexts, here i am mainly interested in its use in everyday life by the general public of the western world, and its acceptance as a mean to reduce stress and improve general mental and physical well-being.
The increasing need for such practice could be attributed to the extensive exposure to digital media technology which many argues has a “damaging effect on our sustained and selective attention” (Gausby, 2015; King, 2016). By multiplying the amount of stimuli our world provides, it would divide of our attention, resulting in heightened stress levels and poor mental and physical well-being.
Nonetheless the practice of mindfulness techniques can be challenging for the novices, often leading to discouragement and early abandon of the practice.
This paper aims at exploring the potential of movement-based interactive experiences in XR to train mindfulness in the beginner and and more specifically four of its main facets: Awareness towards inner experiences, Awareness towards outer experiences, Relativity of thoughts and beliefs, and Openness to experience. It proposes to focus on balance as a central node between sensations of the body, perception of the world, emotions, and their respective stabilities.
Extended Reality (XR) refers to “all real-and-virtual combined environments and human-machine interactions generated by computer technology and wearables” (North of 41, 2018).
In this thesis I firstly review the literature on mindfulness, movement, balance and perception so as to understand the potential of movement-based interactive experiences in XR in training mindfulness. I then detail the design of a series of experiences prototypes which was guided by those initial understandings. These experiences prototypes are shared in two exhibition settings. I note that the experiences prototypes aim at testing the concept and not the specific usability nor technological aspects of the design. I then use a thematic analyses of my observations as well as the gathered feedback from participants to expose my findings on the proposed research question. Those findings are finally applied in a proposal of design development of the experience (Un)Balance, its physical and virtual layers, as well as its experience protocol.
B: Literature review
In this section I review the literature on mindfulness, the role of balance in sensations of the body and perception of the world, possible techniques in bringing one to attend to the sensations of the body and perception of the world through balance, and the notion of relativity of thoughts and beliefs in balance. I do so so as to guide the design of a series of experience prototypes of (Un)Balance.
Definition and practices
Mindfulness has been defined as “the awareness that emerges through paying attention on purpose, in the present moment, and non-judgmentally to the unfolding of experience moment by moment” (Kabat-Zinn, 2003). “Mindfulness techniques commonly involve developing awareness and acceptance of constantly changing experiential phenomena. These phenomena may include cognitions, emotions, bodily sensations, and external stimuli” (Baer 2003).
While being aware that contemporary psychology tends to describe mindfulness inconsistently (Bergomi et al. 2013), it is on those definitions that I will focus for the purpose of this paper.
Mindfulness is rooted in ancient origins, notably in Buddhism (Allen et al. 2006; Baer 2003). Nevertheless, mindfulness techniques can also be practiced without the involvement of any tradition (Hayes and Shenk 2004; Kabat-Zinn 1990).
It can be trained through different practices or techniques such as guided meditation, yoga, praying, walking or Tai Chi.These aim at training mindfulness through encouraging episodes of heightened mindfulness (Salmon et al. 2009). It is believed that practicing mindfulness techniques frequently “lead to an overall diffusion of mindfulness throughout everyday life” (Garland et al. 2015). In the recent years, new forms of mindfulness training have been developed namely through the use of VR, and digital games.
To understand mindfulness, and how it is done, multitude attempts have been done to depict its facets (Park et al. 2013), with fluctuating differences (Bergomi et al. 2013). The Comprehensive Inventory of Mindfulness Experiences (CHIME) (Bergomi et al. 2014), has recently been created in an attempt to merge those conceptual views and provide a comprehensive measure of mindfulness as well as a clear depiction of its facets.
The 8 facets are described as follow: 1- Awareness Towards Inner Experiences: “Ability to be aware of one’s own experiences such as thoughts, feelings and sensations” (Bergomi et al. 2013, 2014); 2- Awareness Towards Outer Experiences: “Clear perception and experience of external stimuli” (Bergomi et al. 2013, 2014);
3- Openness to Experience: “Non-avoidant (confrontational) attitude towards experiences that is characterised by openness and curiosity” (Bergomi et al. 2013, 2014); 4- Decentering: “Ability to experience non-reactively”(Bergomi et al. 2013, 2014); 5- Acceptance: “Encounter of experiences with an accepting attitude and without judgement, e.g., good or bad” (Bergomi et al. 2013, 2014); 6- Relativity of Thoughts and Beliefs. “The recognition that thoughts and beliefs do not possess universal truth but are completely subjective and might not always correspond to reality” (Bergomi et al. 2013, 2014); 7- Insightful Understanding: “Understanding that the quality of an experience is influenced by its subjective evaluation” (Bergomi et al. 2013, 2014); 8- Acting with Awareness. “Staying in the moment, being fully conscious of the here and now” (Bergomi et al. 2013, 2014).
All mindfulness techniques shall encompass the 8 facets listed above.
Today mindfulness are being used throughout the eastern world, but also increasingly in the western world.
Looking at general populations, there is clear rise of general interest in including moments of mindfulness in one’s everyday life. We can observe this through the multiplied use of guided meditation phone applications such as Headspace (2014) or the rise of the practice of yoga or Tai Chi. It is generally used in the aim of managing stress or improving general mental and physical well-being.
On the other hand, mindfulness is also becoming more and more accepted in western medicine as a therapeutic measure (Ekman et al. 2005), to help patients with specific psychological or physical disorders. Is is often used as a complement to classical medical or psychological approaches. Examples of those conditions are depression (Cash and Whittingham, 2010), anxiety disorders (Cash and Whittingham, 2010), chronic pain (Kabat-Zinn et al. 1985), or multiple sclerosis (Mills and Allen, 2000) or substance-abuse addiction (Bowen et al, 2009).
The most popular practices wether used by general public or in clinical contexts are body scan meditation, yoga and Tai Chi. In body scan meditation, while the participant is static, usually sitted “ the attention is directed towards different areas of the body consecutively with an emphasis on noticing and experiencing any sensations, pain or muscle tension in each area” (Mirams et al, 2013). In yoga, the participant “accomplishes sequences of postures, that incorporate regulated breathing and focused attention on proprioceptive and interceptive sensations” (Salmon et al, 2009). Tai Chi is “a traditional Chinese mind–body practice that involves slow movements (i.e., movement of the body trunk and upper and lower limbs so as to form various postures systematically following one another), deep breathing, and visualisations”.(Lee et al, 2017)
It worth noting that several studies note that movement-based meditation such as yoga or Tai Chi correlate with more changes in measures of mindfulness than the practice of sitting meditation (Caldwell et al. 2010).
Limitations of current practices
One of the major limitations of current mindfulness practices is the difficulty for beginners to access an initial entry point in mindfulness practice, such as the difficulty to 6
attend to inner and/or outer experiences.
When it comes to awareness of outer experiences a few VR guided meditation experiences such as ProvataVR (2016) or GuidedMeditationVR (2017) provide a VR meditation backdrop depicting relaxing scenes, most often a nature landscapes, in the aim to help with bringing a participant into a present moment through raising awareness of outer experiences. Research does provide evidence that guided meditation experience has a therapeutic potential in the treatment of anxiety and stress disorders (Tarrant et al, 2018).
Nevertheless, while this is true about awareness of outer experiences, the use of VR is more questioned when it comes to awareness of inner experiences. For example, VR-based experiences have long been used as distraction in pain management therapy (Hoffman et al, 2000), as VR tends to “draw the patient attention into a computer-generated world, leaving less attention available to process incoming pain signals”(Hoffman et al, 2011).
A few experiences address this issue through mixing VR with body responsive technologies. We see for example Juno (2017), which integrates breath responsive technology, or PsychicVR (2017) which uses a brain-computer interface. These experiences seem more likely to provide mindfulness training as they both encourage awareness of inner and outer experiences.
According to my knowledge, the issue of difficulty for beginners hasn’t yet been addressed in the context movement-based mindfulness practices.
While this project won’t address it, it is worth noting that another limitation of current mindfulness practices, again mainly for novice practitioners, is the difficulty of self-evaluation. This has been addressed in MeditAid, “a wearable system integrating electroencephalography technology to identify different meditative states and provide feedback to support users in deepening their meditation” (Sas and Chopra, 2015).
Mindfulness and (Un)Balance
Being aware of the current works in the design field to provide physical and/or virtual experience easing novices in the practice of mindfulness, I see an opportunity to explore the possibilities of movement-based interactive experiences in XR as a mean to provide more intuitive ways to introduce mindfulness practice, and more specifically movement-based mindfulness practice to novices.
Reflecting on the facets of mindfulness as described by Bergomi, I see a particular potential for movement-based interactive experiences in XR to help train the following: 1- Awareness Towards Inner Experiences; 2- Awareness Towards Outer Experiences; 3- Openness to Experience; 6- Relativity of Thoughts and Beliefs. This is what i will address in this paper as well as in the design of (Un)Balance.
B2: Role of balance in sensations of the body and perception of the world
“We are encased by flesh in a physical being that defines the limits of our ability to act upon the world, and provides the medium by which the world acts upon us” (Haugeland, 1998).
Bodily sensations greatly contribute to our minds being embodied and embedded in the world. It is ‘how we experience our bodies and how our bodies experience the world’ (Eccleston, 2015).
As embodied cognition emphasises, our bodies inform how we think and perceive the world. We actively experience the bodily senses, which receive stimuli from the environment. We construct reality through body action in and on the world. Shusterman puts it as “our bodies are our indispensable tool of tools, the necessary medium of our being, perception, action and self-presentation” (2006).
Embodied cognition, a movement in cognitive science sees “cognition as situated: of movement that is always of and for action” (Eccleston, 2015). Here “the mind is a product of physical being in interaction with the world” (Gover, 1996). Maxine Sheets-Johnstone adds the movement is essential in making the world perceivable and accessible for young children (2011).
Shusterman on the other hand points out that “to improve our body awareness, we need to move and furthermore move in ways that shifts us out of our habitual movements and response patterns” (2006).
In the context of mindfulness training, as seen in the practice of yoga and Tai Chi, the body in movement could therefor be seen as a tool to channel attention both
towards the body (inner experience) as well as the world surrounding it (outer experience).
A central element in the practice of both yoga and Tai Chi is also balance. “Yoga entails moving the body mindfully through a series of poses, as well as balancing the body while in a single pose” (Shiffmann, 1996). In the context of (Un)Balance we will explore specifically how the action of balancing, which is an essential component of any movement, can train awareness of the body and its stability, as well as the world and its stability.
Balance is defined by the Oxford dictionary as “An even distribution of weight enabling someone or something to remain upright and steady”. It is the action of “maintaining postural equilibrium _ centring and maintaining the body’s mass in relation to gravity” (Eccleston, 2015). Through balance, we control our bodily positions while moving, whether it be be walking, running, or dancing. It is a continuous and automatic activity which accompanies every movement of the body. It allows us “to take a stand in relation to the world” (Lopez et al. 2008).
The perception of balance is achieved by processing sensory input from the visual, vestibular and proprioceptive and haptic systems and their cortical integration. Proprioception (also referred to as sense of position) is, as first described by Sherrington, ”the ability to feel the position of the body in space and limbs of the body in relation to the rest of the body” (Sherrington, 1906).
Balance, in a psychological sense, “refers to more than just equilibrium: balance is the stable gravitational position in relation to the world, an egocentric sense of position within the world (where i am in relation to other objects), and a sense of agency (control over that position).” (Eccleston, 2016) It is also worth noticing that the vestibular system involved in the sense of balance, through interacting with other sensory systems, seems to also assist to body schema in general (Lopez et al. 2012) and ‘specifically to one’s sense of ownership over all or part of one’s body’ (Merritt and Tharp, 2013).
The sense of balance therefor relies on informations regarding the body as well as the world, and their respective stabilities, and is intrinsically linked to our perception of both. Through attending to the sense of balance, we would both attend to inner and outer experiences. It therefor appears relevant to explore the design of movement-based interactive XR experiences involving the sense of balance.
Balance and openness to experience through play
It is the curiosity of the toddler and the urge of exploring the surrounding environment that lead him to learn how to stand and therefor master his/her sense of balance.
“Babies explore_ they move and act in highly variable and playful ways that are not goal-oriented and are seemingly random. In doing so, they discover new problems and new solutions. This exploration makes intelligence open-ended and inventive.” (Eccleston, 2016) This attitude is also called play.
I see a parallel between play and the facet of mindfulness called Openness to Experience which is characterised by a “Non-avoidant (confrontational) attitude towards experiences that is characterised by openness and curiosity” (Bergomi et al. 2013, 2014). While movement-based mindfulness practices such as yoga or Tai Chi train participants in executing precise movements, which complexity can prevent the beginner to embrace an openness of experience, I see potential in non-goal orientated games and experiences to train this facet of mindfulness in the beginner while retaining the movement aspect of these practices.
I also note that in virtual environment design, the term immersion is defined as “a feeling of being present within a virtual world”. (Pierce et al, 2017) I therefor see potential in a mixed-reality experience to encourage the participant be in the present, with a curious and playful attitude.
It therefor appears relevant to explore non-goal oriented experiences in XR to encourage openness to experience.
B3: Attending to the sensations of the body and perception of the world through balance
While the sense of balance plays a key role in our perception of the body and the world, it operates automatically. This means that it does not require constant attending, but momentary attention does happen (Eccleston, 2016). In this section I will explore how extend those moments of attention.
How could (Un)Balance bring participants to attend to the sensation of stability of the body and perception of the stability of the world?
Firstly it is necessary to explain the basics of the process of attention. As stated above, when we experience the world, we construct reality through body action in and on the world. We often are under the impression that we grasp this reality in full detail but this is an illusion. The brain has evolved mechanisms to only select the informations that are the most important. One of those mechanisms is attention. One can think of attention as a form of mental currency that the brain has to choose very carefully has to spend. Nobody has an unlimited amount of attention. Our brain has enough power to attend on one element at a time. (Nikos Konstantinou, 2015).
Channel attention through bringing the sense of balance to its limits
Being embodied, also means being limited. While the sense of balance operates automatically, when we live experiences pushing balance to its limits we attend to it. Experiences could be negative, such as falling or filling fizzy, or experiences associated with activities requiring a high level of balance control, such as surfing or ballet dancing. (Eccleston, 2016)
Therefor one way to bring attention towards the sense of balance and the sensations in the body and perception of the world it involves, could be to create experiences which would push the sense of balance to its limits. In the context of (Un)Balance, this could be a physical device requiring a higher level of balance control, such as an unstable floor.
This is a technique which already used in the practice of yoga, “where a large the skill lies in sensing just how far to move into a pose. If one does not move far enough, there 8
is no challenge for the muscles; however, going too far may result in pain or injury.” Therefore, the practice of yoga involves learning to listen to the sensations of the body while willingly pushing it to its limits (Shiffman, 1996). Another way to push the participant’s sense of balance to its limits would to encourage the participant to explore the limits of balance willingly, through play for example.
Attending to the sensations of the body through amplifying the sense of balance
Channel attention through augmenting bodily sensations in balance
Another approach to channel attention towards a specific sensation is to amplify it.
Augmenting a sense can be done through amplifying a sensory input, such the apparel worn by Rebecca Horn in her performance “Unicorn”. The effect on the wearer is described as “extended up into space and down through the head, the mythical hybrid feels the pull of gravity and must concentrate on balance, pace and head position” (Horn, 1970). Here the apparel amplifies the senses of balance and gravity through using the very sensory systems involved in these senses naturally, such as the proprioceptive system. It does so in a linear way.
Augmenting a sense can also be done trough using a different sensory input than the one(s) usually normally involved. It can also be referred as feedback. For example a sound feedback can be used to augment proprioception (Kurihara et al, 2012).
In the context of balance, bodily sensations involved are the sensation of weight shifting as well as muscle tensions. The heaviest parts of the body being the pelvis and the head, a lot of those muscle tensions happen in the torso, linking the two. Could part of the experience of (Un)Balance consist in an apparel (physical and/or virtual) augmenting the sensations of weight shifting and muscle tensing in the torso? Should it do so using the proprioceptive system only, or should it also use visual, sound and/or haptic sensory inputs?
Attending and Predictive Processing (PP)
Predictive Processing has a specific characterisation of attention, which it sees as the mechanism for optimising precision (Velasco, 2017).
Its central idea is that the brain uses observations to make hypothesis about the world in the form of predictions. These predictions go top-down (from higher-order areas down to sensory sheets) and get updated by the error from perception bottom-up. The aim of the PP organism is to reduce error over time (by making better predictions, by learning from the world and by transforming the world). In order to do this, it tries to reduce uncertainty. The measure it has of uncertainty is the variance of the error of its own predictions. The inverse of variance is precision, and the PP organism tries to maximise precision over time in order to attain its overall goal of reducing error over time.
According to PP, attention is the mechanism for optimising precision. If a new unexpected sensory input appears (generating a lot of error) more resources will be directed to tackle that source of error and explain the error away with better predictions, therefor channelling the attention towards that sensory input (Velasco, 2017).
In the context of (Un)Balance, and a previously discussed component augmenting the sensations of weight shifting and muscle tensing through additional sensory input, if the latter has an identifiable pattern in the way it responds to the participant’s movement, the participant’s PP organism will quickly find out the pattern behind the source of uncertainty, make better predictions, and..
This thesis illustrates how moulding the space with our bodies is a constant construction of experiences that forms and re-forms who we are. The use of a choreographic object ‘Locus’ enables the bodies within and around it to inhabit and manipulate the experience of the participants. These intentional and non-intentional manipulations of the space are able to create an affective experience and is what many geographers define as the power of built spaces. The architectural space is constantly giving us signals of what to do, where to go and unconsciously moulding our feelings, behaviours and actions. This research involves the study of extensive and intensive space. It demonstrate through series of experiments with a choreographic object, a future of possibilities of architectural and performative spaces that might enable the communication between multiple human bodies and the build environment.
“A body exists in space…moves in space…is contained by space” (Blom and Chaplin, 1989, p.31).
This thesis considers the possibilities of transforming spaces using the human body as an active tool. The thesis aims to define the relationship between architecture and dance using the body as a medium to modify the space. Our bodies are units that are able to transform the environment, according to Kronenburg (2007), human beings are flexible creatures that manipulate objects and operate in a wide range of environments.
“There was a time, not too long ago in evolutionary terms, when our existence was based in our capacity of movement and adaptability; indeed, it is to this that we owe our survival as species. Most cultures now lead a more or less sedentary life, but it could be that flexibility is once again becoming a priority in human development…” (Kronenburg, 2017, p.10).
These notions have been used as the starting point for an immersive interactive installation project Locus which is being developed by myself and Anthos Venizelos. In the context of this project we analyze human movement in contemporary dance performance, classifying these movements based on the selection of some joints of our bodies(1). Derek McCormack argues that “…dance raises questions with which many other disciplines are concerned, including anthropology, architecture, neuroscience, performance studies and philosophy…” (p.1823). Whilst understanding that the use of choreographic principles is only one way to analyze the body’s actions, this thesis will focus on this as the frame of this dissertation.
Our collaborative project, Locus (Simo & Venizelos, 2018), comprises an open 360 performance arena (Figure 2) involving a perfomer, an audience and an interactive mobile object suspended above the inhabitants head. The design of the object is based on the simple module of an equilateral triangle that is repeated multiple times and creates a flexible structure. The term Locus, here means the center of activity, attention or concentration. The installation has been conceived to encourage the audience participation by allowing them to experience how relevant our movements are in the spaces around. This is achieved by giving both performer and audience an opportunity for active participation in the kinetic structure. It is also speculating about future flexible spaces that proposes a cybernetic model of environment capable of guaranteeing participation for all. (2)
The entire configuration (space and object) is conceptually visualized as “space itself” that merges the performer and the audience behaviours in one single place. As such, it is a reciprocal space that is perceptually and affectively moulded by our own bodies, with not only the performer but also with the audience actively participating in the modulations of the dynamics of the space and giving rise to its affective dimensions.
In Locus, we are developing an integrated platform in Unity 3D(3) using a Kinect V2(4) sensor to detect some of the body joints and within this identify relevant body movements in a simpler way. In order to achieve this, it is important to understand the different qualities of space that are generated by movement so as to render the installation of a dynamic space.
To this end I will analyze what space is in choreography and how our bodies respond to interactivity in dance performance (Figure 3). This thesis report will explain the concept of a space moulded by our bodies, which is, in this context a Forsythian choreographic object (5). It focuses particularly on the interaction between performer and the choreographic object. The aim is to explain some of the scenarios in which the physical design of the mobile object could be transformed depending on the movements enacted by participants and the joints of the body selected. To achieve this, my collaborator and I have translated the information extracted from the Kinect sensor into modifications of the physical space/object using serial communication with Arduino and testing different Servo Motors angles to control the physical instrument for modifying the space.
(1) Focusing on some parts of the body is called isolation and some choreographers as Paul Taylor use this technique to enhance the body as a more expressive tool. In the context of this thesis, this technique will be used to analyze our project.
(2) Archigram proposed an idea of cybernetic architectural space in which was relevant the participation for all.
(3) Unity 3D: is a platform to create 3D games and applications for mobile, desktop. In this case we use it as a graphic interface to link the data detected by a Kinect sensor using serial communication with Arduino and transforming the physical space.
(4) Kinect is a sensor by Microsoft Xbox that enables detecting the human body joints using depth cameras.
(5) Based on William Forsythe’s notion, that the choreographic object opens the question of what else a body can do, a question that calls for experimental movement. How does the object create open situations for movement experimentation? “The choreographic object activates experimentation and play.
Figure 1. The body Author: Simo, A. & Venizelos, A. (2018)
Figure 2. Locus. Interaction between perfomer, audience and space (object). | General aim of the project. Figure 3. Locus. The body moulding the space. | Focus of this dissertation. Author: Simo, A. & Venizelos, A. (2018)
Figure 4. The body transforms the space Author: Simo, A. & Venizelos, A. (2018)
2. The Space
2.1. From the architectural space to the performative interactive space.
It is common to understand the space as something measurable and metric. Space “…does not consist in the sum of the width, length and height of the structural elements which enclose space, but in the void itself, the enclosed space in which man lives and moves”. (Zevi, 1957, p. 22).
Zevi (1957) defines space through actual dimensions but agrees that is also affected by less tangible considerations such as lighting, the position of the shadows, colors, verticality, directions, etc. These features make a space dynamic. In the context of the interactions between the human movement and the spaces, we are focusing in the concepts of dynamic, extensive and intensive spaces.
The space we are trying to define might be flexible, mouldable and revolutionary; it is a space of permanent interaction but within the context of this research, ephemeral and a place of experimentation. Zevi (1957) defines the space as its content: for us this content is full of activity, tangible and intangible; it is part real, part ideal.
As established by Gordon Pask (1958) cited in Architectural Design magazine (2008, p.18) “All buildings are embodiments of a mesh of interlinked concepts or interlinked topics…”. Normally, the concept of space limits the definition to tangible qualities but does not focus on those intangible relations we experience in it. When Pask (1958) refers to the ‘interlinked topics’, he was referring to a cybernetic process of actions that are in constant interaction between each other.
For both the architectural and the performance space the body is the main actor, without the human body these places wouldn’t be ‘alive’. “Non-representational theory tries to attend to the ‘on flow’ of everyday life.” (Thrift, 2007 cited by Simpson P. 2010, p. 1). Nigel Thrift (2007) argues that non- representational theory is interested in the human body co- evolution with things, and how this unparalleled ability of the body to co-evolve with things, means that the body and the environment are not separated terms.
Architectural space is constantly giving us signals of what to do, where to go and unconsciously is moulding our feelings, behaviours and actions. We are usually part of this process but we are not aware of it. The same occurs with the performative interactive spaces, where the performer and audience behaviours formulate changes in the environment.
“Moving slowly across a space might generate a gradually shifting visual or sonic environment. Moving your arms rapidly might generate display of hundreds of short sharp images and sounds, building up a sense of chaos.” (Macdonald & Rubidge, 2001, p. 2)
An active performative environment is a place where the viewer becomes part of, and affects the environment. According to Alistair Macdonald & Sarah Rubidge (2001), performative spaces are designed to incorporate the viewer into the artistic vision; the viewers become those who change the qualities of space. Both the performers and the viewers or audiences should develop a sense of awareness of the way their actions affect the environment so that configurations of the space occur intentionally. (Figure 5)
Figure 5. The body occupying and modifying the space. Diagram extracted from ‘Locus project platform’ in Unity 3D. Author: Simo, A. & Venizelos, A. (2018)
2.2. Dynamic interactive spaces and how the human body can mould them
“Performers actively shaping the space in which they move by creating a dynamic network of interweaving vectors, tensions and transient forms which is perceived by both performers and audiences.” (Rubidge, 2011, p.1)
According to Sarah Rubidge (2011) dynamic spaces are those that are being generated in material space, their qualities modulated perceptually by viewers, rather than materially. In the context of dance performance, these dynamic spaces are generated by the interplay between the dancers’ activities. The dancers are able to experience the space through their bodies through subtle sensation. For example, humans have the capability to know how close they are are to the people around them, especially if we ‘feel’ their proximity as they move closer or further away. These perceptive capabilities are related to our haptic perception and how we understand our three dimensional space.
“The word haptic…is used to describe the various sensibilities of the body to its position in the physical environment and to its own condition. This holistic system of environmental perception goes far beyond visual spatial perception, and refers to a more complex geographical space. It involves the integration of many senses, such as touch, positional awareness, balance, sound, movement, and the memory of previous experiences.” (O’Neil, 2001, p. 3)
J.J. Gibson (1966) argues that people will gain environmental understanding from tangible physical experience, in the architectural context, their contact with the built environment, their movement through the spaces and their visualization of the objects in the spaces. I consider that, if we are able to design a space in which our actions become more tangible,
using our behaviours as inputs, it will be possible to visualize transformations in our built environment. In this sense, we will be shaping the space with our movements and creating a space-in-between as Margaret Morse (year) describes it.
Moulding the space in the context of the installation is therefore not talking about creating a specific shape. Nevertheless, I’m relating this concept to the act of ‘shaping’ and constructing a dynamic network of relations that might not be obvious to the naked eye but might be able to construct the performer and viewer experience in a more tangible way, but more subtle way.
Macdonald & Rubidge argues that “strictly, few genuinely ‘interactive’ spaces are created by interactive artists… Rather they constitute (often extremely complex) reactive systems which are programmed to react to given patterns of behaviours in an installation or performance space.” (Macdonald & Rubidge, 2001, p. 2)
SpiderCrab was one of three experimental projects undertaken by a group of researcher from University of Leeds (Wallis, Mckinney, Popat, Bryden, Hogg, 2008). It is an example of a performance interactive object that is intended to become ‘space itself’(1). At the beginning of the workshops invited dancers were collaborating with a CAD digital drawing of the proposed robot and each dancer was embodying a single limb of the robot. The authors were researching the way a multi-sensory robot could be conceived as an architectural environment but also as a dancing partner. (Figure 6 & 7).
(1) Space itself: a cross between architectural environment and dancing partner.
This thesis is part of a critical design project that envisions a future haptic communication device, NeoTouch, that enables people to touch each other at a distance. The project goes beyond imagining the technical development and the precise functionality of the device to speculate on the social and ethical implications of such a technology.
As Dunne and Raby state in their book Speculative Everything (2013):
“While we are more than ever aware of both the promise and the threat of technological advance, we still lack the intellectual means and the political tools for managing progress.”
Their understanding of critical design is that it has the power – and therefore the responsibility – to question and challenge technological development and the limitations it places on people. It does this in the form of a speculative design proposal, to spark debate, to caution or to show alternatives.
The aim of this project is to use the language of speculative design to create a fictional product on which to explore a future projection of current ethical and social issues. This speculative narrative takes current debates around topics such as consent and agency, privacy, digital safety, and online harassment and investigates how they are affected by technological development and related shifts in social norms.
The debate is rooted in the hypothesis that the way in which technology is increasingly integrated into every aspect of our lives causes our digital and physical selves to blend together and overlap more and more. Applying this idea to a future in which haptic communication has become a part of our daily human-to-human interaction greatly amplifies this concept and the connected societal and personal concerns. It would mean literally mapping the digital self onto the physical body. But what happens when you merge digital and physical privacy in this way?
Who gets to touch me? When? And where? What is the haptic equivalent of digital harassment such as a rape threat? And how would this change our physical interactions and understanding of intimacy?
This thesis makes use of the same speculative language to envision and debate the societal and personal effect of such a technology by asking: How could haptic technology affect our perceptions and conception of privacy and intimacy?
For this purpose, it is written as a piece of fiction in the form of an opinion piece that includes different angles on this topic. As is the aim of speculative design, this story is not trying to be real, but rather to be provocative. It is a tool to open up the debate about different possible futures and to discuss current issues around the topic.
The story is set in a world where haptic technology has become a normal element of human-computer interaction (like accessing digital haptic information of products during online shopping) and chip implants in the brain and other parts of the body are a common way to treat various mental and physical conditions and enhance cognitive abilities. The article is published in 2039, eleven years after NeoTouch first came to the market as the first product that offers a different kind of haptic technology for human-to-human communication based on a brain-computer interface (BCI); an interface that connects a brain – either invasively or non-invasively – to an external computational device (Vallabhaneni, Wang and He, 2005), (Lebedev and Nicolelis, 2006). By using nano chips NeoTouch simulates human touch through micro stimulations directly in the brain.
Image by Christine Wuerth
How digital touch is impacting the way we are intimate.
TECH TOMORROW. Christine Wuerth. Sat 21 SEP 2018 14.00 BST.
When Barbara Wells got her first NeoTouch she was over the moon. As one of the last teenagers in her school to not have the BCI (brain-computer interface) she was feeling left out. It had taken her months of arguing with her tech-critical parents to get their permission. For her sixteenth birthday, she finally got her wish. “I can’t wait to finally experience digital touch.” She said in her online diary at the time. Only two years later she just had the interface permanently deactivated. For Barbara, the negative effects greatly outweighed the benefits.
At first adopting haptic communication actually did exactly what she had hoped for. “Before, I just constantly had to admit to people that I didn’t have it and explain why. And it just made me feel really embarrassed. I mean, everyone else my age has it. People would be so surprised and kind of suspicious. Like I was a weirdo.” With NeoTouch she finally felt part of her group, and more confident in making new friends and approaching boys.
As a researcher, I have been observing the adoption of NeoTouch for years now and the numbers are impressive with 78% of teenagers between the age of 12 and 17 using it.  (The age range in which it is legal to get the BCI set up, but only with parental approval). At this age, the tech is particularly common amongst girls. (Unfortunately, this trend shows that platonic touch is still far more common and accepted amongst girls than boys. )
Figure 01: Percentage of girtls and boys (12-17) using NeoTouch.
Figure 02: Adoption rate of NeoTouch.
This has been a massive trend in the eleven years since the technology first came to the market. But NeoTouch is not only popular amongst teenagers: the adoption rate has been surprising amongst all ages, most of all people in their 20s and 30s. 
Soon, however, Barbara felt the pressure of being ‘always on’, being expected to always be accessible to her friends, and her new boyfriend. “I don’t think our brains are designed to constantly be connected to others” she says. “Even though in a way it really does feel the same as being touched, the other person isn’t actually there. And somehow that contradiction really started to mess with my head.”
Despite the success of NeoTouch there have been growing voices of concern. Especially the last few years have seen an increase in people deactivating their devices and specialist questioning the effects on privacy and physical integrity. But there are always those that oppose new technologies, so is NeoTouch really any different to other kinds of digital communication?
1: Numbers inspired by a collection of studies on current use and ownership of mobile phones by teenagers (CITA, n.d.)
2: Major (1981)
3: Cocozza (2018)
The story behind haptic communication
The arrival of NeoTouch on the market in 2028 redefined our understanding of haptic technology. Originally the word haptics described the use of mechanical pressure, vibration and motion to send impulses through the skin. Early attempts to incorporate touch into everyday technology like touch screens on portable devices were pretty basic by today’s standards. Tactile interfaces developed in the 2010s were mainly used to navigate through information by touching the screen rather than touching anything beyond the screen.  Now, haptic technology has developed to the point that we cannot imagine daily activities like online shopping and watching films without a tactile dimension.
Early development of haptics saw competition from various fields. From gaming and VR to medicine and from the automotive to the sex toy (or teledildonics ) industry.  While VR was making the headlines, it was actually the sex industry that first came close to using haptics as an emotive interpersonal connection . Once the communication industry realised the potential that haptics held for a more emotive form of digital communication they started creating wearable devices incorporating vibration and muscle stimulation to express human touch. Smartphones had used vibrations as an alert mechanism for text messages and calls but they wanted to transform haptic input into a message in its own right.
The first Apple watch released in 2015, for example, was in fact introduced as the company’s ‘most personal device’ ever: “…alerts aren’t just immediate. They’re intimate… We found a way to give technology a more human touch. Literally.” 
If this doesn’t show how much haptic tech has developed in the past few decades, what does!? At the same time developments in neuroscience and nanotechnology made brain-computer interfaces (BCIs) more versatile and precise and the process of embedding them into the brain more routine and much less invasive. First just used for medical devices they were soon commonplace in mainstream products for cognitive and physical enhancement.
Fast forward to 2029 when Somas Technologies (ST)  introduced the first NeoTouch. The first technology to create a tactile sensation directly in the brain rather than on the surface of the skin. By approaching the challenge of haptic communication from this angle they managed to completely revolutionise the industry.
Mike Seymour just published his book ‘A New Intimacy’ which investigates NeoTouch’s rapid success within the wider context of digital communication.
“The height of globalisation, the ‘cult of the individual’  and ever busier lives in the early 21st century meant that we spent less and less time with our loved ones and in face-to-face interactions in general ” he says. “It is no surprise that this coincided with the rise of digital communication and social media. After all, being lonely is literally unhealthy.” 
But these technologies – he claims – could not compensate for the loss of real face-to-face interaction.  “Even though we don’t realise it, a major part of our communication is non-verbal and the nature of digital, audio-visual communication just means that most of that is lost. While we are not consciously aware of it, this still subconsciously diminishes the interaction. It’s just less fulfilling if we don’t receive the same variety and density of cues. ”
This paved the way for haptics to be used to create more emotional connections. Digital communication collapses the sense of spatial distance. In the past that has been done mainly using vision and sound but the haptic revolution aimed to create a digital space that allows us to experience synthetic touch as an immersive experience, creating a real sense of physical closeness. The greatest potential was seen in the improvement of long distance relationships and in offering health benefits for touch-deprived people such as the elderly  and the sick. “The special thing about touch is that it’s immediate and emotional. Being touched by someone makes us feel more connected to them. ” Seymour explains. “Touch is our ‘private’ sense  and it has so many benefits for our health  and relationships .”
“Being touched by someone makes us feel more connected to them.”
Despite the hype, early haptic devices were rather clunky and didn’t really live up to the expectations of ‘realistic’ experiences that people had come to expect from image and sound-based communication. A range of devices in the 2010s used localised vibration and muscle stimulation as symbolic messages of ‘touch’ and physical presence.  These novelty gadgets had very little use in everyday life. Later attempts relied on more and more advanced tech that sits on or in the skin. However, this still confined the experience to specific parts of the body and was very limited in the quality and type of touch that could be conveyed. 
4: Parisi (2018)
5: Teledildonics is the technology of remotely linked sex toys that operate in synch (Lynn, 2004).
6: There have been recaps of the development of haptic technology to date (e.g.: Stone (2001) and Parisi (2018)) that conclude that scientists and technologist are still hunting for the ‘holy grail’ of haptics to create a realistic, immersive experience.
7: See n. 5.
8: Apple (2014) as quoted in Lupton (2016)
9: Imaginary company behind NeoTouch
10: See Klinenberg (2012) on the phenomenon of more and more people living alone.
11: Greenfield (2014) reports on a trend of n increasing number of people living alone and correlates this with changes in social interactions and reported feelings of loneliness.
12: Loneliness weakens the immune system (Greenfield, 2014) and increases the risk of stroke (Greenfield, 2014) and heart diseases (Valtorta, 2016).
13: Turkle (2015), Greenfield (2014), Seltzer, Prososki, Ziegler and Pollak (2012), Lord (2013)
14: Seltzer, Prososki, Ziegler and Pollak (2012)
15: Cocozza (2018)
16: Montagu (1971), Pallasmaa (2012), Chillot (2013. B)
17: The scientific field of proxemics defines touch as one of our private senses (along with smell and taste). These senses make us feel immersed in an experience as opposed to a removed observer as we do with everything we experience exclusively through our eyes and ears. Touch is immediate, intimate, close in contrast to vision, sound and even smell. (Hall, 1966)
18: There has been much research done on the effect of touch and tactile input on our physical and emotional health – e.g., warm touch lowers stress hormones and enhances trust and attachment (Chillot, 2013. B). For more research in this field see: Field (2010), Heinrichs, Baumgartner, Kirschbaum and Ehlert (2003), Henricson et al. (2008) and Eaton, Mitchell-Bonair and Friedmann (1989)
19: Gulledge and Fischer-Lokou (2003) found a high correlation between physical affection and overall relationship and partner satisfaction.
20: E.g., Keep in Touch, a fabric touch screen for long distance relationships (Motamedi, 2007), Hug Over a Distance, an air-inflatable vest (Mueller et al., 2005), the HugShirt by Cutecircuit (2002), the Paro Seal Robot (www.parorobots.com) for dementia patients (Sharkey and Wood, 2014), Tactilu, a bracelet for remote tactile communication by Pangenerator (2013), and the Teslasuit (www.teslasuit.io, 2015) for gaming and Virtual Reality to name just a few.
21: Research in on-skin and under-skin technology see: Vega et al. (2017), Leigh et al. (2017), Kapur, A., Kapur, S. and Maes (2018), Kao, Johns, Roseway, and Czerwinski (2016), Masters and Michael (2005).
The rise of NeoTouch
In the late 2020s bioelectronics finally left the lab and found their way into mainstream products for mental and physical health as well as cognitive improvement. These brain-computer interfaces also started being used to control devices remotely, directly through brain signals as well as to merge the human brain with artificial intelligence (AI). In haptic technology, these interfaces made it possible to synthesise the experience of human touch without imitating it mechanically on the skin. Simply by stimulating the relevant brain areas directly, this new kind of haptic technology can create the physical illusion of touch in a very convincing way. 
As Hannah Eisen, Designer at NeoTouch, explained: “In the past technologists and scientists focused on simulating the physical process of touch. Obviously, they never got the technology to advance far enough to really create a realistic experience. But then with the introduction of chip implants that interface directly with the brain came a massive shift in how we looked at the problem. Because suddenly we were able to synthesise the holistic experience of touch instead of just recreating it mechanically.” (NeoTouch. An IA Lab documentary, 2037) 
As the first internal human-to-human interface NeoTouch has – without a doubt – had the biggest impact on the nature of social interaction since the smartphone.
As the company claims in a video from 2028 that introduces the technology:
“NeoTouch lets you truly connect with another person at a distance. It enables instinctive, non-verbal communication through digital touch.” 
“NeoTouch lets you truly connect with another person at a distance.”
The video goes on to explain how it works: “The tactile interaction is received through your phone and then sent to the NeoTouch transducer: The Senser. This unit attaches to the skin behind the ear and communicates wirelessly with a network of nano-electronics in the brain to simulate the sensation of being in touch with another person.  These neural implants serve as an internal brain-computer interface that controls and receives the communication. They interface directly with the somatosensory and motor cortex. It is the stimulation of these particular brain areas that allows us to create a realistic experience of touching and the sensation of being touched. ”
The modern era is characterized by technological revolution and environmental changes. Today’s environments are shifting faster than ever in the way they operate making people’s ability to be adaptable essential. However, this rapid change found them in societies and built environments that were structured to be easily controllable, thus standardized and predictable. Architectural spaces designed according to normalized (male) or ideal bodies set by Ernst Neufert (1970) and Le Corbusier (1950 and 1955) are representative examples of this logic. Moreover, national education systems that were developed according to the demands of the Industrial Revolution (Robinson, 2011, p.8), by their majority have only slightly sifted their structure, continuing to have as their main task the distribution of information in order to be memorized.
As Heinz von Foerster (2003a, p.208) notes humans always had the tendency to ‘trivialize’ their environment, meaning to make it predictable in order to be able to control it. However, trivialization has been applied not only by humans to their environments, but also vice versa. People live their lives in a habitual way, in which determined principles apply in specific circumstances. Frequently, any unpredictable output is considered false or unpleasant and most of the times any diversion has to be restored (Von Foerster, 2003a, p.208).
However, predictability or comfort are states which cannot stimulate negotiation, adaptation and evolution. In Bruno Munari’s photo-essay Ricerca della comodità in una poltrona scomoda (Searching comfort in an uncomfortable chair) (Munari, 2012) the difference between comfort-discomfort or predictability-unpredictability becomes apparent showing the human body negotiating with an environment that does not serve the presupposed function.
Figure 1. Uno torna a casa stanco per aver lavorato tutto il giorno e trova una poltrona scomoda (One comes home tired from working all day and finds an uncomfortable chair), page detail Published in Domus 202 / October 1944 (Munari, 1944).
The above ideas guided my interests on how an unpredictable environment could work as an auto-pedagogical tool stimulating people’s non-trivial responses and triggered the formation of the following research questions: (1) What is the necessary level of legibility of an interactive system that would make people willing to enter an unknown-unpredictable environment? (2) How can this environment remain non-trivial for its participants even when an adjustment time has passed? (3) How could participants learn to not only constantly adapt into the new environment, but also stimulate its non-triviality by their own non-trivial actions?
The analysis of a series of projects (referred to as Hacks) developed by my collaborators (Alexandra Niaka and Isabella Ong) and me, was used by the research to address the above issues. The observations of the projects were examined in relation to a series of theories, concepts and other examples of continuously evolving systems. Finally, our thesis project Ichni, which draws on this research is described. This project aims to continue and stimulate this investigation, test the new assumptions and show further leaks and observations related to the research questions.
2.0 Von Foerster’s Trivial Machines
People always had the tendency to try to understand or simplify their environment in order to be able to control it. Heinz von Foerster (2003a, pp.207-10) uses the term ‘trivialization’ – the conversion of complexity into a ‘trivial machine’ – to describe this process. ‘Machine’ is not referring to a physical structure but to the ‘well-defined functional properties of an abstract entity’ (Von Foerster, 2003a, p.208). A trivial machine is organized as a simple algorithm connecting inputs and outputs in a fixed way. In contrast, the relationship between inputs and outputs of a non-trivial machine is fluctuated, related to its previous outputs (Von Foerster, 2003a, p.208).
As already mentioned ‘trivial’ refers to a standard relation between inputs and outputs. So, a specific input always causes the exact same response regardless of the time passed or the previous stimuli (Von Foerster, 2003a, p.208).
Almost all the devices in everyday life are designed to be trivial. They have a specific set of functions described in manuals for the users (humans) to be able to know exactly what the outputs will be. As Von Foerster (2003a, p.208) mentions whenever a device (he uses the example of turning the starter key to a car) stops giving the predictable outcome is considered broken and the deviation from its supposed function has to be restored.
People also try to convert their non-trivial environments into trivial machines. As an example, Von Foerster mentions ‘the discovery of agriculture’, which ‘is the discovery that some aspects of Nature can be trivialized: If I till today, I shall have bread tomorrow’ (Von Foerster, 2003a, p.208). Consequently, trivialization can take place in two ways. The first way is by transforming a non-fixed input-output relationship into a fixed one (e.g., training a dog). The second way is by understanding a behaviour when discovering the algorithm that connects inputs and outputs (Von Foerster, 2003a, p.202) (e.g., the case of agriculture).
However, people, intentionally or not, are applying trivialization to themselves too. Education systems, frequently, become examples of human trivialization. In general, school trains children in a specific set of questions with predefined answers, while learning is based on memorization. So, tests become the ‘devices to establish a measure of trivialization’ (Von Foerster, 2003a, p.209).
Furthermore, reduction, which is used to help people understand and control their environment, has in many cases become a trap (Lefebvre, 1991, pp.105-7). Henri Lefebvre (1991, p.105) refers to reduction as a scientific method ‘designed to deal with the complexity and chaos of brute observations’. The effect of reduction can be found in the design rules of the architectural space according to normalized (male) or ideal bodies set by Ernst Neufert (1970) and Le Corbusier (1950 and 1955). In many architectural schools, Neufert’s book Architects’ Data (1970) is used constituting the basis of architectural design. However, by constructing their environment according to rules based on reduction, people end up eliminating the possibilities of their own interactions with it. Their environment in relation to the process of socialization through which people have been raised and trained to behave in specific, acceptable ways, trivializes them.
According to Von Foerster ‘in a non-trivial machine […] the output is determined by the input and its internal state’ (2003b, p.196). Every input alters the internal state of a non-trivial machine through time. So, by changing and evolving the way in which the machine operates, it ‘becomes a different trivial machine’ (Von Foerster, 2003c, p.142) after every input. This shift between states makes this type of behaviour unpredictable. In human-environment interaction, non-triviality could be related to human creativity. Creativity here refers to the capacity of people to adapt to new situations as they perceive the stimuli, alter their internal state and respond accordingly.
However, as discussed above people have creativity taught out of them through education and society. As Sir Ken Robinson (2011, p.8) notes, national education systems arose out of the demands of the Industrial Revolution and were designed according to its interests and structure. In contrast, today’s environments are shifting faster than ever due to technological revolution and environmental changes. Therefore, people’s ability to be adaptable becomes essential.
So, a new need related to reversing the trivialization processes is arising. Von Foerster (2003a, p.209) relates detrivialization to teaching people to ask and answer unpredictable questions. In this way, people are perceived as individuals free to evolve their internal state, follow their impulses and act on their desired future, by gaining knowledge from past experiences but without being restrained by the past rules. This process of detrivialization is the one that the research aims to investigate through the following chapters.
3.0 Space-Hacking Projects
As has already been mentioned, people have chosen to design trivial environments in order to be able to control them. However, when the predefined stabilities are sifted people need to operate without having absolute control over their world. This is when tactics arise. The term ‘tactics’ refers to the methods that people use to operate and adapt to new situations (De Certeau, 1984). Michel de Certeau (1984, p.xix) describes them as methods of the weak to ‘turn to their own ends forces alien to them’. When someone has not absolute control over their environment, they need to be in constant alertness, ready to seize the opportunities (De Certeau, 1984, p.xix). Therefore, tactics are not independent of context. However, they do have their own time, which is when different elements of the environment can be combined and used towards goals and needs (De Certeau, 1984, p.xix).
In this case, the environment has its own ‘strategy’ (De Certeau, 1984, p.xix). It sets the rules and the constraints. As De Certeau describes it, the environment controls and promotes a scenario or a score and determines the possible relationships. The rules become the basis that supports the development of tactics. It could be ‘the equivalent of the rules of meter and rhyme for poets of earlier times: a body of constraints stimulating new discoveries’ (De Certeau, 1984, p.xxii).
The relationship between people’s tactics and the strategy of the environment becomes apparent in collective interactivities formed by children games like Dodgeball and Hide and Seek, and a rehearsal process like Stanislavski’s Method of Physical Actions (specifically the use of a sequence of Tasks ) (Benedetti, 1998). In both cases, the strategy of the system is based on people’s conflicting or interrelating goals in relation to the surrounding space or the script. In collective children games, there are two objectives, with the one opposite of the other (e.g., in Hide and Seek); or there are objectives that trigger competitive behaviours (e.g., a running competition), while Stanislavski introduced the use of sequences of Tasks that actors needed to accomplish in the given situations driving their improvisation and performance (Benedetti, 1998). In both cases, the tasks lead to immediate actions and are related to the other players.
In this way, each participant’s actions cause the sifting of the existing circumstances, triggering the others to adapt and evolve their performance. The interconnection of participants’ tactics assures the constant detrivialization of the system.
Triggered by the above ideas my collaborators (A. Niaka and I. Ong) and I developed a number of Hacks, designed to facilitate collective interactivity – aiming to question the trivial environments and people’s trivial responses in their everyday life. Well-established activities and their related environments were chosen. The criterion for this selection was the occurrence of standardized, habitual human behaviours within these spaces, which subsequently were hacked to disrupt the characteristics that made these systems trivial.
The Hacks investigated (1) people’s ability and willingness to start operating and develop tactics in new environments, (2) the level of their non-trivial responses and evolving tactics and (3) the level on which the participants by themselves stimulated the unpredictability and evolution of these systems.
The projects that will be examined in the context of this essay are (1) Body-, (2) Stairs-, (3) Auditorium- and (4) Bench-Hack (2,3 and 4 also referred to as Space-Hacks). They will be elaborated in chronological order since the methodology of designing these Hacks was developed as an evolutionary process where the observations of each project influenced the design of the next ones.
3.1 Reconfiguring the body [Body-Hack]
The first project was developed before the space-hacking methodology was formed. However, it led to interesting conclusions related to the human adaptability into new complicities. The participants’ bodies were linked in different ways with the use of ropes (Figure 2) and they were asked to accomplish simple tasks (Isabella Ong, 2018a). So, people’s movement could be confronted with mechanical limitations and new potentials.
Figure 2. Reconfiguring the body, Diagram of the different complicities (Chrapana, Niaka and Ong, February 2018)
Starting with individuals (Figure 2a and 2b), first, each participant was asked to explore the ways they can move, with the constraints and potentials of their movements. Almost no one could respond successfully to this general task immediately. The participants tried to find what could be a right response for this by asking for more explanations, but they did not try to move without them. However, subsequently, a more specific simple task (like, which is the most comfortable position or way of walking?) helped them explore their possibilities and try new movements. In the end, each participant’s most comfortable movement was different. Therefore, it could be argued that when a task becomes more specific participants may feel that part of the responsibility of their actions is transferred to the task itself. In this way, they are free to explore without having the fear of being judged for their choices (in the first case the task was related to creating a movement, while in the second with finding or discovering a movement).
Another observation was related to the perceived difficulty of the given tasks. The rope interconnections were designed in order for the tasks to be achievable for all the participants. However, several times the participants refused to even to try to perform a task because they thought that the constraints of the ropes would restrict them. In these cases, my collaborators and I were helping them by giving instructions or by making the task simpler in the beginning and then increase the difficulty through time. Interestingly, when people finally achieved the difficult goal, they realized the extents of their abilities. Consequently, this understanding triggered them to ask for more difficult goals, wanting to find the limits of their possible movements.
Interestingly, when the interconnections were created between two participants (Figure 2c and 2d), differences between individual’s and pair’s interaction was observed regarding their willingness to try and achieve a goal, the development of their tactics and the complexity of the created systems.
In the case of pairs, participants were asked to collaborate for the achievement of a common goal. It was observed that they were more willing to start performing in their new environment than individuals. This can be justified in two ways. First, sharing the responsibility for the performance and the developed tactics with another one made each participant less afraid of failure and more willing to try. Second, especially when the pairs were consisting of one beginner and one master of the system (one of my collaborators), the experienced one could help the other feel more comfortable and capable by giving instructions and demonstrating the possible movements.
Moreover, although a system of pairs seemed safer and more accessible, it became more complex and evolving. While the participants were collaborating, the behavioural proposal of the one was affecting the other (because of the restrains of the ropes). So, the other needed to adapt anew, change his/her movement or make a new proposal in order for them to achieve their common goal.
In conclusion, the Body-Hack showed a number of important parameters that can affect the willingness of the participants to enter an unknown environment, explore it and evolve their tactics. First, the goals were giving guidance and meaning to participants’ exploration and undertook part of the responsibility for their behaviour and choices. However, it was important for the task to seem achievable and the system to be well explained for the participants to be willing to try. Therefore, the difficult tasks were divided into smaller steps with progressively increasing difficulty. After the development of the following Space-Hacks, it became apparent that the above method was the key to triggering participants’ exploration. It made the new environment seem safe yet continuously challenging while achieving a task triggered participants to ask for more difficult ones.
Finally, the system of pairs seemed safer, yet more challenging and complex. When participants were collaborating, they were sharing their knowledge and skills and the responsibility for their performance. Moreover, each participant’s action was affecting the other one and as a result, their negotiation and evolution of their tactics were constantly stimulated.
The observations of this first project stimulated the development of the following Space-Hacks, where the above assumptions were tested, and the related ideas were clarified and evolved.
3.2 Mind your step [Stairs-Hack]
The Stairs-Hack (Mind your step) and Auditorium-Hack (Don’t be a couch potato) were developed together introducing the space-hacking methodology.
The repetitive rhythm of climbing the stairs made this architectural typology an ideal choice for the Space-Hacks. Moreover, since staircases are considered as transition spaces, their function is directly related to a specific task (climb up or down in order to go to a specific space). This could assure the existence of a goal that could drive participants’ tactics in the hacked space.
The above parameters led to the design of four moving platforms, as time-based obstacles installed on a staircase. They were constructed from cardboard and they were moving relatively slowly in order to not be a serious hazard. The colors, shape and material of the platforms, made them look familiar, drawing from toy design.
This paper questions notions of spatiality in translating from the physical to the virtual. By discussing and recontextualizing Jean Baudrillard’s concept of the hyperreal in a digital modernity, I question the emerging technologies in contemporary music, art, and architectural practices. I argue that these translations are not lossless, that what is vanished is a sense of presence , or what Walter Benjamin defines as object aura, leading to a holistic cybernetic fantasy which blurs the line between the virtual and the real. The physical object is slowly tranquilized and replaced with less potent simulacra of itself. Furthermore, cybernetic algorithms have largely informed modern conceptions of intelligence, thus ignoring the ways in which naturally-occurring physical systems (e.g., rainforests, murmurations, fluid dynamics) also form networks encoded with complex information. Interactive artwork which is mediated by the digital screen poses a layer of abstraction between the viewer’s eyes and the originary subject.
My research aims to imagine a lost future which reengages materiality. I discuss the technological principles of weaving to develop a mechanical cellular autonomer whose conditional movements are materially encoded during construction. By discussing passive compliance in soft robotics, I demonstrate how smart materials can be employed to address environmental unpredictability more effectively than computerized control systems. The computational properties of bar linkages are examined: their original use for computing continuous functions is compared to their subsequent application as early binary converters. This quantization of analog information, from continuous data to discrete data, from unfiltered information to 1’s and 0’s, is a process which intentionally simplifies and restructures the natural world in order to increase control and accuracy. My works seeks to voluntarily relinquish this computerized regulation in favor of an analog aesthetic.
The Hyperreal and the Real
I.: Technological Utopianism
Cyberspace, in its conception, was once a radical utopian experiment for Silicon Valley hackers in the early 90’s—a parallel world separate from the aggregate political problems in a Bush-era United States. Rather than a space to be controlled and monitored, the Internet was imagined with a freedom empty of borders and capital, a type of escapism from the preexisting hierarchies in the real world. It was not a loss of faith but instead a reconceptualization of idealisms, a way to start over from scratch in an ungoverned wilderness. In his manifesto The Declaration of Independence of Cyberspace , John Perry Barlow writes:
Governments of the industrial world, cyberspace does not lie within your borders. We are creating a world where anyone, anywhere, may express his or her beliefs, no matter how singular, without fear of being coerced into silence or conformity. I declare the global social space we are building to be naturally independent of the tyrannies you seek to impose on us. We will create a civilisation of the mind in cyberspace. May it be more humane and fair than the world your governments have made before (Barlow, 1996).
This statement was copied in over 40,000 sites upon its publication. While there still exists the promise of an uncontrolled virtual space as a location where impossible dreams can be materialized (as evidenced by developments in VR, AR, MR, and AI as aesthetic disciplines), global networks have also lead to the proliferation of cybernetic technologies which have dangerous consequences for the future of civilian regulation. What Barlow describes is a lost future, one which never fully materialized.
What did materialize is a future based in the micromanagement of risks and assets, numerical calculations meant to maximize growth and minimize loss for the population as a whole (Curtis, 2016). The anonymous French leftist collective Tiqqun writes at length about the development of a cybernetic state, one that idealizes management and control. According to Tiqqun, the goal of cybernetics is to solve “the metaphysical problem of creating order out of disorder.” The unity of cybernetics “has imposed itself as the world-wide method of universal enrollment, simultaneously a rage to experiment, and a proliferating oversimplification.” This oversimplification is necessary to control instability and crisis created from capitalist growth, and is empowered by the state of emergency (Tiqqun, 2010).
The expansion of cyberspace‚ following a consumerist model of growth, becomes entangled with the pursuit of profit. Technological progress becomes unilaterally equivalated with corporate prosperity: wearable gadgets, security software, the internet of things, interactive advertising, smart cars, smart devices, and handheld screens dominate modern conceptions of innovative technology. These technologies do not address the multiplicity of the user as they inherently atomize individuals into singularities of capital; even personal customization becomes a marketing technique.
This also leads to what Henry Giroux describes as a “disembodied self” — the rise of a posthuman subjectivity ultimately void of geospatial culture, instead defined by a virtual world which is fully monetized under the illusion of escapism.
Cybernetics—automated control systems meant to mechanize the flow of bodies—are now invisibly used for predictive policing, targeted advertising, health and fitness apps, monetary management, and other mundane elements of daily life. Continued dependence on algorithmic forecasting leads to “technological solutionism,” where layer upon layer of control is added in order to mediate between reality and the increasing complexity of an interconnected world (Morozov, 2013).
II. Technological Solutionism:
In China, a country with one of the highest pollution and overpopulation rates in the world, companies sought to address issues of overcrowding in cities with the creation of bike sharing apps. Their software allows users to rent a bike cheaply by the hour, perceived to be a simple way to reduce carbon emissions through communal property. In 2017, dozens of bike-share companies quickly flooded the streets with millions of rental bicycles in order to seize the market. Chinese infrastructure was not meant to handle the sudden insurgency, resulting in mountains of abandoned bikes which were soon impounded and left in vacant lots, or chaotic piles (Taylor, 2018). Their debris have become a haunting sight, a reminder of the market bubble bursting on false dreams of corporatized sustainability.
Figure 1. A pile of discarded ride share bikes in Xiamen, Fujian province, China (2017).
Figure 2. An aerial view of unused rideshare bikes collected from Shanghai streets by local authorities, arranged neatly into row (2017).
Translations between imagined Internet movements and its real-life implementation outside of the cybernetic utopia are not seamless. Software solutions are often incongruent with the material world, as they oversimplify already existing political problems, further complicating them until they spiral into madness. In his documentary HyperNormalisation ,Adam Curtis argues that, since the 1970s, governments have created a simplified and completely simulated version of reality, run by corporations and kept stable by politicians under the guise of maintaining a functioning society (Curtis, 2016). Social media sites such as Twitter and Facebook are effective for joining masses of people under a common emblem, but fail to mobilize populations into real agents of resistance. This can be seen in movements that began as a hashtag on the netscape but never metamorphosed into physical scenes with lucid goals: the Arab Spring and Occupy Wall Street are potent examples of this (Curtis, 2016) .
The discord between initial hysteria and long-term endurance are markers of the anxious and schizophrenic present, a pathological temporality which fail to be foundational for real change. The desire to program virtual simulacra could indicate a cultural panic over the loss of material conditions, namely that the real world has become so impermeable that users must create alternate realities in order to retroactively imagine a future that includes them. A symptom of this intense alienation, atomization, and privatization is what Jean Baudrillard refers to as hyperreality —a postmodern semiotics in which it is impossible for human consciousness to distinguish reality from a simulation of reality (Baudrillard, 1994). It is the condition in which what is real and fiction are seamlessly blended, augmented by a digitization in which we seek stimuli from a copy world and nothing further. Baudrillard argues that this “reality by proxy” operates not simply as a perverse image, but instead as an autonomous world that lacks definite origin.
Figure 4. Hyperreality: images of clear skies on an LED screen in Tiananmen Square, in front of a smoggy Beijing sky (2014).
Smart Cities and the Myth of Interactivity
The smart city is an urban space engineered to manage resources with maximized efficiency. This is implemented through electronic data collection systems which allow built structures to respond intelligently to human stimuli. What constitutes intelligent interactivity will become mediated by these technologies which fully regulate the population via the passive collection of data on its habitual trends. A citizen’s participation in the smart city is thus reduced to a data point. And yet responsive architecture—which is meant to creatively responds to the user’s demands—has been delineated by these surveillant gadgets, constructing a prevailing myth of interactivity. In his book The City is NotATree,C hristopherAlexanderwritesabouthowconceptionsofarchitectural interactivity have been confined to information transmitted between the eyeball and the hard drive. He gives the example of a traffic light at a crosswalk as a network which responds to its users:
“For example, in Berkeley at the corner of Hearst and Euclid, there is a drugstore, and outside the drugstore a traffic light. In the entrance to the drugstore there is a newsrack where the day’s papers are displayed. When the light is red, people who are waiting to cross the street stand idly by the light; and since they have nothing to do, they look at the papers displayed on the newsrack which they can see from where they stand. Some of them just read the headlines, others actually buy a paper while they wait. This effect makes the newsrack and the traffic light interactive; the newsrack, the newspapers on it, the money going from people’s pockets to the dime slot, the people who stop at the light and read papers, the traffic light, the electric impulses which make the lights change, and the sidewalk which the people stand on form a system—they all work together” (Alexander, 2015).
These physical systems are already systems of interactivity—by existing in the world and reacting to human stimuli, they form networks of information between a myriad of agents and objects. Physical systems which allow chaos to occur (vs. a preprogrammed finite list of options) leads to an interactivity that is less calculated and more variable in its function than a fully computerised system. The crosswalk model described by Alexander can be contrasted with a vision for the smart city crosswalk, temporarily installed in South London by urban technology developer Umbrellium.
Figure 5. Umbrellium’s model of the smart crosswalk, as it lights up for a pedestrian unknowingly crossing into traffic while being distracted by their smartphone (2017).
In a promotional video for the project, a pedestrian is absorbed in her mobile device and unknowingly walks into oncoming traffic. The smart crosswalk detects this trajectory and lights up a LED crosswalk beneath her feet. Umbrellium writes, “If a person is too close to the road surface when a car is nearby, a warning pattern lights around them to fill their field of vision” (Umbrellium, 2017). This appears as an improvement to public safety, but it comes ingrained with a contradiction. The user was not engaged with the physical world because they were engrossed in a virtual one, but the proposed solution for the issue is more technology, another layer of cybernetic abstraction. Conversely, this prototype leads to less interaction between the users and their environments; it blurs the line between image and world, allowing users to move seamlessly between the two as they collapse into a singularity. In Virtual City, or The Wiring and Waning of the World, Sanford Kwinter writes:
“Though today we in the west receive far more information than at any other point in human history, we certainly receive far less than ever before in an unfiltered, raw, or unmediated form, in what might be called its ‘whole’ state, that is, naturally embedded in a sensuous complex array and apprehended directly by actual experience… All background noise and all free, unchanneled flows are eliminated in the name of creating frictionless, ‘dedicated,’ or task directed environments” (Kwinter, 1996).
The example of the smart city shows how models of interactivity have been ingrained in processes of digitized data collection in which the active citizen becomes a point on a graph. This pushes old models of interactivity into obsolescence, overwriting the body’s dynamic synergy with its landscape unless it has been holographically rendered. Total digital connectivity ignores how information transmitted between analog systems is also interactive without being recorded in a database. A techno-futurism is one in which the body is simulated and phantasmic, a semi-presence in a world of automation.
Not only do these smart cities reinforce surveillant biopolitical agendas (e.g., predictive policing, mobile device tracking, CCTVs blended in the scenery), they also render the population invisible by lacking the agency to restructure their own habitats according to their needs. In Alberto Vanolo’s essay Is Anybody Out There? The Place and Role of Citizens in Tomorrow’s Smart Cities , he writes:
“There is arguably little space for citizens’ voices in this imaginary, because planners and technological gurus seems to know exactly what citizens desire and how to provide it to them, much in line with the approach assumed in the tradition of colonial and modernist utopian planning…The real agency of the active smart citizen populating this imaginary is very limited, because it is reduced to the generation of data that are manipulated, controlled and mobilised in ways that are completely out of control of most of citizens’ understanding of technologies” (Vanolo, 2016).
While the smart city gives the illusion of structural fluidity, it is precisely the opposite in practice: the population is passively regulated and responds mechanistically, choosing between a limited selection of possible outcomes selected by an elite class of architects and engineers. What is lost is the ability and freedom to mould our own physical environments.
Hackney Wick & Maker Culture
The contemporary fixation on software skills leads to the obsolescence of hand skills: the keyboard, mouse, and touchscreen have become universal instruments for every career. This represents a flattening of skills and the disembodiment of the working mind. In the mass exodus of the hand-making from urban culture, urban citizens are stripped of a wide array of empirical knowledge such as how to build furniture or grow food from the earth, leading to a kind of tactile starvation and depreciation of skills that are explicitly tangible to the body. The disappearance of communities of makers can be seen as a symptom of this transference of skills. An example of this is in the gentrification of Hackney Wick, an urban artists’ neighborhood in East London, and the location of the newest Bartlett School of Architecture campus.
Hackney Wick was home to over 600 artist studios in 2008. Artists lived in cooperatives within converted warehouses, often with 8-24 people per unit (Brown, 2012). These concentrated spatial arrangements became foundational to the community which thrives within it—garbage found on the street was collected and made into murals, graffiti artists climbed perilous heights to vandalize high-rise apartments, and underground raves took place deep within the Hackney Marshes to the tune of hand-made sound systems. The structural elements of the warehouses are often not engineered for these purposes by professional architects; the communities maintain full jurisdiction to alter their habitats to suit their changing desires.
Figure 6. Exterior and interior drawings of the Victoria Wharf warehouses (2012).
These warehouses, which were once home to industrial plants, have been refurbished by hand and made into workspaces where artists can freely rearrange its elements without contact with the landlord. In this way, warehouses units operate as individual ecosystems rather than isolated singularities (such as apartment buildings or suburban-style houses). This generates communities where many objects are communal and resources are managed with egalitarian efficacy. By seeking to live cheaply and autonomously, these artists form communities rooted in the unique spatiality and material resonance of their habitats. The ability for artists to self-regulate their spaces, having the power to materially reconstruct them as needed, was crucial to Hackney Wick’s vitality. But what were once self-sustaining communes are quickly being disappeared due to the expansionist nature of capital accumulation.
The feeling of presence (FoP) is a fleeting sensation often associated with neurological disorders or spiritual encounters, and yet its experience among healthy individuals are scarcely researched. Upon constructing a media-rich installation with diverse sensory stimuli, I hope to elicit the experience of FoP in normal subjects. Two key areas to explore include altering one’s bodily self-consciousness and supplying ambiguous sensory cues in the environment. A temporary mismatch between the sense of agency and the sense of ownership is suspected to be a potential method to induce FoP from participating visitors. After a critical evaluation of the preliminary prototypes, I shall propose an installation art piece that relies on the latest computer vision and video projection technologies to enable the sensorimotor conflicts that encourage the FoP experience based on cast-body shadows.
Background and Objectives
The feeling of presence (FoP), sometimes referred to as “sensed presence”, and contentiously defined as “extracampine hallucinations” (Chan & Rossor, 2002) or “leibhaftige Bewusstheit” (Jasper, 1913; Sato & Berrios, 2003), is a sensation far more common than the general public take notice of. This phenomenon is most frequently associated with hallucinations accompanying sleep paralysis attacks (Cheyne & Girard, 2007), illusory sensations arise in Parkinson’s disease and dementia with Lewy bodies (Fénelon et al., 2011; Wood et al., 2015; Chan & Rossor, 2002), the sensing of a deceased one following bereavement (Steffen & Coyle, 2011), and hallucinatory experiences in extreme survival scenarios (Brugger et al., 1999; Geiger, 2010). The rapid development in the field of cognitive neuroscience has brought renewed attention to FoP in the past decades, and yet, the lack of a consistent body of literature indicates the need of a systematic investigation on the topic.
Despite the variety of experiences associated with FoP, a few salient features emerge across the board. First, similar to other forms of hallucination, FoP occurs convincingly without identifiable external stimuli (Nielsen, 2007; Solomonova, Frantova & Nielsen, 2011; Alderson-Day, 2016; Barnby & Bell, 2017). Any visual, auditory or tactile cues perceived from the environment are only interpreted “in subordination to the presence” (Solomonova, Frantova & Nielsen, 2011), as they contribute to the construction of a coherent narrative. Second, the presence is of “a purely spatial nature” (Nielsen, 2007), which means its emergence does not rely on mental representations we abstract from sensory channels such as vision, hearing or touch. Instead, it contains “the most basic of all invariant perceptual information available in the environment” (Gibson, 1966) such as that of “spatial location, orientation, mass, volume, extent, movement, trajectory”(Nielsen, 2007). Third, FoP is always experienced through the perception of a human, contrary to a non-human figure (Chan & Rossor, 2002). Some further propose that the presence carries a sense of intentionality and that they “seem to have goals or intentions with respect to the self” (Nielsen, 2007).
In the past, a few theories have been brought forth to explain how the anomalous phenomenon of FoP came into being, which include the “illusory reduplications” of one’s body projected into the extrapersonal space (Brugger, Regard & Landis, 1996; Blanke, Arzy & Landis, 2008), the generation of “hallucinatory variants of social imagery” (Nielsen, 2007; Solomonova et al., 2008; Fénelon et al., 2011), or a threat activated “hyper-vigilant and biased attentive state” (Cheyne, 2001; Cheyne & Girard, 2007). Unfortunately, there has yet to be a tried and tested method that replicates the sensation of FoP in the lab to unveil the fundamental sensory elements that trigger such an experience. The only experimental elicitation of FoP by transcranial magnetic stimulation (Cook & Persinger, 1997; Persinger & Healey, 2002) was quickly debunked through a double-blind trial in later years (Granqvist et al., 2005). On the other hand, a single case targeting electrical stimulation towards the temporoparietal junction of the brain was reported successful to induce the sensation of FoP (Arzy et al., 2006). The scholars called attention to the functions of the temporoparietal junction involved in “self-processing, self–other distinction, the integration of multisensory body-related information, and other illusory own-body perceptions”. This has motivated the suspicion that a disturbance in “multisensory (proprioceptive and tactile) and sensorimotor integration of information” with respect to the body can lead to a bodily illusion that promotes FoP.
In the years to follow, a series of research led by Blanke proceeded in the direction that sees FoP as “an illusory own-body perception with well-defined characteristics that is associated with sensorimotor loss” (Blanke, 2014). He and his colleagues designed a master-servant robotic system to produce specific sensorimotor conflicts in the human body, aiming to verify FoP “caused by misperceiving the source and identity of sensorimotor (tactile, proprioceptive, and motor) signals of one’s own body”. The results were most interesting when asynchronous feedback stimulation was introduced, which caused a shift from illusory self-touch to the feeling of being touched by other. The test was found successful for inducing FoP on normal participants and gave profound insight into atypical sensorimotor signal integration leading to the phenomenon.
This particular finding has helped focus my research and revealed the significance of two critical aspects that contribute to the FoP experiences: the fluctuation of self-attribution and the ambiguity of sensory stimulants. In this light, it might be valuable to first gain an understanding of the human bodily self-consciousness and then work towards altering the phenomenal selfhood through a digital environment. When it comes to the self, there has been such an abundance of research conceptualising its definition all across the spectrum that any attempt to forge a comprehensive review would appear but inadequate. In later sections of this paper, I shall organise the most relevant findings taken from cognitive science and establish a theoretical foundation for my design decisions.
The primary objective of my research is not to agonise over an infallible explanation for the occurrence of FoP, but to test out existing psychiatric theories through experiments and hopefully extend their application to the realm of installation art. The final design project would aim to push the boundary of bodily self-consciousness via implementing a series of computer-mediated sensory cues in the environment. More than sixty years after Ryle has abolished the “ghost in the machine” argument (Ryle, 1949), I hope to further the conversation and to discover if the computer manipulation of external conditions – corresponding to the human body – would result in the human mind expanding its self-identity. During the design research, I will speculate the minimum multi-sensory stimuli it requires to induce the FoP experience. A range of mechanisms and materials will be examined in the hope that the eventual outcome is a rewarding one.
As a remarkably intriguing theory goes, “no such things as selves exist in the world” (Metzinger, 2003). In a seemingly absurd fashion, Metzinger stresses that the construction of the self is an ongoing and dynamic process, where “the subjective experience of being someone emerges if a conscious information-processing system operates under a transparent self-model.” He introduces a distinctive model of such cognitive operation called the “phenomenal self-model”, which corresponds to what Gallagher calls the “minimal self” – as opposing to the “narrative self” (Gallagher, 2000). The minimal self-awareness contains two pivotal concepts: the “sense of agency” and the “sense of ownership”.
Sense of agency: The sense that I am the one who is causing or generating an action. For example, the sense that I am the one who is causing something to move, or that I am the one who is generating a certain thought in my stream of consciousness.
Sense of ownership: The sense that I am the one who is undergoing an experience. For example, the sense that my body is moving regardless of whether the movement is voluntary or involuntary.
These two aspects are vital to the moment-to-moment motor planning process as well as to the creation of the immediate self-consciousness. A computational model proposed to illustrate the sensorimotor integration cycle in the body (Wolpert, Ghahramani & Jordan, 1995; Wolpert & Ghahramani, 2000) could also be elevated to the phenomenal level of selfhood. As shown in Figure 1, Gallagher demonstrates the two comparators involved in executing an intended action, with the match in the forward model providing a sense of agency and the match in the feedback model yielding a sense of ownership. Upon confirming both matches, a sensorimotor loop would then be rendered complete. We will find this procedure equally beneficial for constructing the phenomenal level of representation that constitutes the fundamental experience of our conscious existence, although not all agree on the matter. Unlike many scholars who thought agency to be a necessity in achieving self-consciousness (Gallagher, 2000; Metzinger, 2003; Pacherie, 2007), Blanke and Metzinger maintain that “bodily agency is a causally enabling, but not a constitutive condition, for phenomenal selfhood” (Blanke & Metzinger, 2009).
Figure 1: The forward and feedback comparators (Gallagher, 2000)
Figure 2: Neuropsychological model of inserted thought (Gallagher, 2000)
On the contrary, as discovered in the master-servant robot experiment, a reduction in the sense of agency is likely to result in sensorimotor conflicts leading to the generation of FoP (Blanke, 2014). It should come as no surprise once we compare this with symptoms found in schizophrenia, such as thought insertion and delusions of control. Figure 2 shows the sense of agency being lost through the lack of a match in the forward model, while the cognitive feedback model completes its regular loop. As a result, the schizophrenic patients would fail to integrate the reafferent feedback signals into their intended state and therefore feel their thought or action being controlled by an external agent (Frith, 1987).
It is important to note that these symptoms differ from another abnormality associated with the deficiency of agency, the anarchic hand sign (Blakemore, Wolpert & Frith, 2002; Pacherie, 2007), which is when patients report complex movements from an upper limb noncompliant to their intention. The intriguing difference here is the fact that the anarchic hand sign patients never express a sensation of alienation from their limb, whereas the experience from delusions of control is always attributed to an alien force. The schizophrenic patients are recorded making statements such as “feeling like an automaton”, “guided by a female spirit who had entered me”, or “spirits moving my shoulder” (Spence et al., 1997). Scholars discovered that the anarchic hand sign only offends the agentive self-awareness in a higher-order narrative self, but not the low-level mechanisms of sensorimotor integration pertaining to the minimal self (Blakemore, Wolpert & Frith, 2002; Pacherie, 2007). Such discrepancy draws attention to an important feature of the alien presence in FoP: it is a hallucination bearing the “most basic type of perceptive information” (Nielsen, 2007). Cheyne poetically describes it as “a feeling of raw otherness present-at-hand, its only quale an ineffable sense of ‘thereness’” (Cheyne, 2001). Regardless of the impact FoP brings towards our narrative self-understanding, it is foremost a disturbance to the most fundamental perception of existence. This notion should inform my depiction of FoP as an experience that challenges our utmost basic instincts about the selfhood.
Meanwhile, it is worth considering the three popular hypotheses of FoP: external body mapping, social imagery, and threat activation. On sufficient theoretical grounds, a focused effort will undoubtedly enhance the effectiveness of my design outcome.
To start with, FoP proclaims a shift of the bodily proprioceptive localisation outwards to one’s extrapersonal space. The occurrence of proprioceptive drift was first observed in the rubber hand illusion, a seminal experiment to achieve visuotactile integration of a fake limb into the body perception (Botvinick & Cohen, 1998). Ever since then, plentiful research deriving from the original experiment investigated representations and actions of body parts, yet the global and unitary character of the self was often neglected. The holistic framework which founds our immediate experiences is an elementary condition that enables our bodily self-consciousness. Blanke believes there to be three major aspects related to the phenomenal selfhood: “self-identification with the body (that is, the experience of owning a body), self-location (that is, the experience of where I am in space) and the first-person perspective (that is, the experience from where I perceive the world)” (Blanke, 2012). In his research, Blanke listed extensively the experiment set-ups that target these three facets and revealed how each condition could be altered to mimic the episodes generally occur in out-of-body experiences, heautoscopy, and autoscopic hallucinations. These procedures use video, virtual reality, robotic devices, etc., to induce in healthy individuals “out-of-body illusions or full-body illusions, that arise from visuotactile and visuo-vestibular conflicts”(Blanke, 2012). Blanke’s work is a substantial resource of inspiration for creating the full-body proprioceptive drift during a FoP experience.
Figure 3: Two models of FoP production during sleep paralysis attacks (Nielsen, 2007)
On the other hand, the FoP production models of social imagery and threat activation (see Figure 3) are under much debate. Both are preconditioned through ambiguous sensory cues in the environment, by which influence the brain to “create the perception of a person from partial sensory stimuli during states of increased arousal (fear, paranoia)” (Chan & Rossor, 2002). Simard and Nielson were the first to compare the phenomenon of FoP to social anxiety. Once triggered, the anxiety prompts spontaneous social imagery that replicates past negative social events (Simard & Nielsen, 2005). An alternative explanation of FoP involves the vigilance system being activated by detection of possible threat. According to Cheyne, it would then initiate procedures that “monitor the environment for further cues to corroborate or disconfirm the existence of an emergency” (Cheyne, 2001). In both cases, the ambiguity of sensory stimuli works in favour of the FoP generation. This feature is especially valuable for creating “immersion by suggestion”, a concept first formulated by Frantova in referring to the Holy Theatre. Inside a production of the Holy Theatre, an actor usually becomes “an embodiment of the invisible: a prophet, an oracle or a priest through whom the Other-world talks” (Frantova, Solomonova & Sutton, 2011). It is an art form of invisible-made-visible, but “not automatically – it can only be seen given certain conditions” (Brook, 1996). In a similar way, an environment conducive to FoP should provide ample sensory signals that are open for interpretation, allowing the gap between reality and immersion be filled by the imagination of the visitors.
Results and Discussion
In an effort to induce FoP through immersive artwork, I have designed a series of preliminary prototypes to analyse different aspects of the theoretical foundation. The experiments to date follow a similar formula: The visitor’s location and movements are monitored by a tracking device such as a digital camera or a Kinect sensor. Converting these data into spatial coordinates, the computer would calculate a specific function corresponding to the visitor trajectory. Visual and auditory feedback is then created by the use of a CNC plotter, affecting a layer of physical material with forces like magnetic energy or wind power. The latest trials, nonetheless, diverge away from the X-Y plotting mechanics, turning to computer vision and digital projection for a more immersive design.
THE MACHINE AMONG US - Vimeo
The Machine Among Us (Lea, 2018) was my first attempt to achieve the uncanny – a feeling often accompanies the FoP experience. The prototype consists of a basic CNC plotter, a light-box, some playing sand, and a spherical metal scribe. The detection of a visitor approaching the apparatus would trigger the plotter to run a predetermined script. At this point the carriage moves the metal scribe through a magnetic piece underneath the drawing surface, and soon a written message appears in the sand. With the plotter hidden away from sight, this prototype relies on mere illusion to the eye. It is a fanciful stimulant that gives away the bare minimum which becomes an open invitation to a variety of suitable narrative. And yet, this version of the installation does not touch upon a disturbed sensorimotor loop affecting the self-consciousness.