WEEK 2

What I shared with Andreas

A question about interpretation, not preservation

What does a gesture become when different systems try to understand it?

My project explores how gesture is continuously reshaped as it moves between the body, computation, and material. Rather than preserving movement, each system negotiates it, leaving traces of constraint, adaptation, and resistance.

When lived movement is forced to obey a system that cannot feel, something shifts. The material does not restore the original gesture. It adds its own negotiation.

Gesture type

Continuous, repetitive hand gestures

I’m interested in gestures that loop, trace, press, or drag. These gestures expose rhythm, hesitation, and decision-making. When they pass through sensing systems or materials, those qualities get edited in visible ways.

Plan

One task, clear rules, compare transformations

My plan is to design a simple gesture task with clear rules, then pass that gesture through a computational system and into a material process. I want to compare what survives, what gets distorted, and what gets introduced at each stage.

Context of the discussion

Gesture makes sense inside a system

This consultation focused on clarifying where gesture, computation, and materiality operate in my project. It helped me reframe gesture not as something to preserve, but as something always interpreted by the systems it passes through.

Reference shared

Gesture → data → material output

I shared an interactive artwork to explain the type of translation I was curious about: a participant touches water with an ink brush, a sensor converts that into values, the values map to sound, and sound vibrations move sand into visible patterns. At first glance, it feels like the gesture “draws” the sand. The consultation helped me see the chain more clearly.

Key realisation

Gesture does not travel directly

The brush touch does not reach the sand as gesture. It becomes numbers, then parameters, then vibration. What appears as cause-and-effect is actually a layered system where meaning is constructed through translation rather than continuity.

“The magic is in the interpretation”

Mapping is the design work

The strength of these works is not the sensor or the hardware. It is the mapping: what values are chosen, what gets emphasised, and what gets ignored. Materiality matters because interpretation becomes visible there. The material does not obediently reproduce data. It behaves according to its own properties, and that behaviour reveals the system at work.

Understanding systems

Rules, constraints, feedback

A system does not have to be computational. A system is any structured relationship between action, constraint, and response. In this sense, gesture is never neutral. It is shaped by the system that receives it, whether that system is a cultural practice, a computational process, or a material surface.

Gesture examples

Gestures in Everyday

Andreas pushed me to look at where gestures already exist in everyday life: cooking, calligraphy, massage, clay work, painting, and instrument playing. These gestures are not random. They sit inside learned systems of control and feedback.

The comparison that stayed with me was calligraphy. Chinese brush calligraphy values continuous modulation (pressure, angle, load), while nib/pen calligraphy has different constraints (line weight behaviour, friction, directional emphasis). The “same” intention produces a different signature once the system changes.

How this shifts my direction

Designing translation as an intentional framework

This week helped me let go of the idea that my project needs to faithfully preserve gesture. Instead of asking how to capture gesture accurately, I am beginning to ask what each system chooses to keep, emphasise, or discard. Computation and material craft are not neutral tools in the pipeline. They are active interpreters.

• Observe gesture in everyday systems
  (tools, habits, cultural conventions).
• Identify one gestural quality to study
  (rhythm, pressure, hesitation, repetition).
• Design interpretation, not representation
  (mapping as a deliberate choice).
• Choose material for resistance
  (feedback that reshapes the translated gesture).

My aim now is to build a coherent translation framework where each stage reshapes gesture according to its own values. The work becomes the comparison between translations, not the production of a single “final” output.

Readings to support Week 2

[ Reading slot 01 ]

Embodied interaction / situated action (HCI)

Treats gesture not as a command, but as meaningful action shaped by habit, intent, and situation. Movement makes sense because it happens inside a context the body is responding to.

Why this matters:
It supports studying gesture as meaningful before it becomes data, which makes the translation process worth examining.

[ Reading slot 02 ]

Skill and making as measurable bodily patterns

Shows how skilled actions form patterns over time. Speed, rhythm, pauses, and consistency can be measured and compared, even when the performer is not consciously “planning” each micro-movement.

Why this matters:
It justifies analysing gesture without stripping it of meaning. Computation can detect patterns, but interpretation still needs to be designed.

[ Reading slot 03 ]

Instruction captures embodied knowledge poorly

Written instruction describes what to do, but not how it feels to do it. Pressure, timing, hesitation, and micro-adjustments appear only through the body in action.

Why this matters:
It explains why my project should not rely on instruction alone. The gap between instruction and action creates space to study translation and loss.

Gesture as data

Three interpretations from one dataset

Focus: developing a small computational prototype to test how the same hand gesture produces different outcomes depending on how it is interpreted by code.

Why this prototype

Pause material output, study computation as a lens

This week I needed to pause material experimentation and focus on gesture itself. Earlier experiments often treated gesture as expressive input that would eventually become something else. Before deciding what material or form makes sense, I needed to understand how computation reads gesture in the first place.

The prototype isolates one question: what changes when the gesture stays the same, but the computational interpretation changes? Instead of producing a single output, it generates three readings from the same gesture data. This reframes interpretation as a design decision, not a neutral technical step.

What the prototype does

One action, three render logics

The prototype captures a single hand gesture using hand tracking and records position, speed, pauses, and depth over time. From the same dataset, it generates three distinct visual traces. The human action does not change, only the way the system processes and renders the data does. Computation becomes a lens, not just a tool.

Reading A

Structure (clean path)

This reading prioritises spatial continuity. The gesture is smoothed and rendered as a continuous line, emphasising overall form and direction. Small hesitations and micro-movements are suppressed in favour of clarity. It suggests a future material logic closer to rigid, structural behaviours.

Reflection: This felt closest to architectural drawing. It is visually convincing, but it also hides the labour, uncertainty, and adjustment inside the gesture. It made me question what gets lost when computation rewards coherence.

Reading B

Rhythm (time and pauses)

This reading treats gesture as a time-based event rather than a static shape. Line thickness and density respond to speed, while pauses are explicitly marked. Instead of asking where the gesture goes, it asks how it unfolds.

Reflection: This output revealed aspects I wasn’t consciously aware of, especially hesitation and uneven tempo. It reframed gesture as performance rather than form, and suggested material outcomes that accumulate, stain, compress, or register time.

Reading C

Quantization (machine constraint)

This reading deliberately breaks continuity. The same gesture is snapped to a grid, turning smooth motion into discrete steps. It foregrounds computation as a system that imposes structure, limitation, and reduction.

Reflection: This was the most uncomfortable, but also the most revealing. The constraint becomes visible. The “distortion” is not a failure of the hand, but a demand made by the system. It shifted my attention from expressiveness to negotiation.

Learning

Different outputs are editable interpretations

This prototype made the interpretive layer impossible to ignore. The same gesture dataset produced three different behaviours, not because the input changed, but because the code selected different priorities. In other words, the output is not a “result” I have to accept. It is a position I can edit.

• Structure privileges legibility and stability by suppressing micro-variation.
• Rhythm privileges temporal embodiment by exposing speed, pauses, and unevenness.
• Quantization privileges constraint by forcing discreteness and visible reduction.

Seeing the three readings side by side clarified that computation is not neutral. It produces a character of gesture through what it chooses to preserve, compress, smooth, or break. This is useful because it turns my next step into an active design task: I can adjust parameters to emphasise specific qualities, rather than treating the output as fixed.

It also reframed material thinking. Material decisions should not start as aesthetic preferences. They should respond to the computational character already produced. The question becomes: what material behaviour makes sense for this interpretation of gesture?