JAS
← All projects

Research / AI-assisted design / 2026

Design Tacit Knowledge Capture

Turning Figma interaction traces into reviewable knowledge for designers.

Year
2026
My contribution
Research, system design, prototyping & evaluation
Context
Master’s research · Keio University SDM
Participants
5 professional UI/UX designers
Knowledge-capture framework: observable behaviour supports language-model proposals, with validity held by the expert designer.

Design decisions disappear while the interface remains.

UI/UX designers make many small decisions while working: what to prioritise, what to compare, when to revise, and why one option is preferred over another. Much of this knowledge is tacit. It is expressed through action and judgement but may never be written down.

Retrospective interviews and think-aloud methods can help recover design rationale, but they rely on memory and require designers to explain their work after or during the task. Continuous questioning can also interrupt the activity being studied.

This research investigated another source of evidence already produced during digital design work: fine-grained Figma interaction traces. The aim was to examine whether those traces, combined with the design brief, could surface candidate design knowledge for the designer to review.

AI proposes. The designer decides.

The research purpose was to examine whether Figma interaction traces and a task brief could be transformed into candidate tacit-knowledge cards for expert review, and to compare the resulting material with retrospective think-aloud reflection.

The system does not claim to reconstruct a designer’s private thoughts. It uses observable actions and task context to propose possible goals, concerns, strategies, and reasoning. Designers then decide whether each proposal should be kept, edited, or discarded.

Feasibility and quality

The project focused on the feasibility and quality of candidate knowledge capture.

Scope

It did not test knowledge transfer to novice designers, evaluate a deployed workplace product, or perform full visual analysis of the rendered interface.

Turning an invisible process into a reviewable system.

I translated the research problem into a six-stage system that connects behavioural evidence with expert judgement. The prototype records fine-grained Figma actions and combines them with the design-task context. It groups related actions into design episodes and selects the most informative episodes for analysis.

For each selected episode, the language model generates a short candidate knowledge card. Each proposal remains connected to its source episode so the designer can inspect the basis of the inference.

From behavioural evidence to reviewed knowledge
  1. 01

    Capture

    Record Figma interaction events during a design task.

  2. 02

    Segment

    Organise the action stream into meaningful design episodes.

  3. 03

    Select

    Prioritise episodes with stronger potential for knowledge capture.

  4. 04

    Generate

    Combine episode evidence and task context to propose knowledge cards.

  5. 05

    Review

    Let the designer keep, edit, or discard each proposal.

  6. 06

    Report

    Preserve reviewed cards and comparison material for analysis.

Candidate knowledge card about iterative colour exploration, showing related episodes, an interpretation, and a supporting evidence summary.
Candidate output — an interpretation to inspect, correct, or reject.Full size ↗

Scroll within the image to inspect details.

A reviewable proposal
  1. Candidate knowledge

    A card can describe the design object being changed, the immediate goal, the concern being addressed, and the strategy used.

  2. Supporting evidence

    The evidence in the interaction trace informs the interpretation. Each proposal remains connected to its source episode.

Three decisions shaped the system.

Decision 01

Use episodes as the unit of interpretation.

Individual interface events are too small to explain a design decision, while an entire session contains too much unrelated activity. Grouping actions into episodes created a middle layer that could represent a coherent piece of design work.

Decision 02

Make every output reviewable.

The system presents interpretations as candidate cards with links to their supporting actions. Keep, edit, and discard controls preserve disagreement and correction instead of turning a plausible AI response into an assumed fact.

Decision 03

Capture reflection before showing AI output.

During evaluation, retrospective think-aloud took place before card review. This ordering protected the designer’s independent recollection from being shaped by the system’s suggestions and created a cleaner comparison between the two sources.

Testing the workflow with professional designers.

Professional UI/UX designers with at least four years of experience completed a design task in Figma while the recorder captured their interaction traces. After a short break, each participant reflected through retrospective think-aloud, reviewed the generated cards, and completed a questionnaire.

The analysis compared reviewed cards with retrospective material. It examined what each candidate concerned, the goal or concern it described, the strategy it proposed, and the interaction evidence supporting it.

This evaluation treated quality as more than whether a sentence sounded convincing. It examined whether the proposal was supported, whether the designer accepted it, and whether relevant knowledge was missing.

Study order — independent reflection precedes AI exposure
  1. 01

    Design task

    Figma interaction traces recorded.

  2. 02

    Short break

    Logs exported.

  3. 03

    Retrospective think-aloud

    Reflection before seeing AI output.

  4. 04

    Card review

    Keep, edit, or discard candidate knowledge.

  5. 05

    Questionnaire

    Feedback collected.

Commission · inaccurate proposals

Keep, edit, and discard decisions exposed interpretations the system proposed inaccurately or too strongly.

Omission · missing knowledge

Comparison with retrospective think-aloud made relevant knowledge visible that the selected episodes or generated cards did not capture.

A reviewable capture process, with clear limits.

The completed prototype turned fine-grained design interactions into evidence-linked candidate knowledge that professionals could inspect, correct, and reject. Its main contribution is the combination of episode-based capture, selective language-model interpretation, and structured expert review.

The comparison with retrospective reflection provides a way to examine both inaccurate proposals and missing knowledge. This makes human review part of the knowledge-capture method rather than a final approval step added after generation.

Limits

The study involved professionals and one design-task setting. One researcher conducted the qualitative coding. The system analysed interaction traces and task context rather than the complete visual meaning of the interface. Prototype reliability issues also affected some sessions. The findings support feasibility and methodological development, not broad claims of generalisation or production readiness.

Next steps

A future version could improve recorder stability, examine multi-cursor and collaborative activity, include carefully bounded visual context, and evaluate the workflow across different design tasks and larger expert samples. Independent or multiple coders could also strengthen the qualitative analysis.