MotifBeyond Articulation: Grounding End-User Program
Specifications in Observed Behavior

Shaokang Jiang1, Minseo Kang2, Daye Nam1

1 University of California, Irvine  |  2 Yonsei University

Abstract

People can use AI to write programs, but they still need to know what to automate and explain how it should work. Motif starts with what people already do: it observes everyday browsing, discovers recurring routines, and lets users review and refine them before generating programs. In a controlled study of 100 patterns, adding behavioral evidence doubled the number of working programs from 21 to 42. A multi-day deployment with eight participants surfaced useful automation opportunities they had overlooked. These findings suggest that observed behavior can complement what users express in words.

Behavior-Grounded Specification

Everyday activity becomes a library of recurring routines, each backed by observed actions and variations. Users select and refine a routine to specify what their program should do.

Observation of everyday activity across several days builds a specification library of grounded routines, including polishing email drafts with Gemini and sorting Google Scholar by year. User articulation selects and refines a routine to create a personalized program.

Video

A walkthrough of Motif, from discovering browser routines to reviewing and generating programs. Watch on YouTube.

Pipeline

Motif pipeline. From everyday browsing activity to discovered patterns, reviewed designs, and personalized programs.
Motif pipeline: data collection, action abstraction, pattern mining, program design, ranking, user review, and program generation.

Motif captures browser interactions and suggest potential programs. Users can then revise and deploy a Tampermonkey program.

Interface

Annotated Motif interface: monitoring and URL blacklist controls at upper right, the discovered-pattern menu at lower right, and an editable program design on the left.
Motif interface. Monitoring controls let users pause observation and exclude URLs. The pattern menu lists discovered routines. Selecting a pattern opens a program design that users can edit, regenerate, remove, or turn into a userscript. Click the figure to view it at full size.

Results

Study 1: Behavioral Specification improves program performance

We evaluated 200 programs across 100 patterns constructed from WebLINX1. Each pattern was generated under two conditions: a task description alone, or the same description supplemented with observed actions. Researchers assessed functionality and fidelity to the described goal.

Study 1 Results: Behavioral specification improves both the functionality and pattern fidelity of the generated programs.
ConditionFunctionalityPattern Fidelity
WorkingNot WorkingGoal-onlyInventedMissing
Behavior-grounded42588569
Description-only2179661618

Behavioral evidence doubled the number of working programs and reduced invented functionality. Goal-only programs stayed within the described goal, although they did not necessarily execute successfully.

Study 2: Real-World Deployment

Eight participants used Motif during multiple days of browsing, then selected, reviewed, and tested five programs each.

Motif surfaced personalized administrative workflows, including renewing multiple Facebook Marketplace listings in one action, finding and forwarding enrollment-proof emails to an employer, and batch-filling timesheet entries for non-service days.

Discovered patterns and program outcomes. Original figure (PDF).
Post-study survey results. Scores range from 1 to 5; higher is better. Original figure (PDF).
“I found that the pattern identification in motif is very useful, because it’s very subtle, so I didn’t realize it, but if I can have a program to help me do that, I feel it’s very useful.”— P4

Citation

Cite us!

@misc{jiang2026motifdiscoveringautomatingpersonal,
      title={Motif: Discovering and Automating Personal Web Workflows}, 
      author={Shaokang Jiang and Daye Nam},
      year={2026},
      eprint={2607.10531},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={https://arxiv.org/abs/2607.10531}, 
}