Methodology Reference

Activity Stability (Vocabulary Jaccard)

Jaccard similarity of the unique state label sets between consecutive activity passes. Measures vocabulary overlap, not per-frame agreement.

Higher values indicate more stable vocabulary across passes. Penalizes valid refinement (e.g., "editing_video" becoming "editing_video_in_premiere" counts as instability).

Intent Stability (Vocabulary Jaccard)

Jaccard similarity of the unique intent label sets between consecutive intent passes. Measures vocabulary overlap, not per-frame agreement.

Penalizes valid refinement. Intents are meant to become more specific across passes, so lower stability is not necessarily bad.

Activity-Intent Consistency

Measures alignment between activity states and their inferred intents. Fraction of frames where the activity and intent share at least one significant word.

Values closer to 1.0 indicate activities and intents are well-aligned. Low values may indicate mismatched abstraction levels.

Entropy

Shannon entropy measuring the diversity/disorder of state labels. Higher entropy means more varied states.

p(x) is the probability of each unique state. Higher values indicate more diverse state labels.

Normalized Entropy

Entropy normalized to [0,1] range for comparison across different state vocabularies.

n = number of unique states. 1.0 = maximum disorder (uniform distribution). 0 = all frames have same state.

Average Confidence

Mean confidence score across all frames in a pass. The LLM's self-reported certainty on a 1-10 integer scale, enforced by Pydantic validators. Anchored as: 1-3 = occluded/blurry/ambiguous, 4-6 = visible but multiple descriptions possible, 7-8 = clearly visible with minor ambiguity, 9-10 = only one reasonable description.

Known limitation: 8B VLMs exhibit ceiling effects on verbalized confidence, with scores clustering at 9-10 regardless of actual ambiguity. This metric is most informative for identifying genuinely unclear frames (scores 1-3) rather than distinguishing between "good" and "great" labels. Inter-pass label agreement may be a more reliable uncertainty signal.

High Confidence Ratio

Proportion of frames with confidence ≥ 8 (out of 10).

Typically very high (97%+) due to the confidence ceiling effect in small VLMs. Most useful for comparing across videos rather than interpreting absolute values.

Self-Transition Rate

Proportion of consecutive frame pairs where the primary state remains the same.

High values (90%+) indicate stable, long-duration states. Low values suggest frequent state changes or jittery labeling.

Confidence Delta

Change in confidence between first and last activity pass for a frame.

Positive values indicate refinement improved confidence. Negative values suggest later passes were less certain.

Frames

Total number of screenshots analyzed in this pass. Each frame is a single point-in-time observation of the user's screen.

Unique States

Number of distinct primary state labels assigned across all frames in this pass. A lower count means the model consolidated frames into fewer categories.

Unique Intents

Number of distinct intent labels inferred across all frames in this pass. Intents describe the purpose behind the observed actions.

Total Passes

Number of analysis passes run over the footage. Odd passes annotate observable actions, even passes infer intents. The two levels are mutually informative: action evidence anchors intent inference, intent context disambiguates ambiguous actions.

Vocabulary empirically stabilizes by pass 8 (schematic equilibrium). Running beyond this point yields diminishing returns, though additional passes may still refine individual frame labels.

Schematic Equilibrium

The point at which the raw state vocabulary converges to a stable size across successive passes. Both action and intent vocabularies empirically stabilize by pass 8, indicating the iterative refinement has reached conceptual stability.

Convergence is measured by vocabulary size plateauing across consecutive same-type passes. Schematic equilibrium validates the multi-pass design: early passes produce noisy, inconsistent labels; later passes converge to a stable behavioral taxonomy without an imposed ontology.

Behavioral Complexity Profile

Per-video comparison of normalized entropy, vocabulary size, and self-transition rate from the last activity/intent pass.

Higher entropy and vocabulary indicate more diverse behavior. Higher self-transition rate indicates longer dwell times.

State Embedding (SentenceBERT)

Each state label is converted from snake_case to natural language and encoded into a 384-dimensional dense vector using the all-MiniLM-L6-v2 SentenceBERT model. Semantically similar states (e.g., "editing code" and "writing code") produce nearby vectors.

The embedding captures semantic similarity between state labels. States that describe similar activities cluster together in the projected 2D space.

UMAP 2D Projection

High-dimensional SentenceBERT embeddings are projected to 2D using UMAP (Uniform Manifold Approximation and Projection). Preserves both local neighborhood structure and global topology.

Nearby points represent semantically similar states. Clusters indicate groups of related activities. Edge thickness shows transition frequency. The projection is computed separately per pass; use Global mode for cross-pass comparison.

Transition Probability

Probability of transitioning from one state to another, learned from the training set using a first-order Markov chain with Laplace smoothing.

Higher probability means a transition is more likely. Self-transition probabilities indicate how long the model expects the user to stay in a state. Laplace (add-one) smoothing ensures no transition has zero probability (Chen & Goodman 1999).

Top-1 Accuracy

Percentage of test transitions where the model's single highest-probability prediction matched the actual next state.

Higher is better. Compare against majority-class baseline to assess whether transition structure adds predictive value beyond marginal frequency.

Top-3 Accuracy

Percentage of test transitions where the actual next state appeared among the model's 3 highest-probability predictions.

Higher is better. A large gap between top-1 and top-3 suggests the model knows the likely transitions but struggles to pick the single best one.

Perplexity

Measures how surprised the model is on average. A perplexity of N means the model is as uncertain as choosing uniformly among N states.

Lower is better. Standard language model evaluation metric (Jurafsky & Martin 2024, §3.4). All models use Laplace smoothing to avoid infinite perplexity from unseen transitions.

Cross-Video Predictability

Best-pass Markov top-1 accuracy compared across videos. A next-action prediction task: User Models trained on the first 60% of frames predict the held-out final 40%, measuring whether captured behavioral dynamics generalize to unseen activity.

Action passes typically score higher than intent passes due to smaller vocabulary. Videos with repetitive activities (e.g., coding) score highest. The 60/40 temporal split avoids data leakage from shuffled frames.

Vocabulary Consolidation (Label Normalization)

Agglomerative clustering of state labels by SentenceBERT cosine similarity. Merges synonym clusters (e.g., "editing_code" and "writing_code") using an optimal threshold calibrated on 200 human-annotated label pairs.

Higher reduction percentage means the raw vocabulary had more synonyms. The normalized label for each cluster is the most frequent member. Threshold t* = 0.4268 selected by maximizing F1 on human annotations (precision = 0.768, recall = 0.883).

Semantic Alignment (Activity-Intent)

Per-frame cosine similarity between the SentenceBERT embedding of the activity state and the intent state from paired passes (P1-P2, P3-P4, P5-P6).

Values above 0.5 indicate strong semantic agreement between what the user is doing (activity) and why (intent). Distribution histogram reveals whether alignment is consistent or bimodal.

Intent Grounding Score

How well each intent is grounded in observable activities. Computed as the maximum cosine similarity between the intent embedding and all activity embeddings.

Grounded intents (similarity > 0.5) have a close observable activity counterpart. Ungrounded intents may represent abstract goals without direct behavioral evidence.

Cross-Pass State Lineage Drift

Tracks how state labels evolve across same-type passes (P1→P3→P5 for activities, P2→P4→P6 for intents). Drift measures how far the final label has moved from the root label.

Low drift means the concept is stable across passes. High drift indicates significant refinement or reconceptualization.

Vocabulary Convergence

Tracks vocabulary size and overlap between successive same-type passes using Jaccard and semantic overlap metrics.

Increasing Jaccard between later passes indicates vocabulary stabilization. Trend is classified as converging, diverging, or stable.

Transition Matrix Divergence

Jensen-Shannon divergence between transition probability matrices of successive same-type passes. Measures structural behavioral change.

Decreasing JSD indicates the transition structure is stabilizing. Converging transitions with converging vocabulary suggests the model has found a stable behavioral representation.

Normalized Mutual Information

NMI between each pass's state sequence and the final pass. Measures how much information each pass shares with the final result.

NMI > 0.9 indicates a pass is near-redundant with the final pass. Low NMI in early passes shows meaningful refinement occurred.