INKWELL METHODOLOGY
Sequential Probability Ratio Test (SPRT)
Drift Portal

Knowledge Base & Feature
Drift Methodology

A comprehensive guide to the Sequential Probability Ratio Test (SPRT) framework used by AgentInk to monitor semantic alignment and detect divergence between user requests and implemented features.

1. Overview & Vision

For an advanced, self-evolving software engineering system like AgentInk, maintaining alignment with stakeholder requirements is paramount. While static metrics like "Knowledge Base Completeness" or "Graph Density" are useful for checking structural database integrity, they are completely blind to semantic and operational drift. An agent could have a 100% complete, highly dense graph that is completely out of alignment with what was actually requested.

To solve this, Inkwell implements the Sequential Probability Ratio Test (SPRT) (located in inkwell/metrics/drift_sprt.py). This statistical framework evaluates whether the stream of implemented tasks (from Linear/headway) is staying aligned with the initial user requests (from the request backlog) or if the project is experiencing semantic drift.

2. Hypotheses & Parameter Definitions

The SPRT is formulated around two competing hypotheses regarding the cosine similarity scores (\(x_i \in [0, 1]\)) between implemented tasks and the best-matching user requests:

Null Hypothesis (\(H_0\))

No Drift (High Alignment)

The implemented features remain highly aligned with the initial user requests. Under \(H_0\), similarity scores are expected to be high, centered around a mean of \(\mu_0 = 0.75\).

Alternative Hypothesis (\(H_1\))

Drift (Divergence)

The implemented features have diverged from the initial user requests. Under \(H_1\), similarity scores are expected to be lower, centered around a mean of \(\mu_1 = 0.50\).

The Default Parameter Choices:

Null Mean (\(\mu_0\)) 0.75
Alternative Mean (\(\mu_1\)) 0.50
Std Dev (\(\sigma\)) 0.15
  • \(\mu_0 = 0.75\): A cosine similarity of 0.75 or higher indicates very strong semantic and keyword alignment. This allows for minor variations in phrasing (e.g., synonyms or technical jargon) while confirming alignment.
  • \(\mu_1 = 0.50\): A cosine similarity of 0.50 or lower indicates weak keyword overlap and significant divergence. This represents the threshold where the features being implemented are starting to deviate from the core requirements.
  • \(\sigma = 0.15\): Represents the expected natural variation/noise in similarity scores. It acts as a buffer to prevent false alarms from minor phrasing differences.

3. Mathematical Mechanics: The Log-Likelihood Ratio

With each new task analyzed, the detector calculates a similarity score \(x_i\) and updates a cumulative log-likelihood ratio sum. Assuming a normal distribution with equal variance \(\sigma^2\) under both hypotheses, the log-likelihood ratio increment (\(\Lambda_i\)) is:

\[\Lambda_i = \ln \left( \frac{f(x_i \mid \mu_1)}{f(x_i \mid \mu_0)} \right) = \frac{\mu_1 - \mu_0}{\sigma^2} \left( x_i - \frac{\mu_0 + \mu_1}{2} \right)\]

Plugging in our default values (\(\mu_0 = 0.75\), \(\mu_1 = 0.50\), \(\sigma = 0.15\)):

1

The Decision Midpoint:

\[\frac{\mu_0 + \mu_1}{2} = \frac{0.75 + 0.50}{2} = 0.625\]

2

The Scaling Factor:

\[\frac{\mu_1 - \mu_0}{\sigma^2} = \frac{0.50 - 0.75}{0.15^2} = \frac{-0.25}{0.0225} \approx -11.11\]

3

The Increment Formula:

\[\Lambda_i \approx -11.11 \times (x_i - 0.625)\]

How the Midpoint Governs Decisions:

  • If \(x_i > 0.625\) (High similarity, closer to \(\mu_0\)): The term \((x_i - 0.625)\) is positive. Multiplying by the negative scaling factor (\(-11.11\)) makes the increment \(\Lambda_i\) negative. This pulls the cumulative sum down towards the lower boundary \(A\) (confirming `NO_DRIFT`).
  • If \(x_i < 0.625\) (Low similarity, closer to \(\mu_1\)): The term \((x_i - 0.625)\) is negative. Multiplying by the negative scaling factor (\(-11.11\)) makes the increment \(\Lambda_i\) positive. This pushes the cumulative sum up towards the upper boundary \(B\) (confirming `DRIFT_DETECTED`).

4. Decision Boundaries & Error Rates

The decision boundaries are determined by the acceptable Type I error (\(\alpha = 0.05\)) and Type II error (\(\beta = 0.05\)):

Lower Boundary (A) \[A = \ln \left( \frac{\beta}{1 - \alpha} \right) = \ln \left( \frac{0.05}{0.95} \right) \approx -2.9444\]

If the cumulative sum reaches or falls below this boundary, the detector confirms NO_DRIFT (alignment is solid) and resets the cumulative sum to 0.

Upper Boundary (B) \[B = \ln \left( \frac{1 - \beta}{\alpha} \right) = \ln \left( \frac{0.95}{0.05} \right) \approx 2.9444\]

If the cumulative sum reaches or exceeds this boundary, the detector triggers DRIFT_DETECTED (divergence confirmed) and resets the cumulative sum to 0.

5. Interactive SPRT Simulator

Test the methodology on the fly! Adjust the similarity score below to see how the log-likelihood ratio increment is calculated, or simulate a sequence of scores to see how the cumulative sum updates and triggers decisions.

Single Score Calculator

0.0 (No overlap) 1.0 (Perfect)
Value 0.75
Increment (\(\Lambda_i\)) -1.39
Hypothesis Pull Pulls to H0 (No Drift)

Sequence Simulator

Click the buttons below to feed a sequence of scores into the detector and watch the cumulative sum update.

Cumulative Sum 0.0000
Decision Boundary A: -2.9444 | B: 2.9444
Current Decision CONTINUE
Simulation History:
// Simulator initialized. Feed scores to begin...

6. Practical Impact in the Inkwell Workspace

Running the sequential analysis across our active tasks and Slack requests yields highly actionable and revealing operational insights:

Phase I: High Alignment

Early tasks like bastrop-cad-parcels-retrieval, fema-floodplains-retrieval, and config-vm-deployment are directly mapped to detailed siting search and GCP hosting requests, showing exceptionally high similarity scores (\(\sim 0.85\) to \(0.95\)) and pulling the SPRT sum down.

Phase II: The Start of the Drift

The drift starts immediately when generic/placeholder items or unrelated tasks are added to the backlog:

  • 100x: Had a near-zero similarity score to real requests, immediately shooting our SPRT sum past the upper boundary \(B\).
  • Cursor Admin Tasks: Tasks like INK-347 (Deactivate unused Cursor seats) represent infrastructure/licensing work that, while necessary, drifts away from core product feature requests.
  • Mock/Testing Tasks: Identical test tasks (LIN-105 through LIN-133 titled "Test Task Title") have extremely low semantic similarity, causing the SPRT sum to climb rapidly.