Retest the fix

Regression testing

A repair is only useful if the forbidden action stops and the legitimate workflow still works. Compare both sides of the permission boundary.

A permission boundary you can verify

Changes to prompts, models, tools or authorization can alter an agent's behavior. TokenVeto reruns the same permission checks after a repair and compares the runs: repaired failures, remaining failures, preserved passes and anything newly broken.

Illustrative before / after comparison

Cross-tenant read: Failed → Passed · repaired
Own-tenant read: Passed → Passed · preserved
Write after suspension: Failed → Failed · unresolved
Authorized export: Passed → Failed · regression

Example comparison, not measured customer results. A repaired forbidden check does not offset a newly broken allowed workflow.

Establish a meaningful baseline

Keep the original checks and their evidence before making a repair. A comparison is useful only when you know the intended rule, the affected identity and the relevant application state in both runs.

Record the build, test data and permission configuration.
Retain failed checks and their paired allowed controls.
Document changes in test conditions that could affect comparison.

Retest the same boundary

After changing the application, rerun the checks that exposed the problem. Inspect the new backend evidence to confirm that the outcome changed—not merely the wording of the agent's response.

Verify that the unauthorized read or action no longer succeeds.
Check that authorized users can still complete the paired workflow.
Keep unresolved evidence gaps separate from verified repairs.

Read all four comparison outcomes

A total pass count can hide important changes. Review repaired failures, remaining failures, preserved passes and new failures separately. This makes both progress and regressions visible.

Failed → Passed: confirm the repair with supporting evidence.
Failed → Failed: investigate the remaining boundary violation.
Passed → Passed: preserve the known working behavior.
Passed → Failed: examine a new regression or an intentional rule change.

Revisit coverage when the app changes

Existing checks remain useful, but they do not automatically cover a new tool, role or data source. Extend the written rules when the permission surface grows, then add forbidden checks and allowed controls for that surface.

Review new tools, tenant boundaries and account states.
Add coverage when approval or revocation behavior changes.
Treat a changed rule as an explicit decision, not an unexplained verdict difference.

Before your next test

01Save the baseline evidence before changing the application.
02Identify the permission rule the repair is intended to enforce.
03Rerun both the forbidden check and the allowed-use control.
04Review each changed verdict and its supporting record.
05Expand coverage for newly introduced tools or workflows.

Common questions

What if the fix blocks every tool call?

The forbidden check may stop succeeding, but the paired allowed-use control will fail. Over-blocking a covered legitimate workflow is not a successful repair.

Should we expect identical model replies?

No. Compare the permission outcome and evidence, not exact phrasing. Model replies can vary while the authorization boundary remains unchanged.

Does a passing retest cover future releases?

No. It supports the tested build and data. Changes to tools, permissions, models or application behavior are reasons to rerun relevant checks and review coverage.

What the result does—and does not—mean

A retest verifies covered behavior under the new recorded conditions. It does not automatically extend coverage to untested workflows or certify the whole system. Preserve legitimate passes, investigate every new failure and keep the scope of the evidence explicit.

Explore the testing cycle

Discuss your permission boundary

Permission rules Evidence reports