WIP: P2 support: fixed-version existing-product evidence #54

2026-09-09T19:17:30+08:00

CoderLambert commented

2026-09-09 19:17:30 +08:00

(Migrated from github.com)

Support task

P2-T11 fixed-version existing-product evidence harness

Status: PASS — SUPPORT EVIDENCE COMPLETE FOR CURRENT DECISION

What this support branch established

fixed product identities/contracts for AnythingLLM v1.16.1 and Open WebUI v0.11.3;
CI headless/smoke coverage;
independent AnythingLLM history probe;
reproducible AnythingLLM 50-query local runner;
completed AnythingLLM formal run with raw evidence preserved.

AnythingLLM formal run:

50 records
47 success
3 first-attempt runtime/API failures
dev-011 / dev-040 / dev-041
JSONL SHA-256 d62197426809c28636c8d04d609a901c613237c9be1a448364775c487c4b1cb9

Independent semantic adjudication:

41 correct
2 partial
4 incorrect
3 execution failure
39/39 successful answerable responses correct

Historical citation path: PASS / A — old ORCHID citation reopened Version A after same-name COBALT replacement and restart.

Open WebUI scope change

A fixed-version long run was started locally, but completion is intentionally de-scoped. Partial diagnostics are not scored, retried or tuned. A full Open WebUI benchmark is no longer required for P2-T11 because the architecture decision now separates Pi-owned canonical Knowledge semantics from replaceable retrieval/RAG infrastructure.

Targeted Open WebUI probes may be reopened later only for a concrete external-provider contract question.

Scope

Base: research/p2-existing-product-comparison (#43)
Head: chore/p2-t11-product-evidence-harness
Support-only draft PR
Existing five-file direct-base scope
No production Knowledge code
No holdout rerun
No P3 implementation in this support PR
No automatic merge

The harness is closed for the current architecture decision. ADR-029 was subsequently Accepted in PR #44; this support branch does not implement that architecture.

## Support task P2-T11 fixed-version existing-product evidence harness Status: **PASS — SUPPORT EVIDENCE COMPLETE FOR CURRENT DECISION** ## What this support branch established - fixed product identities/contracts for AnythingLLM v1.16.1 and Open WebUI v0.11.3; - CI headless/smoke coverage; - independent AnythingLLM history probe; - reproducible AnythingLLM 50-query local runner; - completed AnythingLLM formal run with raw evidence preserved. AnythingLLM formal run: ```text 50 records 47 success 3 first-attempt runtime/API failures dev-011 / dev-040 / dev-041 JSONL SHA-256 d62197426809c28636c8d04d609a901c613237c9be1a448364775c487c4b1cb9 ``` Independent semantic adjudication: ```text 41 correct 2 partial 4 incorrect 3 execution failure 39/39 successful answerable responses correct ``` Historical citation path: **PASS / A** — old ORCHID citation reopened Version A after same-name COBALT replacement and restart. ## Open WebUI scope change A fixed-version long run was started locally, but completion is intentionally de-scoped. Partial diagnostics are not scored, retried or tuned. A full Open WebUI benchmark is no longer required for P2-T11 because the architecture decision now separates Pi-owned canonical Knowledge semantics from replaceable retrieval/RAG infrastructure. Targeted Open WebUI probes may be reopened later only for a concrete external-provider contract question. ## Scope - Base: `research/p2-existing-product-comparison` (#43) - Head: `chore/p2-t11-product-evidence-harness` - Support-only draft PR - Existing five-file direct-base scope - No production Knowledge code - No holdout rerun - No P3 implementation in this support PR - No automatic merge The harness is closed for the current architecture decision. ADR-029 was subsequently Accepted in PR #44; this support branch does not implement that architecture.

You are not authorized to merge this pull request.

This pull request can be merged automatically.

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin chore/p2-t11-product-evidence-harness:chore/p2-t11-product-evidence-harness

git checkout chore/p2-t11-product-evidence-harness

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lambert/pi-knowledge-workspace#54