wip: qa7920 sidecar + rag stream timeout

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/superpowers/plans/2026-08-20-5090-rag-stream-timeout.md

@@ -0,0 +1,232 @@
+# 5090 RAG Stream Timeout Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make PlantAI RAG requests on 5090 complete through the full 7920 Qdrant + Neo4j + Agentic pipeline without fixed 120/150-second client aborts.
+
+**Architecture:** The browser uses the existing SSE endpoint and treats timeout as inactivity rather than total elapsed time. PlantAI proxies 7920 keepalives unchanged, while the Python adapter uses separate connect, stream-idle, and non-stream compatibility timeouts. Retrieval and answer-generation behavior remain unchanged.
+
+**Tech Stack:** FastAPI, `requests`, browser Fetch/ReadableStream, SSE, pytest.
+
+## Global Constraints
+
+- Preserve complete Qdrant + Neo4j retrieval; no Qdrant-only fallback.
+- Do not change Agent prompts, rounds, evidence selection, or answer composition.
+- Stream connect timeout: 10 seconds.
+- Stream idle timeout: configurable, default 45 seconds.
+- Non-stream compatibility timeout: configurable, default 900 seconds.
+- Do not commit or push; the user will handle Git later.
+
+---
+
+### Task 1: Python timeout policy
+
+**Files:**
+- Modify: `gfa_core/mas/dual_kb_qa.py`
+- Modify: `tests/test_dual_kb_qa_timeout.py`
+
+**Interfaces:**
+- Consumes: `QA_V1_1_73_TIMEOUT`, new `QA_V1_1_73_STREAM_IDLE_TIMEOUT` environment variables.
+- Produces: `_non_stream_timeout(timeout: float) -> tuple[float, float]` and `_stream_timeout() -> tuple[float, float]` policies used by both adapter entry points.
+
+- [ ] **Step 1: Replace the hard-cap assertions with failing policy tests**
+
+```python
+def test_non_stream_defaults_to_900_second_read_timeout(monkeypatch):
+    _force_v1173(monkeypatch)
+    # capture requests.post timeout
+    assert captured == [(10.0, 900.0)]
+
+def test_non_stream_honors_timeout_above_150(monkeypatch):
+    monkeypatch.setenv("QA_V1_1_73_TIMEOUT", "600")
+    assert captured == [(10.0, 600.0)]
+
+def test_stream_uses_45_second_idle_read_timeout(monkeypatch):
+    assert captured == [(10.0, 45.0)]
+```
+
+- [ ] **Step 2: Run the focused tests and verify RED**
+
+Run: `pytest -q tests/test_dual_kb_qa_timeout.py`
+
+Expected: failures showing the current 150-second cap.
+
+- [ ] **Step 3: Implement separate timeout policies**
+
+```python
+QA_CONNECT_TIMEOUT_SECONDS = 10.0
+QA_V1_1_73_DEFAULT_TIMEOUT_SECONDS = 900.0
+QA_V1_1_73_DEFAULT_STREAM_IDLE_TIMEOUT_SECONDS = 45.0
+
+def _positive_timeout(value: object, default: float) -> float:
+    try:
+        parsed = float(value)
+    except (TypeError, ValueError):
+        return default
+    return parsed if parsed > 0 else default
+```
+
+Use `QA_V1_1_73_TIMEOUT` for non-stream reads and `QA_V1_1_73_STREAM_IDLE_TIMEOUT` for stream idle reads. Remove `min(..., 150)`.
+
+- [ ] **Step 4: Run focused tests and verify GREEN**
+
+Run: `pytest -q tests/test_dual_kb_qa_timeout.py`
+
+Expected: all tests pass.
+
+### Task 2: Browser SSE inactivity timer and RAG route
+
+**Files:**
+- Modify: `apps/web/vue3/index.html`
+- Modify: `apps/web/vue3/index_ml.html`
+- Create: `tests/unit/test_rag_stream_frontend_contract.py`
+
+**Interfaces:**
+- Consumes: `/api/agent/rag/stream` SSE frames and comment keepalives.
+- Produces: `readRagEventStream(url, options, idleTimeoutMs)` that resets its timer on every received chunk; RAG mode calls `requestRagWithProgress`.
+
+- [ ] **Step 1: Add failing source-contract tests for both page variants**
+
+```python
+@pytest.mark.parametrize("name", ["index.html", "index_ml.html"])
+def test_rag_mode_uses_streaming_request(name):
+    source = (FRONTEND_DIR / name).read_text(encoding="utf-8")
+    rag_block = source[source.index("// V2: RAG模式"):source.index("} catch (err)", source.index("// V2: RAG模式"))]
+    assert "requestRagWithProgress(" in rag_block
+    assert "fetchJsonWithTimeout('/api/agent/rag'" not in rag_block
+
+@pytest.mark.parametrize("name", ["index.html", "index_ml.html"])
+def test_stream_timer_is_reset_after_each_chunk(name):
+    source = (FRONTEND_DIR / name).read_text(encoding="utf-8")
+    assert "resetIdleTimer();" in source
+    assert "const { value, done } = await reader.read();" in source
+```
+
+- [ ] **Step 2: Run the frontend contract test and verify RED**
+
+Run: `pytest -q tests/unit/test_rag_stream_frontend_contract.py`
+
+Expected: both page variants fail because RAG mode uses the non-stream endpoint and the timer is total-duration based.
+
+- [ ] **Step 3: Implement the inactivity timer in both page variants**
+
+Define `resetIdleTimer()` to clear and re-arm the abort timer. Call it before fetch and immediately after every successful `reader.read()` that returns a non-empty chunk. Keep `AbortController` for genuine inactivity.
+
+- [ ] **Step 4: Route both RAG-mode handlers through SSE**
+
+Replace the direct `fetchJsonWithTimeout('/api/agent/rag', ..., 120000)` call with:
+
+```javascript
+const data = await requestRagWithProgress({
+    query: userInput,
+    mode: 'rag',
+    top_k: 10
+}, 45000);
+```
+
+- [ ] **Step 5: Run frontend contract tests and verify GREEN**
+
+Run: `pytest -q tests/unit/test_rag_stream_frontend_contract.py`
+
+Expected: all tests pass.
+
+### Task 3: PlantAI API proxy contract
+
+**Files:**
+- Modify: `apps/web/vue3/main.py`
+- Modify: `tests/test_dual_kb_qa_timeout.py` or create `tests/unit/test_rag_stream_proxy_contract.py`
+
+**Interfaces:**
+- Consumes: `stream_dual_kb_qa(..., idle_timeout=45)`.
+- Produces: SSE response with unchanged keepalive/progress/complete/error frames.
+
+- [ ] **Step 1: Add a failing contract test**
+
+```python
+from pathlib import Path
+
+
+ROOT = Path(__file__).resolve().parents[2]
+MAIN = ROOT / "apps/web/vue3/main.py"
+
+
+def test_rag_proxy_has_no_inline_150_second_timeout():
+    source = MAIN.read_text(encoding="utf-8")
+    start = source.index('@app.post("/api/agent/rag")')
+    end = source.index("# ==================== DiagnosisAgent API", start)
+    rag_routes = source[start:end]
+    assert "timeout=150.0" not in rag_routes
+    assert "complete QA exceeded 150 seconds" not in rag_routes
+```
+
+- [ ] **Step 2: Run the proxy contract test and verify RED**
+
+Run: `pytest -q tests/unit/test_rag_stream_proxy_contract.py`
+
+Expected: failure on the current `timeout=150.0` literals.
+
+- [ ] **Step 3: Remove inline fixed timeout literals**
+
+Let `stream_dual_kb_qa` select the 45-second idle timeout and `query_dual_kb_qa` select the 900-second compatibility timeout. Preserve structured timeout and connection error responses.
+
+- [ ] **Step 4: Run proxy and adapter tests and verify GREEN**
+
+Run: `pytest -q tests/unit/test_rag_stream_proxy_contract.py tests/test_dual_kb_qa_timeout.py`
+
+Expected: all tests pass.
+
+### Task 4: Regression and 5090 deployment validation
+
+**Files:**
+- No production source additions.
+- Runtime candidate service uses the mounted `sj/PlantAI-dev` tree.
+
+**Interfaces:**
+- Consumes: 5090 port 7920 and candidate PlantAI port 9010.
+- Produces: health evidence, SSE keepalive evidence, final answer with graph chains/citations.
+
+- [ ] **Step 1: Run focused and adjacent regression tests**
+
+Run: `pytest -q tests/test_dual_kb_qa_timeout.py tests/unit/test_rag_stream_frontend_contract.py tests/unit/test_rag_stream_proxy_contract.py`
+
+Expected: all pass.
+
+- [ ] **Step 2: Validate syntax**
+
+Run: `python -m py_compile gfa_core/mas/dual_kb_qa.py apps/web/vue3/main.py`
+
+Expected: exit code 0.
+
+- [ ] **Step 3: Start/restart only the candidate 9010 service on 5090**
+
+Synchronize the mounted source into the explicit candidate directory `/home/userlab/plantAI/sj/PlantAI-dev-rag-stream-20260820` on 5090, excluding `.git`, caches, results, and virtual environments. Start `apps/web/vue3/main.py` on port 9010 with:
+
+```bash
+GENE_FAMILY_MAS_API=http://127.0.0.1:7920 \
+DUAL_KB_QA_ROUTE=qa-v1-1-73 \
+QA_V1_1_73_TIMEOUT=900 \
+QA_V1_1_73_STREAM_IDLE_TIMEOUT=45 \
+python apps/web/vue3/main.py --host 0.0.0.0 --port 9010
+```
+
+Do not restart or replace 7920.
+
+- [ ] **Step 4: Verify health and SSE keepalive path**
+
+Run:
+
+```bash
+curl -N --max-time 60 -H 'Content-Type: application/json' \
+  -d '{"query":"What biological processes are regulated by the NAC transcription factor family in plants?","mode":"rag","top_k":10}' \
+  http://127.0.0.1:9010/api/agent/rag/stream
+```
+
+Expected: an SSE progress frame or `: keepalive` is received before curl reaches 60 seconds; no 504 frame is returned.
+
+- [ ] **Step 5: Verify a real long question end to end**
+
+Run the same curl without a 60-second cap and save the SSE response under `/tmp/plantai-9010-rag-stream-validation.sse`. Success requires an `event: complete` frame whose JSON has a non-empty answer, citations, and `native_graph_chains` or `graph_knowledge_chains`. Record curl's `time_total`; a value greater than 150 seconds is acceptable when earlier progress/keepalive frames are present.
+
+- [ ] **Step 6: Inspect process and Neo4j state**
+
+On 5090, check `ps -ef` for only the intended 9010 candidate process and query Neo4j `SHOW TRANSACTIONS` through the existing Neo4j container/CLI. Success requires no transaction from the validation request remaining more than 10 seconds after the complete event.

docs/superpowers/plans/2026-08-20-qa7920-formal-integration.md

@@ -0,0 +1,21 @@
+# QA 7920 Formal Integration Implementation Plan
+
+**Goal:** Make the validated full-graph QA service and PlantAI timeout behavior versioned in this repository.
+
+## Constraints
+
+- Work only in this repository for persistent source changes.
+- Preserve Qdrant + Neo4j and `QA_V1_BOUNDED_NEO4J_TIMEOUT=8`.
+- Apply a 10-second connect timeout and 150-second complete-answer timeout.
+- Validate on a sidecar before replacing 7920.
+
+## Tasks
+
+1. Add `tests/test_qa7920_formal_deployment.py` and verify it fails because the formal package and startup entrypoint do not exist.
+2. Vendor the validated QA package under `services/qa_v1_1_73/src/rag/qa_v1_1_73/`, its three direct support modules, and two YAML configs.
+3. Add `scripts/qa_v1_1_73_server.py` and `scripts/start_qa_v1_1_73_7920.sh`; make the deployment test pass and compile all Python sources.
+4. Add `tests/test_dual_kb_qa_timeout.py`; verify it fails against the current 300-second/scalar timeout behavior.
+5. Change `gfa_core/mas/dual_kb_qa.py` to `timeout=(10.0, 150.0)` and change `apps/web/vue3/main.py` to emit HTTP 504 for non-stream timeout and use 150 seconds for streaming.
+6. Run focused tests and compilation, then start a sidecar with the existing full backends.
+7. Require a correct `Solanum lycopersicum` answer, non-empty native graph chains, and wall time below 150 seconds before switching 7920.
+8. Restart PlantAI through its controlled endpoint and repeat the same assertion through 9010. Roll back runtime processes on any failed assertion.

docs/superpowers/specs/2026-08-20-5090-rag-stream-timeout-design.md

@@ -0,0 +1,44 @@
+# 5090 RAG 流式超时修订设计
+
+## 目标
+
+让 5090 上的 PlantAI 问答完整使用 7920 的 Qdrant + Neo4j + Agentic QA。复杂问题即使总耗时超过 150 秒，只要 7920 仍持续发送进度或 keepalive，就不能被 PlantAI 客户端误判为超时。此次不减少检索步骤、不跳过辅助问题、不降级为 Qdrant-only。
+
+## 已确认根因
+
+- 7920 的 `/api/v1/qa-v1-1-73/chat/stream` 每 15 秒发送 SSE keepalive。
+- 页面已经定义流式 RAG 客户端，但实际 RAG 模式仍调用旧的 `/api/agent/rag`，并在 120 秒主动中止。
+- PlantAI 的非流式代理又显式传入 150 秒；`gfa_core/mas/dual_kb_qa.py` 还会将 V1.1.73 的超时限制为最多 150 秒，并忽略更高的环境配置。
+- 普通 `/chat` 调用在 150 秒时断开后，7920 的同步后台计算仍会继续，造成用户收到超时但服务端任务继续占用资源。
+- 5090 当前运行 7920 和旧 Web 9017；指定的 `sj/PlantAI-dev` 尚未作为 9010 运行。
+
+## 设计
+
+1. 两套页面入口的 RAG 模式都改用已有的 `requestRagWithProgress`，由 PlantAI `/api/agent/rag/stream` 转发 7920 `/chat/stream`。
+2. 浏览器流读取从“固定总时长计时器”改为“读取空闲计时器”；每次收到 progress、keepalive 或其他数据块就重置计时器。
+3. 移除 120/150 秒总时长上限。客户端只保留：
+   - 10 秒连接建立超时；
+   - 可配置的读取空闲超时，默认 45 秒。
+4. 7920 每 15 秒的 keepalive 会重置读取空闲计时，因此正常长请求可以一直运行；如果 7920、网络或代理真正失联，客户端仍会在空闲超时后返回明确错误。
+5. 非流式兼容接口不再硬编码 150 秒，改用独立的可配置总超时，默认 900 秒。交互式前端不得走该接口。
+6. 保持完整图谱检索和现有答案生成逻辑，不调整 Agent 提示词、轮次和证据裁剪策略。
+
+## 错误处理
+
+- 连接失败返回 `service_unavailable`。
+- 流在超过空闲超时后无任何字节返回 `request_timeout`。
+- 收到 keepalive 但总执行超过 150 秒不报错。
+- 上游返回结构化错误时原样保留错误码和可读信息。
+
+## 验证
+
+- 单元测试证明环境变量高于 150 秒时不再被截断。
+- 前端契约测试证明 RAG 模式使用流式入口，且每个数据块会重置空闲计时器。
+- 代理测试模拟持续 keepalive 后最终成功。
+- 单元测试模拟真正的读取空闲超时并验证 504 错误。
+- 在 5090 启动指定 `PlantAI-dev` 的候选服务，直连 7920 执行健康检查和真实长问题。
+- 验证答案仍包含 Neo4j 图链和引用；不以 Qdrant-only 返回作为成功。
+
+## 回退
+
+不改动 7920 当前服务。PlantAI 代码修改可通过恢复相关文件或停止候选 9010 服务回退；不 push，由用户后续统一处理 Git。

docs/superpowers/specs/2026-08-20-qa7920-formal-integration-design.md

@@ -0,0 +1,17 @@
+# QA 7920 正式接入设计
+
+## 目标
+
+将完成 200 题评估的 QA V1.1.73 源码和启动入口纳入本 Git 仓库，由本机 `127.0.0.1:7920` 为 PlantAI 提供完整 Qdrant + Neo4j 问答，并将整次请求的异常边界设为 150 秒。
+
+## 设计
+
+- 将已验证 QA 包放入隔离的 `services/qa_v1_1_73/src/rag/qa_v1_1_73/`，并纳入它直接依赖的 cascaded graph retrieval、relation taxonomy 和 entity alias 模块，避免遮蔽 PlantAI 已使用的外部 `src.rag`。
+- 配置文件放入 `configs/rag/`；大模型、Qdrant、Neo4j 和论文元数据仍由环境变量指向现有运行资源，不提交数据或密钥。
+- 正式入口为 `scripts/qa_v1_1_73_server.py`，正式启动脚本为 `scripts/start_qa_v1_1_73_7920.sh`；运行命令和环境中不得出现 `/sandboxes/`。
+- PlantAI 保持调用同机 `127.0.0.1:7920`。连接超时为 10 秒，完整回答读取超时为 150 秒；超时返回 HTTP 504 或 SSE `request_timeout`。
+- `QA_V1_BOUNDED_NEO4J_TIMEOUT` 保持 8 秒，不切换 Qdrant-only，不修改图谱或索引。
+
+## 验证与回退
+
+先用契约测试证明仓库缺少正式 QA 服务，再加入源码使测试通过。在旁路端口验证健康信息、完整后端身份和历史测试题；只有在 150 秒内正确返回且图谱链非空时才替换 7920。失败时恢复旧 PID，不切换 9010。