forked from admin/PlantAI-rag-new
feat(qa-db): QA数据库裸进程标准(deploy/qa-db五脚本+守卫去版本锁定+打包/文档/Excel同步)
- deploy/qa-db/: qdrant/neo4j systemd 裸进程安装脚本+unit模板+整库替换(replace_db.sh) * 数据双制品模型: <部署根>/qa-db 与 install 分开传输, 支持频繁整库替换 * 免编译(官方静态二进制/tarball)+本地 bin/ 离线分发+国内镜像回退 * 端口 27433/27434+19888/19885(127.0.0.1), 175 docker 容器(27333/19788)为其他项目共享不动 - backend_guard.py: 移除精确计数相等校验(数据库会整库替换/持续更新, 不得锁死版本) 保留身份校验(release/collection/kb_version/标签/关系类型), 计数仅随 health 返回 - qa/scripts/start.sh: reranker 模型路径回退(install/models/bge-reranker-base) - build_all.sh: qa-db 脚本+reranker 模型入包; .gitignore: qa-db/bin 离线载体 - DEPLOY_FLOW.md: §0/§5.0/§5.0b/§6.0b/§7 QA 数据库裸进程标准 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -18,3 +18,6 @@ coverage/
|
||||
tooling/build/
|
||||
bioflow_data
|
||||
bioflow_service/bioflow.log
|
||||
|
||||
# QA 数据库裸进程离线二进制载体(构建机从镜像/官网提取,不入 git)
|
||||
deploy/qa-db/bin/
|
||||
|
||||
@@ -23,6 +23,11 @@ set -a
|
||||
. "$ENV_FILE"
|
||||
set +a
|
||||
|
||||
# reranker 模型路径回退:env 未指定时用安装包内模型(install/models/bge-reranker-base)
|
||||
if [[ -z "${QA_V1_BGE_RERANKER_MODEL:-}" && -d "$AGENT_ROOT/../models/bge-reranker-base" ]]; then
|
||||
export QA_V1_BGE_RERANKER_MODEL="$AGENT_ROOT/../models/bge-reranker-base"
|
||||
fi
|
||||
|
||||
cd "$AGENT_ROOT"
|
||||
setsid "$PYTHON" -m uvicorn scripts.qa_v1_1_73_server:app \
|
||||
--host 127.0.0.1 --port "$PORT" \
|
||||
|
||||
@@ -42,15 +42,8 @@ FORBIDDEN_RELATIONSHIP_TYPES = {
|
||||
"SRT2_ASSERTION_SOURCE",
|
||||
"SRT2_ASSERTION_TARGET",
|
||||
}
|
||||
EXPECTED_NEO4J_PAPERS = 31_207
|
||||
EXPECTED_NEO4J_CANONICAL_ENTITIES = 3_820_996
|
||||
EXPECTED_NEO4J_CANONICAL_ASSERTIONS = 8_246_875
|
||||
EXPECTED_NEO4J_CANONICAL_ASSERTS = 8_246_875
|
||||
EXPECTED_NEO4J_SUPPORTED_BY = 8_202_284
|
||||
EXPECTED_NORMALIZED_ENTITIES = 6_316_654
|
||||
EXPECTED_NORMALIZED_PAPERS = 96_648
|
||||
EXPECTED_NORMALIZED_ASSERTS = 12_963_472
|
||||
EXPECTED_NORMALIZED_SUPPORTED_BY = 13_012_368
|
||||
# 2026-09-08 起不再固化精确计数(数据库会频繁整库替换/更新,见 DEPLOY_FLOW.md §5.0b):
|
||||
# 身份校验只看 release/collection/kb_version/标签/关系类型;实际计数随 /health 信息返回。
|
||||
|
||||
|
||||
class BackendIdentityError(RuntimeError):
|
||||
@@ -97,11 +90,7 @@ def _validate_normalized_backends() -> Dict[str, Any]:
|
||||
f"{qdrant_url}/collections/{collection}"
|
||||
).get("result", {})
|
||||
qdrant_points = int(qdrant_info.get("points_count") or 0)
|
||||
if qdrant_points != EXPECTED_NORMALIZED_ASSERTS:
|
||||
raise BackendIdentityError(
|
||||
"normalized Qdrant identity mismatch: "
|
||||
f"expected {EXPECTED_NORMALIZED_ASSERTS} points, got {qdrant_points}"
|
||||
)
|
||||
# 计数不再参与校验:数据库可整库替换/持续更新(2026-09-08 决策)
|
||||
sample_payload = _qdrant_sample(qdrant_url, collection)
|
||||
if sample_payload.get("kb_version") != NORMALIZED_RELEASE:
|
||||
raise BackendIdentityError(
|
||||
@@ -167,16 +156,7 @@ def _validate_normalized_backends() -> Dict[str, Any]:
|
||||
asserts_count,
|
||||
supported_by_count,
|
||||
)
|
||||
expected = (
|
||||
EXPECTED_NORMALIZED_ENTITIES,
|
||||
EXPECTED_NORMALIZED_PAPERS,
|
||||
EXPECTED_NORMALIZED_ASSERTS,
|
||||
EXPECTED_NORMALIZED_SUPPORTED_BY,
|
||||
)
|
||||
if counts != expected:
|
||||
raise BackendIdentityError(
|
||||
f"normalized Neo4j counts mismatch: expected={expected}, got={counts}"
|
||||
)
|
||||
# 计数不再参与校验:数据库可整库替换/持续更新(2026-09-08 决策)
|
||||
return {
|
||||
"backend_policy": "neo4j_qdrant_only",
|
||||
"release_id": release_id,
|
||||
@@ -217,10 +197,7 @@ def validate_backends() -> Dict[str, Any]:
|
||||
"result", {}
|
||||
)
|
||||
qdrant_points = int(qdrant_info.get("points_count") or 0)
|
||||
if qdrant_points != 8_246_777:
|
||||
raise BackendIdentityError(
|
||||
f"Qdrant identity mismatch: expected 8246777 points, got {qdrant_points}"
|
||||
)
|
||||
# 计数不再参与校验:数据库可整库替换/持续更新(2026-09-08 决策)
|
||||
sample_payload = _qdrant_sample(qdrant_url, collection)
|
||||
if sample_payload.get("source_system") != "V15_V16_canonical":
|
||||
raise BackendIdentityError(
|
||||
@@ -296,20 +273,7 @@ def validate_backends() -> Dict[str, Any]:
|
||||
canonical_assertions = int(counts["canonical_assertions"])
|
||||
canonical_asserts = relationship_counts.get("SRT2_CANONICAL_ASSERTS", 0)
|
||||
supported_by = relationship_counts.get("SRT2_SUPPORTED_BY", 0)
|
||||
expected_identity = (
|
||||
papers == EXPECTED_NEO4J_PAPERS
|
||||
and canonical_entities == EXPECTED_NEO4J_CANONICAL_ENTITIES
|
||||
and canonical_assertions == EXPECTED_NEO4J_CANONICAL_ASSERTIONS
|
||||
and canonical_asserts == EXPECTED_NEO4J_CANONICAL_ASSERTS
|
||||
and supported_by == EXPECTED_NEO4J_SUPPORTED_BY
|
||||
)
|
||||
if not expected_identity:
|
||||
raise BackendIdentityError(
|
||||
"Neo4j store does not match the July 2 direct canonical identity: "
|
||||
f"papers={papers}, canonical_entities={canonical_entities}, "
|
||||
f"canonical_assertions={canonical_assertions}, "
|
||||
f"canonical_asserts={canonical_asserts}, supported_by={supported_by}"
|
||||
)
|
||||
# 计数不再参与校验:数据库可整库替换/持续更新(2026-09-08 决策)
|
||||
|
||||
return {
|
||||
"backend_policy": "neo4j_qdrant_only",
|
||||
|
||||
Executable
+108
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy/qa-db/install_neo4j.sh —— PlantAI QA 数据库裸进程标准:Neo4j Community
|
||||
# 用法(需 root):
|
||||
# sudo -S bash install_neo4j.sh [QA_DB_ROOT] [BOLT_PORT] [HTTP_PORT]
|
||||
# QA_DB_ROOT 默认 <安装根>/qa-db(release 树中即 install 的上一级;仓库树中即仓库根)
|
||||
# BOLT_PORT 默认 19888;HTTP_PORT 默认 19885
|
||||
# (与 docker 共享实例 19788/19575 互不冲突,175 上容器为其他项目服务、不动)
|
||||
# 产物:/opt/neo4j-community-5.15.0 + 软链 /opt/neo4j-plantai、systemd 单元 neo4j-plantai.service
|
||||
# 数据目录:$QA_DB_ROOT/neo4j/data(conf 的 server.directories.data 指向它;整库替换见 replace_db.sh)
|
||||
# 版本与 2026-08 容器实例 neo4j:5.15.0-community 对齐;官方 tarball,免编译;需 JDK17(宿主已有则用)
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
INSTALL_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
QA_DB_ROOT="${1:-${QA_DB_ROOT:-$INSTALL_ROOT/qa-db}}"
|
||||
BOLT_PORT="${2:-19888}"
|
||||
HTTP_PORT="${3:-19885}"
|
||||
NEO4J_VERSION="${NEO4J_VERSION:-5.15.0}"
|
||||
RUN_USER="${RUN_USER:-${SUDO_USER:-xzq}}"
|
||||
NEO4J_HOME="/opt/neo4j-community-$NEO4J_VERSION"
|
||||
NEO4J_LINK=/opt/neo4j-plantai
|
||||
UNIT=/etc/systemd/system/neo4j-plantai.service
|
||||
TPL="$SCRIPT_DIR/neo4j.service"
|
||||
DATA="$QA_DB_ROOT/neo4j/data"
|
||||
LOGS="$QA_DB_ROOT/neo4j/logs"
|
||||
|
||||
[[ $EUID -eq 0 ]] || { echo "需要 root:sudo -S bash $0 $*" >&2; exit 1; }
|
||||
command -v curl >/dev/null || { echo "缺少 curl(apt install -y curl)" >&2; exit 1; }
|
||||
command -v java >/dev/null || { echo "缺少 JDK17(apt install -y openjdk-17-jre-headless)" >&2; exit 1; }
|
||||
id -u "$RUN_USER" >/dev/null 2>&1 || { echo "运行用户不存在:$RUN_USER" >&2; exit 1; }
|
||||
|
||||
# 1) 发行包(版本已符合则跳过;官方 tarball,免编译)
|
||||
if [[ -x "$NEO4J_HOME/bin/neo4j" ]]; then
|
||||
echo "[neo4j] 发行包已就绪:$NEO4J_HOME"
|
||||
else
|
||||
# 优先使用随包携带的本地发行包(离线/受限网络部署时从构建机 scp 到此目录)
|
||||
LOCAL_TGZ="$SCRIPT_DIR/bin/neo4j-community-$NEO4J_VERSION-unix.tar.gz"
|
||||
if [[ -f "$LOCAL_TGZ" ]]; then
|
||||
echo "[neo4j] 使用本地发行包:$LOCAL_TGZ"
|
||||
mkdir -p "$NEO4J_HOME"
|
||||
tar -xzf "$LOCAL_TGZ" -C /tmp
|
||||
cp -a "/tmp/neo4j-community-$NEO4J_VERSION/." "$NEO4J_HOME/"
|
||||
rm -rf "/tmp/neo4j-community-$NEO4J_VERSION"
|
||||
else
|
||||
echo "[neo4j] 下载 neo4j-community-$NEO4J_VERSION ..."
|
||||
ok=0
|
||||
for u in \
|
||||
"https://dist.neo4j.org/neo4j-community-$NEO4J_VERSION-unix.tar.gz" \
|
||||
"https://github.com/neo4j/neo4j/releases/download/$NEO4J_VERSION/neo4j-community-$NEO4J_VERSION-unix.tar.gz" \
|
||||
"https://gh-proxy.com/https://github.com/neo4j/neo4j/releases/download/$NEO4J_VERSION/neo4j-community-$NEO4J_VERSION-unix.tar.gz" \
|
||||
"https://ghfast.top/https://github.com/neo4j/neo4j/releases/download/$NEO4J_VERSION/neo4j-community-$NEO4J_VERSION-unix.tar.gz"; do
|
||||
if curl -fL -A "Mozilla/5.0" --connect-timeout 10 --max-time 1800 -o /tmp/neo4j.tar.gz "$u" 2>/dev/null; then
|
||||
ok=1; echo "[neo4j] 下载成功:$u"; break
|
||||
fi
|
||||
echo "[neo4j] 下载失败,换下一源:$u"
|
||||
done
|
||||
if [[ $ok -eq 0 ]] && command -v docker >/dev/null 2>&1 && docker image inspect "neo4j:$NEO4J_VERSION-community" >/dev/null 2>&1; then
|
||||
echo "[neo4j] 回退:从本地 docker 镜像提取发行包(只读,不动运行中的容器)"
|
||||
CID=$(docker create "neo4j:$NEO4J_VERSION-community")
|
||||
docker cp "$CID:/var/lib/neo4j" "$NEO4J_HOME"
|
||||
docker rm "$CID" >/dev/null
|
||||
ok=1
|
||||
fi
|
||||
[[ $ok -eq 1 ]] || { echo "所有下载源失败,请手动放置发行包到 $NEO4J_HOME 或 $LOCAL_TGZ" >&2; exit 1; }
|
||||
if [[ -f /tmp/neo4j.tar.gz ]]; then
|
||||
mkdir -p "$NEO4J_HOME"
|
||||
tar -xzf /tmp/neo4j.tar.gz -C /tmp
|
||||
cp -a "/tmp/neo4j-community-$NEO4J_VERSION/." "$NEO4J_HOME/"
|
||||
rm -rf /tmp/neo4j.tar.gz "/tmp/neo4j-community-$NEO4J_VERSION"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
ln -sfn "$NEO4J_HOME" "$NEO4J_LINK"
|
||||
|
||||
# 2) 数据/日志目录(整库替换见 replace_db.sh;目录属主对齐运行用户)
|
||||
mkdir -p "$DATA" "$LOGS"
|
||||
chown -R "$RUN_USER:$RUN_USER" "$QA_DB_ROOT/neo4j"
|
||||
|
||||
# 3) neo4j.conf(追加覆盖项;同名键后值生效)
|
||||
CONF="$NEO4J_HOME/conf/neo4j.conf"
|
||||
cat >> "$CONF" <<EOF
|
||||
|
||||
# ==== PlantAI-rag-new 裸进程覆盖项(deploy/qa-db/install_neo4j.sh 生成)====
|
||||
server.bolt.listen_address=127.0.0.1:$BOLT_PORT
|
||||
server.http.listen_address=127.0.0.1:$HTTP_PORT
|
||||
server.directories.data=$DATA
|
||||
server.directories.logs=$LOGS
|
||||
dbms.security.auth_enabled=true
|
||||
server.memory.heap.initial_size=16G
|
||||
server.memory.heap.max_size=32G
|
||||
server.memory.pagecache.size=64G
|
||||
EOF
|
||||
|
||||
# 4) systemd 单元(模板替换占位符)
|
||||
sed -e "s|@NEO4J_HOME@|$NEO4J_HOME|g" -e "s|@RUN_USER@|$RUN_USER|g" "$TPL" > "$UNIT"
|
||||
# 5) 幂等启动 + 探活(bolt 端口)
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now neo4j-plantai.service >/dev/null 2>&1 || true
|
||||
for _ in $(seq 1 90); do
|
||||
if (exec 3<>"/dev/tcp/127.0.0.1/$BOLT_PORT") 2>/dev/null; then
|
||||
exec 3>&- 2>/dev/null || true
|
||||
echo "[neo4j] UP bolt://127.0.0.1:$BOLT_PORT (data: $DATA)"
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "[neo4j] 启动失败:journalctl -u neo4j-plantai -n 50" >&2
|
||||
exit 1
|
||||
Executable
+92
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy/qa-db/install_qdrant.sh —— PlantAI QA 数据库裸进程标准:Qdrant
|
||||
# 用法(需 root):
|
||||
# sudo -S bash install_qdrant.sh [QA_DB_ROOT] [HTTP_PORT] [GRPC_PORT]
|
||||
# QA_DB_ROOT 默认 <安装根>/qa-db(release 树中即 install 的上一级;仓库树中即仓库根)
|
||||
# HTTP_PORT 默认 27433;GRPC_PORT 默认 27434
|
||||
# (与 docker 共享实例 27333/27334 互不冲突,175 上容器为其他项目服务、不动)
|
||||
# 产物:/usr/local/bin/qdrant、/etc/qdrant/config.yaml、systemd 单元 qdrant-plantai.service
|
||||
# 数据目录:$QA_DB_ROOT/qdrant/storage(整库替换见 replace_db.sh)
|
||||
# 版本默认 v1.18.3(与 2026-08 容器实例 qdrant/qdrant:latest 对齐)
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
INSTALL_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
QA_DB_ROOT="${1:-${QA_DB_ROOT:-$INSTALL_ROOT/qa-db}}"
|
||||
HTTP_PORT="${2:-27433}"
|
||||
GRPC_PORT="${3:-27434}"
|
||||
QDRANT_VERSION="${QDRANT_VERSION:-v1.18.3}"
|
||||
RUN_USER="${RUN_USER:-${SUDO_USER:-xzq}}"
|
||||
BIN=/usr/local/bin/qdrant
|
||||
CONF=/etc/qdrant/config.yaml
|
||||
UNIT=/etc/systemd/system/qdrant-plantai.service
|
||||
STORAGE="$QA_DB_ROOT/qdrant/storage"
|
||||
TPL="$SCRIPT_DIR/qdrant.service"
|
||||
|
||||
[[ $EUID -eq 0 ]] || { echo "需要 root:sudo -S bash $0 $*" >&2; exit 1; }
|
||||
command -v curl >/dev/null || { echo "缺少 curl(apt install -y curl)" >&2; exit 1; }
|
||||
id -u "$RUN_USER" >/dev/null 2>&1 || { echo "运行用户不存在:$RUN_USER" >&2; exit 1; }
|
||||
|
||||
# 1) 二进制(版本已符合则跳过;官方静态二进制,免编译)
|
||||
if [[ -x "$BIN" ]] && "$BIN" --version 2>/dev/null | grep -q "$QDRANT_VERSION"; then
|
||||
echo "[qdrant] 二进制已就绪:$("$BIN" --version 2>&1 | head -1)"
|
||||
else
|
||||
# 优先使用随包携带的本地二进制(离线/受限网络部署时从构建机 scp 到此目录)
|
||||
LOCAL_BIN="$SCRIPT_DIR/bin/qdrant"
|
||||
if [[ -x "$LOCAL_BIN" ]] && "$LOCAL_BIN" --version 2>/dev/null | grep -q "$QDRANT_VERSION"; then
|
||||
echo "[qdrant] 使用本地二进制:$LOCAL_BIN"
|
||||
install -m 0755 "$LOCAL_BIN" "$BIN"
|
||||
else
|
||||
echo "[qdrant] 下载 qdrant $QDRANT_VERSION ..."
|
||||
URLS=(
|
||||
"https://github.com/qdrant/qdrant/releases/download/$QDRANT_VERSION/qdrant-x86_64-unknown-linux-gnu.tar.gz"
|
||||
"https://gh-proxy.com/https://github.com/qdrant/qdrant/releases/download/$QDRANT_VERSION/qdrant-x86_64-unknown-linux-gnu.tar.gz"
|
||||
"https://ghfast.top/https://github.com/qdrant/qdrant/releases/download/$QDRANT_VERSION/qdrant-x86_64-unknown-linux-gnu.tar.gz"
|
||||
)
|
||||
ok=0
|
||||
for u in "${URLS[@]}"; do
|
||||
if curl -fL --connect-timeout 10 --max-time 300 -o /tmp/qdrant.tar.gz "$u" 2>/dev/null; then
|
||||
ok=1; echo "[qdrant] 下载成功:$u"; break
|
||||
fi
|
||||
echo "[qdrant] 下载失败,换下一源:$u"
|
||||
done
|
||||
[[ $ok -eq 1 ]] || { echo "所有下载源失败,请手动放置二进制到 $BIN 或 $LOCAL_BIN" >&2; exit 1; }
|
||||
tar -xzf /tmp/qdrant.tar.gz -C /tmp
|
||||
install -m 0755 /tmp/qdrant "$BIN"
|
||||
rm -f /tmp/qdrant.tar.gz /tmp/qdrant
|
||||
"$BIN" --version | head -1
|
||||
fi
|
||||
fi
|
||||
|
||||
# 2) 数据目录 + 配置(config 由本脚本参数生成,不手改)
|
||||
mkdir -p "$STORAGE" /etc/qdrant
|
||||
cat > "$CONF" <<EOF
|
||||
# PlantAI-rag-new Qdrant 裸进程配置(由 deploy/qa-db/install_qdrant.sh 生成,勿手改)
|
||||
storage:
|
||||
storage_path: $STORAGE
|
||||
snapshots_path: $STORAGE/snapshots
|
||||
service:
|
||||
host: 127.0.0.1
|
||||
http_port: $HTTP_PORT
|
||||
grpc_port: $GRPC_PORT
|
||||
enable_tls: false
|
||||
telemetry_disabled: true
|
||||
EOF
|
||||
# 数据目录属主对齐运行用户(快照恢复时文件由 qdrant 进程自己创建)
|
||||
chown -R "$RUN_USER:$RUN_USER" "$QA_DB_ROOT/qdrant"
|
||||
|
||||
# 3) systemd 单元(模板替换占位符)
|
||||
sed -e "s|@HTTP_PORT@|$HTTP_PORT|g" -e "s|@BIN@|$BIN|g" -e "s|@CONF@|$CONF|g" -e "s|@RUN_USER@|$RUN_USER|g" -e "s|@QA_DB_ROOT@|$QA_DB_ROOT|g" "$TPL" > "$UNIT"
|
||||
|
||||
# 4) 幂等启动 + 探活
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now qdrant-plantai.service >/dev/null 2>&1 || true
|
||||
for _ in $(seq 1 30); do
|
||||
if curl -sf --max-time 2 "http://127.0.0.1:$HTTP_PORT/collections" >/dev/null 2>&1; then
|
||||
echo "[qdrant] UP http://127.0.0.1:$HTTP_PORT (storage: $STORAGE)"
|
||||
exit 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "[qdrant] 启动失败:journalctl -u qdrant-plantai -n 50" >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=PlantAI Neo4j bare-metal instance (QA graph DB)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=@RUN_USER@
|
||||
WorkingDirectory=@NEO4J_HOME@
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
LimitNOFILE=60000
|
||||
ExecStart=@NEO4J_HOME@/bin/neo4j console
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=PlantAI Qdrant bare-metal instance (QA vector DB)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=@RUN_USER@
|
||||
WorkingDirectory=@QA_DB_ROOT@/qdrant
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
LimitNOFILE=1048576
|
||||
ExecStart=@BIN@ --config-path @CONF@
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Executable
+74
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy/qa-db/replace_db.sh —— PlantAI QA 裸进程数据库整库替换(用户高频操作)
|
||||
# 用法(需 root;数据目录换新后属主会重对齐,按提示执行即可):
|
||||
# sudo -S bash replace_db.sh qdrant <新storage目录> # 替换 Qdrant 数据
|
||||
# sudo -S bash replace_db.sh neo4j <新data目录> # 替换 Neo4j 数据
|
||||
# 行为:停 unit → 旧目录 mv 为 .bak.<时间戳> → 新目录就位+属主对齐 → 启 unit → 探活
|
||||
# → 通过后提示删除 .bak 与回滚命令。失败自动回滚。
|
||||
# 配置默认与 install_qdrant.sh / install_neo4j.sh 一致:
|
||||
# QA_DB_ROOT 默认 <安装根>/qa-db;Qdrant HTTP 27433;Neo4j bolt 19888
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
INSTALL_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
|
||||
QA_DB_ROOT="${QA_DB_ROOT:-$INSTALL_ROOT/qa-db}"
|
||||
HTTP_PORT="${QDRANT_HTTP_PORT:-27433}"
|
||||
BOLT_PORT="${NEO4J_BOLT_PORT:-19888}"
|
||||
KIND="${1:-}"
|
||||
NEW_DIR="${2:-}"
|
||||
|
||||
[[ $EUID -eq 0 ]] || { echo "需要 root:sudo -S bash $0 $*" >&2; exit 1; }
|
||||
case "$KIND" in
|
||||
qdrant) UNIT=qdrant-plantai; CUR="$QA_DB_ROOT/qdrant/storage"; OWNER="${SUDO_USER:-xzq}";;
|
||||
neo4j) UNIT=neo4j-plantai; CUR="$QA_DB_ROOT/neo4j/data"; OWNER="${SUDO_USER:-xzq}";;
|
||||
*) echo "用法: $0 qdrant|neo4j <新目录>"; exit 2;;
|
||||
esac
|
||||
[[ -n "$NEW_DIR" && -d "$NEW_DIR" ]] || { echo "新目录不存在:$NEW_DIR" >&2; exit 2; }
|
||||
systemctl is-active --quiet "$UNIT" || { echo "unit $UNIT 未运行,请先启动或检查"; exit 2; }
|
||||
|
||||
TS="$(date +%Y%m%dT%H%M%S)"
|
||||
BAK="${CUR}.bak.$TS"
|
||||
|
||||
echo "== 整库替换 $KIND =="
|
||||
echo " 旧数据:$CUR"
|
||||
echo " 新数据:$NEW_DIR"
|
||||
echo " 备份为:$BAK"
|
||||
systemctl stop "$UNIT"
|
||||
sleep 3
|
||||
|
||||
rollback() {
|
||||
echo "[回滚] 恢复旧数据并重启"
|
||||
rm -rf "$CUR"
|
||||
mv "$BAK" "$CUR"
|
||||
chown -R "$OWNER:$OWNER" "$(dirname "$CUR")" 2>/dev/null || true
|
||||
systemctl start "$UNIT"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [[ -e "$CUR" ]]; then mv "$CUR" "$BAK"; fi
|
||||
mkdir -p "$(dirname "$CUR")"
|
||||
mv "$NEW_DIR" "$CUR"
|
||||
chown -R "$OWNER:$OWNER" "$CUR"
|
||||
|
||||
systemctl start "$UNIT" || { echo "[失败] 启动失败,自动回滚"; rollback; }
|
||||
|
||||
probe_ok=0
|
||||
for _ in $(seq 1 90); do
|
||||
if [[ "$KIND" == qdrant ]] && curl -sf --max-time 2 "http://127.0.0.1:$HTTP_PORT/collections" >/dev/null 2>&1; then
|
||||
probe_ok=1; break
|
||||
elif [[ "$KIND" == neo4j ]] && (exec 3<>"/dev/tcp/127.0.0.1/$BOLT_PORT") 2>/dev/null; then
|
||||
exec 3>&- 2>/dev/null || true; probe_ok=1; break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
if [[ $probe_ok -eq 0 ]]; then
|
||||
echo "[失败] 新库探活未通过,自动回滚"
|
||||
rollback
|
||||
fi
|
||||
|
||||
echo "== 替换成功 =="
|
||||
echo " 新库在线:$CUR"
|
||||
echo " 旧库备份:$BAK"
|
||||
echo " 确认稳定后删除备份:rm -rf $BAK"
|
||||
echo " 如需回滚:rm -rf $CUR && mv $BAK $CUR && systemctl restart $UNIT"
|
||||
+43
-1
@@ -19,6 +19,7 @@
|
||||
| 打包产物目录 | `tooling/build/<VERSION>/install/`(VERSION 默认读 `tooling/tooling.env`,当前 2.0.0) |
|
||||
| 已就绪辅助脚本 | `deploy/ollama_restart.sh`(本地嵌入模型拉起)、`vstore/vstore_restart.sh`(路径自推导重启)已入包 |
|
||||
| nginx 代理模板 | `deploy/nginx/plantai.conf`(SSE 长连接 + 大文件上传适配,随安装包发布到 `install/nginx/`) |
|
||||
| QA 数据库裸进程 | **systemd 裸进程为最终部署标准**(无 docker):qdrant v1.18.3 + neo4j 5.15.0-community,全部 127.0.0.1;**端口 27433/27434 + 19888/19885**;数据在 `<部署根>/qa-db/`;安装脚本 `install/qa-db/install_qdrant.sh`、`install_neo4j.sh`(均免编译:官方静态二进制/tarball,支持本地 bin/ 离线分发);175 上 docker 容器(27333/19788)为其他项目共享服务、不动 |
|
||||
|
||||
## 1. 编译阶段(构建机)
|
||||
|
||||
@@ -108,8 +109,40 @@ rsync -av --progress tooling/build/2.0.1/install/ <target-host>:/srv/plantai/
|
||||
|
||||
## 5. 目标机部署
|
||||
|
||||
### 5.0 QA 数据库(systemd 裸进程,先于业务服务)
|
||||
|
||||
```bash
|
||||
cd /srv/plantai/install
|
||||
bash qa-db/install_qdrant.sh # 需 sudo;默认装 /usr/local/bin/qdrant + 27433/27434
|
||||
bash qa-db/install_neo4j.sh # 需 sudo;默认装 /opt/neo4j-community-5.15.0 + 19888/19885
|
||||
# 全新部署先按 5.0b 同步数据库文件再启动;换库用 qa-db/replace_db.sh
|
||||
```
|
||||
|
||||
### 5.0b 数据库数据同步与整库替换(数据与安装包双制品分开传输)
|
||||
|
||||
> **数据库文件不进 install 安装包**(否则每次发布拖 82G 且整树覆盖会波及在用的库)。
|
||||
> 全新部署/整库更新时,按本小节显式同步到目标机 `<部署根>/qa-db/`;程序发布只传 install/。
|
||||
|
||||
```bash
|
||||
# 1) 从源机(现网/构建机)同步数据库到目标机(显式步骤,与 install 包传输并列):
|
||||
rsync -av --progress <源机>:/<源部署根>/qa-db/qdrant/storage/ /srv/plantai/qa-db/qdrant/storage/
|
||||
rsync -av --progress <源机>:/<源部署根>/qa-db/neo4j/data/ /srv/plantai/qa-db/neo4j/data/
|
||||
# 2) 属主对齐(install 脚本也会做;rsync 后建议立即执行):
|
||||
sudo chown -R <运行用户>:<运行用户> /srv/plantai/qa-db/{qdrant,neo4j}
|
||||
# 3) 启动/重启裸进程(探活 curl 27433/collections 与 bolt 19888):
|
||||
sudo systemctl restart qdrant-plantai neo4j-plantai
|
||||
# 4) 整库替换(用户高频操作,数据目录换新,自动探活+回滚):
|
||||
sudo bash qa-db/replace_db.sh qdrant <新storage目录>
|
||||
sudo bash qa-db/replace_db.sh neo4j <新data目录>
|
||||
```
|
||||
|
||||
- **位置约定**:数据根 = `<部署根>/qa-db/`(默认 /srv/plantai/qa-db;可用 QA_DB_ROOT env 覆盖,install 脚本与 replace_db.sh 都认)。
|
||||
- **免编译说明**:qdrant 官方 release 静态二进制、neo4j 官方 community tarball,均无需编译;版本校验 = `qdrant --version`(v1.18.3)与 `neo4j version`(5.15.0)。受限网络用 `qa-db/bin/` 离线载体(install 包自带,无则自动回退下载含国内镜像源)。
|
||||
- **运行用户**:RUN_USER 默认 xzq(sudo 装 systemd 时自动取 SUDO_USER);systemd 单元已配 WorkingDirectory 与重启策略。
|
||||
|
||||
### 5.1 环境准备
|
||||
|
||||
```bash
|
||||
# 5.1 环境准备(目标机):
|
||||
# - 安装 py3.12 venv(或复用构建机 venv 的 rsync 拷贝),安装依赖(sdk/pyproject.toml 与 vstore/pyproject.toml 的 dependencies)
|
||||
# - 编辑 .env:sdk/.env(DOCGEN_LLM_* 与 VSTORE_BASE_URL)、vstore/.env(VSTORE_KEY、VSTORE_ROOT、VSTORE_EMBED_URL、VSTORE_EMBED_MODEL=bge-m3:567m)
|
||||
# - KB 密钥(若 SKIP_KB=0):把 kb.key 放到目标机并 chmod 600
|
||||
@@ -168,6 +201,14 @@ nginx 模板关键适配(勿删勿改,详见 `deploy/nginx/plantai.conf` 注
|
||||
# 6.1 进程与端口
|
||||
ss -tln | grep -E ':(8090|8091|8092|8093|8094|8098)' # 8090=0.0.0.0,其余 127.0.0.1
|
||||
|
||||
# 6.0b QA 数据库裸进程专项
|
||||
systemctl is-active qdrant-plantai neo4j-plantai # 均 active
|
||||
ss -tln | grep -E ':(27433|27434|19888|19885)' # 全部 127.0.0.1
|
||||
curl -s http://127.0.0.1:27433/collections | python3 -m json.tool | head -8 # 含 plant_srt_pc2025_v3_20260817_normalized
|
||||
curl -s http://127.0.0.1:8091/health | python3 -m json.tool | head -8 # neo4j_entities 6438960 / qdrant_points 13040841
|
||||
# 经 19888 bolt 登录验证(凭据用 QA 运行中进程的 NEO4J_PASSWORD,与容器一致):
|
||||
/opt/neo4j-community-5.15.0/bin/cypher-shell -a bolt://127.0.0.1:19888 -u neo4j -p "$NEO4J_PASSWORD" "MATCH (n) RETURN count(n);"
|
||||
|
||||
# 6.1b nginx 专项
|
||||
systemctl is-active nginx 2>/dev/null || pgrep -x nginx # nginx 存活
|
||||
ss -tln | grep ':80 ' # 80 端口监听(0.0.0.0 或目标机内网 IP)
|
||||
@@ -212,3 +253,4 @@ curl -s 'http://127.0.0.1:8094/api/vstore/search?q=测试&as_template=false' | p
|
||||
4. **构建机与目标机若为同一台机器**(内网 192.168.0.175 现状),部署就是替换现有 `plantai-rag-new-deploy-unified/` 部署树;执行前先备份、执行后按 §6 全量质检。
|
||||
5. 本手册基于 2026-09-08 服务器实测编写;如工具链或目录结构有变,先对照仓库实际脚本修正本手册再执行。
|
||||
6. **当前生产服务器(192.168.0.175)仍是 0.0.0.0:8090 直连形态,尚未执行 §5b 的 nginx 迁移**——启动脚本默认值已改为 127.0.0.1,但现网进程仍是旧绑定;迁移时按 §5b 顺序执行(先 nginx 后改绑),不要直接无参重启 bff_restart.sh 以免客户访问断档。
|
||||
7. **175 上 QA 数据库 docker 容器(27333/19788)为其他项目共享服务,不随 PlantAI 部署流程管理**;PlantAI-rag-new 一律走裸进程(§5.0/5.0b);遗留库(26333/18788)现网无引用,清理需用户单独命令。
|
||||
|
||||
@@ -103,6 +103,21 @@ cp "$PROJECT_ROOT/deploy/ollama_restart.sh" "$RELEASE_ROOT/ollama_restart.sh"
|
||||
# nginx 80 端口反代模板(目标机从零安装章节引用,见 docs/DEPLOY_FLOW.md §5b)
|
||||
mkdir -p "$RELEASE_ROOT/nginx"
|
||||
cp "$PROJECT_ROOT/deploy/nginx/plantai.conf" "$RELEASE_ROOT/nginx/plantai.conf"
|
||||
# QA 数据库裸进程部署脚本(systemd 标准,见 docs/DEPLOY_FLOW.md §5「QA 数据库」)
|
||||
mkdir -p "$RELEASE_ROOT/qa-db"
|
||||
cp -r "$PROJECT_ROOT/deploy/qa-db/"*.sh "$RELEASE_ROOT/qa-db/" 2>/dev/null || true
|
||||
cp -r "$PROJECT_ROOT/deploy/qa-db/"*.service "$RELEASE_ROOT/qa-db/" 2>/dev/null || true
|
||||
# 离线二进制载体(deploy/qa-db/bin/ 为 gitignore;存在则随包分发,目标机免下载)
|
||||
[ -d "$PROJECT_ROOT/deploy/qa-db/bin" ] && cp -r "$PROJECT_ROOT/deploy/qa-db/bin" "$RELEASE_ROOT/qa-db/bin" || true
|
||||
# QA reranker 模型入包(构建机仓库同级 PlantAI-rag;缺失只告警不阻断)
|
||||
RERANKER_SRC="$PROJECT_ROOT/../PlantAI-rag/bge-reranker-base"
|
||||
if [ -d "$RERANKER_SRC" ]; then
|
||||
mkdir -p "$RELEASE_ROOT/models"
|
||||
cp -r "$RERANKER_SRC" "$RELEASE_ROOT/models/bge-reranker-base"
|
||||
echo "reranker 模型已入包: models/bge-reranker-base"
|
||||
else
|
||||
echo "警告: reranker 模型源不存在($RERANKER_SRC),安装包不含本地 reranker"
|
||||
fi
|
||||
# env 模板(VSTORE_ROOT/KEY/EMBED 等;真实值不入包)
|
||||
cp "$PROJECT_ROOT/vstore/.env.example" "$RELEASE_ROOT/vstore/.env.example" 2>/dev/null || true
|
||||
|
||||
|
||||
Reference in New Issue
Block a user