mirror of
https://github.com/RuoJi6/ARTEX.git
synced 2026-11-21 09:01:50 +08:00
Compare commits
18
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4d2c327df7 | ||
|
|
c6f7c29b71 | ||
|
|
6997a46619 | ||
|
|
cc1d84d573 | ||
|
|
a4ad95d716 | ||
|
|
900c176435 | ||
|
|
437a28cddc | ||
|
|
32d8619d36 | ||
|
|
c9b1e240ec | ||
|
|
9b8073a763 | ||
|
|
faf4d4fb4a | ||
|
|
0fed4ba3a3 | ||
|
|
4d66a84ac5 | ||
|
|
570dc1a14c | ||
|
|
171db405dd | ||
|
|
17c2fc5e08 | ||
|
|
40746901db | ||
|
|
e45fa6e7b3 |
No files matched your search
@@ -13,3 +13,6 @@ ARTEX_LLM_PROVIDER=
|
||||
ARTEX_LLM_MODEL=
|
||||
ARTEX_LLM_BASE_URL=
|
||||
ARTEX_LLM_PROXY=
|
||||
|
||||
# HTTP User-Agent;留空时从浏览器 UA 样本中随机抽选,填写则使用固定值
|
||||
ARTEX_HTTP_USER_AGENT=
|
||||
+40
-1
@@ -4,8 +4,47 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- 新增 MCP 服务端:在「系统设置 → MCP 服务端」开启后,外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务、补充提示。新增访问令牌(权限范围、限定企业、任务限额、有效期、吊销)与调用审计。支持来源白名单(IP/CIDR,可配置可信反向代理),在令牌校验之前生效。默认关闭;`/mcp` 只认访问令牌,不经过也无法使用 HTTP Basic Auth。详见 README「MCP 服务端」。
|
||||
### 新增与改进
|
||||
|
||||
- 新增一键备份与迁移(#160):在系统配置中下载包含数据库、任务工作区、会话、流量、证据、归档包、MITM 证书和 skills 的单个备份文件,并可在另一实例上传恢复。恢复先校验备份,再在重启时整体替换;数据库替换失败时保留原数据,中断后下次启动继续完成文件替换。
|
||||
|
||||
## [0.3.30] - 2026-11-14
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增 MCP 服务端:外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务及补充提示。提供访问令牌的权限范围、企业限制、任务限额、有效期和吊销,以及调用审计、来源 IP/CIDR 白名单与可信代理配置。默认关闭;该端点使用独立访问令牌认证,不使用 HTTP Basic Auth。
|
||||
- 统一出站 HTTP 请求标识:Go 请求与两个 Python 采集模板从 20 个标准桌面浏览器 User-Agent 中随机选择,保留环境变量和显式请求头覆盖,同一请求的重定向与重试复用选定值。此变化仅涉及 HTTP 请求头。
|
||||
|
||||
### 修复
|
||||
|
||||
- 命令拦截除匹配原始工具输入 JSON 外,也匹配解码后的嵌套字符串值,修复换行、制表符和转义引号导致危险命令漏拦的问题;保留原始 JSON 规则及非 JSON 输入的兼容行为。
|
||||
- 升级 `go-mitmproxy` v1.9.2 → v1.9.5,修复上游 SOCKS5 握手失败时的空指针 panic;失败时返回 502,并正常触发代理失败回调。
|
||||
- 修正 README 中前端构建产物的复制命令,避免目录嵌套并创建目标目录。
|
||||
|
||||
## [0.3.29] - 2026-11-11
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增可选「无成果止损」:按实际运行时长、独立漏洞数量与最低等级做一次检查,未达标时自动收尾并显示「无成果停止」;暂停、排队及计划窗口外不计时,复活后重新计时,默认关闭。停止状态复用 `timeout`,已有 `on_task_timeout` 触发器仍会执行并可能消耗 token。
|
||||
- 任务详情新增只读文件浏览,支持目录导航、最近文件、文本与图片预览、单文件下载及目录 ZIP 下载;展示意图目录对应的标题和状态,并提供中英文界面。
|
||||
|
||||
### 修复
|
||||
|
||||
- 无成果检查与漏洞写入串行化,并在同一事务内重新计算独立漏洞数量,避免临界时刻已达到成果要求的任务被误停止。
|
||||
- 旁路问答摘要为空、被截断或过长时自动重试一次,并细分失败原因;失败摘要不写入缓存,重试计入原有调用次数、用量及超时限制。
|
||||
|
||||
## [0.3.28] - 2026-11-10
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增英文界面和语言切换,覆盖任务、审批、设置、通知及计划任务等页面,并统一常用状态、日期与错误提示的翻译。
|
||||
- 流量页面展示完整流量 ID,支持按 ID 搜索及在 HTTP 环境复制;Agent 可直接使用该 ID 读取已有流量。
|
||||
- 漏洞统计与列表使用相同筛选条件,区分独立漏洞和原始上报记录数量。
|
||||
|
||||
### 修复
|
||||
|
||||
- Basic Auth 会话失效时在当前标签页恢复认证,避免后台请求反复弹出浏览器认证框,并保留应用登录状态。
|
||||
- 修复英文界面预览、覆盖度图谱生命周期及部分页面运行时错误,并为认证、出站探测等错误补充可翻译的错误码。
|
||||
|
||||
## [0.3.18] - 2026-10-12
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# HTTP 请求标识排查
|
||||
|
||||
本次检查覆盖仓库中的 Go HTTP 请求、Python / JavaScript 脚本、安装脚本、前端示例,以及实际使用的 norma v0.4.3 的 WebFetch / WebSearch 实现。排查范围包括 User-Agent 的大小写变体、请求头构造、代理转发、重定向及代理失败后的直连重试。
|
||||
|
||||
## 已处理
|
||||
|
||||
| 通道 | 排查结果与处理 |
|
||||
|---|---|
|
||||
| 内置 WebFetch | SDK 将 `norma/0.4` 写死在首发及重试请求。worker、planner、main、chat 通过统一 session 入口使用本地适配器;保留 SDK 的工具 schema、权限、静态资源限制、代理、CA 验证及 HTML / RECON 输出。 |
|
||||
| 资产自动补全 | 移除 `artex-enrich/1.0`,HTTP 探测使用统一策略。 |
|
||||
| 出口连通性探测 | 额外发现 `artex-egress-probe/1.0`,已移除并接入统一策略。 |
|
||||
| 静态采集 / 同源爬虫模板 | 移除 `spa-api-recon` / `spa-api-recon spider`,避免 urllib 自动填入自己的默认 UA;入口、下载、重定向及重试沿用同一配置。 |
|
||||
| HTTP 自定义工具 | 原先没有 ARTEX 专属 UA,但未设置时会由 Go 填充默认值。现采用统一策略,模板显式请求头优先。 |
|
||||
| 请求重放器 | 原先没有 ARTEX 专属 UA。现采用统一策略,用户显式请求头(包括空值)优先,保持原请求的配置能力。 |
|
||||
| 记录代理 | 转发缺失 UA 的请求时,Go 会补入默认客户端标识。现保留显式省略语义,录制中不展示内部用于抑制默认值的空头。保留客户端已选择的随机或固定 UA。 |
|
||||
| GitHub 自更新 | 额外发现查询版本、下载校验和与发布包共用的两处 `artex-selfupdate`。均已移除;与其他请求统一采用随机浏览器样本或显式配置的非空 UA,不含默认 ARTEX 名称。 |
|
||||
| 前端 mock | `ARTEX-worker/1.0` 仅为示例 HTTP 文本,不会发出请求;已删除该行,避免误导。 |
|
||||
|
||||
## 配置
|
||||
|
||||
默认采用随机 UA 池方案,Go 与两个独立 Python 模板内置相同的 20 个完整浏览器 UA 样本,覆盖 Chrome / Edge / Firefox / Safari 的标准桌面格式。随机选择完整样本,允许重复,不拼接随机版本,不调用 AI 服务。
|
||||
|
||||
2026-11-11 核对并更新样本,采用主流浏览器近期稳定版本的标准格式:
|
||||
|
||||
| 浏览器 | 版本 | 平台 | 数量 |
|
||||
|---|---|---|---|
|
||||
| Chrome | 154 / 155 / 156 / 157 | Windows / macOS / Linux | 12 |
|
||||
| Edge | 156 / 157 | Windows / macOS | 4 |
|
||||
| Firefox | 159 | Windows / macOS / Linux | 3 |
|
||||
| Safari | 27.1 | macOS | 1 |
|
||||
|
||||
保留浏览器正常的 UA 缩减字段(如 Chrome 的 `0.0.0` 和 macOS 的冻结版本),不添加工具名称、随机设备型号、扩展标记或自定义后缀。“常见”指主流浏览器的标准 UA 格式,不代表已测量这 20 项在目标站点中的使用占比;浏览器类型和版本仍可从 UA 中读出。
|
||||
|
||||
格式与版本参考:
|
||||
|
||||
- [Chromium UA Reduction](https://www.chromium.org/updates/ua-reduction/)
|
||||
- [Chrome 稳定版发布记录](https://chromereleases.googleblog.com/search/label/Stable%20updates)
|
||||
- [Edge UA 格式](https://learn.microsoft.com/en-us/microsoft-edge/web-platform/user-agent-guidance)
|
||||
- [Edge 稳定版发布记录](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-relnote-stable-channel)
|
||||
- [Firefox UA 格式](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent/Firefox)
|
||||
- [Firefox 159 发布记录](https://www.firefox.com/en-US/firefox/159.0/releasenotes/)
|
||||
- [Safari 27.1 发布记录](https://developer.apple.com/documentation/safari-release-notes/safari-27_1-release-notes)
|
||||
|
||||
- Go 每次 HTTP 逻辑请求抽选一次;WebFetch 每次工具调用重新抽选,重定向、代理请求及失败后的直连重试复用同一值。
|
||||
- Python 脚本每次运行抽选一次,入口、并发资源下载及重试复用同一值;独立调用 `fetch()` 时每次调用单独选择。
|
||||
- `ARTEX_HTTP_USER_AGENT` 非空且合法时覆盖随机值;留空、仅空格或非法值回退随机池。Go 使用系统加密随机源;若随机源出错,保留池中首个非空有效样本。
|
||||
- Python 的 `--user-agent VALUE` 优先于环境变量,`--user-agent ''` 仍显式省略;自定义 HTTP 工具 / 重放器显式提供的 UA(包括空值)同样优先,保留原请求的控制能力。
|
||||
- 固定值只接受可打印 ASCII,去除首尾空格;控制字符和非 ASCII 值回退随机池,防止头注入。
|
||||
|
||||
Go 服务重启后使用新配置;Docker 在 `.env` 设置后重建容器。直接运行 Python 模板时继承同名环境变量。已同步 agent OPSEC 提示,避免旧提示与平台随机策略冲突;遇到限速等异常时仍停止,不以换 UA 继续当前方向。
|
||||
|
||||
UA 随机化不会同步改变 TLS、Cookie 或其他浏览器特征,不能保证客户端不可识别;第三方出口代理也可能再次改写请求头。
|
||||
|
||||
## 检查过的其他通道与边界
|
||||
|
||||
| 通道 | 结果 |
|
||||
|---|---|
|
||||
| SDK 联网搜索 | norma 的 DDGS 后端含固定 Firefox 128 浏览器 UA,没有 ARTEX / norma 名称;仅发往搜索服务,属于依赖的兼容策略,未改动,也不受此环境变量控制。Brave / Tavily 及 DeepSeek 搜索未发现 ARTEX 专属 UA。 |
|
||||
| 浏览器自动化 / runtime_harvest.js | 沿用浏览器请求头;运行时转发复制捕获到的请求头,没有硬编码 ARTEX UA。 |
|
||||
| LLM 请求 / 模型列表 / JEV | 未发现硬编码 ARTEX UA;仍可能有 HTTP 库、SDK、服务协议和用户自定义标识,不套用目标流量配置。 |
|
||||
| MCP HTTP / SSE | 未发现 ARTEX UA;显式自定义头保留。初始化协议体仍有 `clientInfo.name = artex`,这是对用户配置的 MCP 服务的协议自报身份,本次不改动。 |
|
||||
| Webhook / 通知 | 未发现硬编码 ARTEX UA;保留用户自定义头及客户端默认行为。 |
|
||||
| 安装脚本 | 未发现 ARTEX UA;获取 Docker 安装脚本的 curl 仍使用其原生身份。外部命令及用户自建脚本不由本次策略接管。 |
|
||||
| phpfinger | 仓库未包含其源码,无法核验默认值;需要在独立工具中通过 `--user-agent` 配置。 |
|
||||
|
||||
服务端响应中的 `X-Artex-Auth` 仍用于前端识别 Basic Auth 未认证状态,属于站点自身对访问者的响应协议;它不是向目标站点发送的请求头。本次不改变该协议,也不承诺消除服务端页面、API、TLS 或其他流量特征。
|
||||
|
||||
## 回归验证
|
||||
|
||||
- Go:检查真实 HTTP 线上头、随机样本选择 / 固定值覆盖 / 显式空值、代理及失败直连重试、重定向、CA 信任、静态资源限制、HTML / RECON 解析、agent session 注册、自定义工具 / 重放器覆盖、自更新查询及下载和代理录制。
|
||||
- Python:本地 HTTP 服务验证默认随机 / 环境变量 / CLI 覆盖 / 显式空值、非法头值、重定向、Cookie 保留及产物生成;不访问真实目标。
|
||||
|
||||
本地命令:
|
||||
|
||||
```sh
|
||||
go test ./internal/httpidentity ./internal/webfetch ./enrich ./agent ./selfupdate ./traffic
|
||||
go test ./server -run 'TestProbe|TestRepeater|TestRunHTTPTool|TestHTTPTool|TestTargetHTTPIdentity' -count=1
|
||||
go test -race ./internal/httpidentity ./internal/webfetch ./enrich ./traffic
|
||||
python3 -m unittest discover -s skills/api-recon/scripts -p 'test_*.py' -v
|
||||
```
|
||||
|
||||
以上专项回归、Go race 检查及 Python 3 项测试均通过。池扩展后的回归还检查恰好 20 个不重复样本、逐项选择以及 Go 与两个 Python 模板的样本一致性。
|
||||
|
||||
2026-11-11 使用项目提供的本地 PostgreSQL,在新建的独立测试库中执行了 `go test ./... -count=1 -p 1 -timeout 180s`,测试库用后删除。完整套件未全部通过:
|
||||
|
||||
- `agent` 的 `TestGraphOverviewExpandsAssociatedCompanyScope`、`TestCaptureApprovalLifecycle` 和 `db` 的 `TestInterceptApprovalFilters`、`TestSideTaskArchiveVersions` 失败;在原提交 `17c2fc5` 的独立副本和另一新建测试库中复现了相同失败。
|
||||
- `server` 首次运行有 3 个用例在临时目录清理阶段报 `directory not empty`,涉及 `TestInheritedActivityDetailAndRelationDeletion`、`TestTaskMetadataPatchReturnsRenameAndPin`、`TestTaskMetadataPatchGlobalAssetSharing`,不是 UA 断言失败。再次在新建隔离库中运行完整 `server` 套件时这 3 项通过,但 `TestCoreTaskLifecyclePG` 报 `expected goal nodes seeded, got 0`。原提交的完整 `server` 套件通过;修改后完整 `server` 套件仍未获得全绿结果。
|
||||
@@ -12,8 +12,13 @@ AI 自主渗透测试系统(Go 后端 + Next.js 前端)
|
||||
本仓库为独立维护的 [RuoJi6/ARTEX](https://github.com/RuoJi6/ARTEX),保留 https://github.com/Autumn-27/ARTEX 的原始提交历史与 AGPL-3.0 许可证。程序发布包从本仓库 Releases 下载,Docker 镜像 [ruoji6/artex](https://hub.docker.com/r/ruoji6/artex) 由本仓库 GitHub Actions 构建并发布。
|
||||
|
||||
---
|
||||
## 致谢
|
||||
致谢米斯特安全团队
|
||||
## 致谢
|
||||
米斯特安全团队
|
||||
|
||||
洺熙
|
||||
|
||||
wowo
|
||||
|
||||
|
||||
## 截图预览
|
||||
|
||||
@@ -157,9 +162,11 @@ cp config.example.json config.json # 填好 database 连接
|
||||
```bash
|
||||
# 1) 前端静态导出
|
||||
cd web && npm ci && npm run build:static && cd ..
|
||||
# 2) 拷进内嵌目录
|
||||
cp -r web/out server/webui/dist
|
||||
# 3) 编译(-tags embedui 才内嵌前端)
|
||||
# 2) 创建内嵌目录
|
||||
mkdir -p server/webui/dist
|
||||
# 3) 拷进内嵌目录
|
||||
cp -r web/out/. server/webui/dist
|
||||
# 4) 编译(-tags embedui 才内嵌前端)
|
||||
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
./start.sh
|
||||
```
|
||||
@@ -182,6 +189,18 @@ ARTEX_TARGETS=linux/amd64,windows/amd64 ./build.sh --release
|
||||
|
||||
---
|
||||
|
||||
## 一键备份与迁移
|
||||
|
||||
在 **系统配置 → 系统与账号** 的 **备份与迁移** 卡片中:
|
||||
|
||||
- **下载备份**:生成单个 `artex-backup-<时间>.tar.zst`,浏览器直接落盘。导出前需先暂停或等待所有运行中/排队中的任务完成,避免导出期间文件写入导致不一致。内容包括 PostgreSQL 全部业务表(同一快照事务内分片流式导出,表间一致)、数据目录(任务工作区、会话、证据、任务归档包、MITM CA 证书、流量及其 SQLite 索引快照)和 skills 目录。不含系统运行日志(`server_logs`)和 `jwt.key`。
|
||||
- **从备份恢复**:在新机器上先正常安装并启动 ARTEX,登录后上传备份文件。服务端先完整解包并按当前数据库结构校验,通过后重启;重启时、在任何组件打开数据之前替换数据:数据库在一个事务内整体替换(失败则原数据不变,恢复包移到 `data/.restore-failed` 供排查),数据目录逐项替换;若文件替换失败,进程会非零退出由守护脚本重试,直到全部替换完成才启动业务引擎。卡片会显示上次恢复的结果。
|
||||
- 恢复会**覆盖当前实例的全部数据**并重启停止当前运行中的任务;备份中未结束的任务在恢复后会保持暂停,需手动恢复,不会在新机器上自动继续向目标发起请求。登录密码、Basic Auth、LLM 与其他配置都随数据库迁移,恢复后用原实例的密码登录。
|
||||
- 新机器上已存在的同名 skill 保留本机版本(避免旧备份覆盖新版内置 skill),其余 skill 从备份装入。
|
||||
- 只能恢复到**相同或更新版本**的 ARTEX;备份含当前版本不认识的表或列时会拒绝恢复。旧路径(如任务归档包的绝对路径)会自动改写到新数据目录。
|
||||
- 恢复依赖 `start.sh` / `start.bat`(Docker 下为镜像 ENTRYPOINT)重启程序,与页面一键更新相同。
|
||||
- 恢复期间需在数据目录所在磁盘额外预留约一份**未压缩备份包**大小的临时空间(解包后的数据库文本与文件);若前面有 Nginx 反向代理,请按下方反代示例为 `/api/system/backup/` 放宽 `client_max_body_size` 并关闭 `proxy_request_buffering`。
|
||||
|
||||
## 更新升级
|
||||
|
||||
> 升级只换程序、不动数据:Postgres 数据卷 `pgdata`、`./data`(jwt.key / SQLite 等)、`./skills` 都会保留。**数据库迁移无需手动执行**——`artex` 每次启动会幂等重跑 `schema.sql`(含 `ADD COLUMN` / `CREATE INDEX IF NOT EXISTS`),即“重启即迁移”。升级前仍建议先备份 `./data` 与数据库。
|
||||
@@ -290,7 +309,8 @@ cp -r <解压目录>/skills ./ && cp <解压目录>/artex ./
|
||||
```bash
|
||||
git pull
|
||||
cd web && npm ci && npm run build:static && cd ..
|
||||
cp -r web/out server/webui/dist
|
||||
mkdir -p server/webui/dist
|
||||
cp -r web/out/. server/webui/dist
|
||||
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
# 重启 ./start.sh
|
||||
```
|
||||
@@ -314,6 +334,10 @@ CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
**LLM**:`export ANTHROPIC_API_KEY=sk-...`(或 `OPENAI_API_KEY`),也可在 UI 的「LLM 配置」页填写。
|
||||
可选:`ARTEX_LLM_PROVIDER` / `ARTEX_LLM_MODEL` / `ARTEX_LLM_BASE_URL` / `ARTEX_LLM_PROXY`。
|
||||
|
||||
**HTTP User-Agent**:内置 WebFetch(worker / planner / main / chat)、资产 HTTP 自动补全、出口连通性探测、HTTP 自定义工具、请求重放器、GitHub 自更新和可选的 `harvest_static.py` / `spider_mpa.py` 默认从 20 个浏览器 UA 样本中随机抽选,不携带工具名称。Go 每次逻辑请求选择一次,重定向及 WebFetch 的代理失败直连重试复用同一个值;Python 脚本每次运行选择一次,入口、下载与重试保持一致。随机抽选允许重复,不依赖 AI 服务。通过 `ARTEX_HTTP_USER_AGENT` 可指定固定值;留空或非法值使用随机策略。直接运行时导出环境变量并重启服务,Docker 在 `.env` 配置后重建容器。Python 的 `--user-agent VALUE`、自定义工具和重放器中的显式 UA 优先,显式空值仍可省略此头。值去除首尾空格,仅接受可打印 ASCII。记录代理保留已有 UA 及显式省略语义。UA 随机化不代表 TLS、Cookie 或其他浏览器特征同步改变,也不保证无法识别客户端。此设置不影响浏览器、SDK 联网搜索、LLM / MCP / Webhook 服务通信或外部独立工具;本仓库未包含 phpfinger,其 `--user-agent` 需在该工具中单独配置。
|
||||
|
||||
完整排查范围、额外发现及未接管的服务通信见 [HTTP 请求标识排查](HTTP_USER_AGENT.md)。
|
||||
|
||||
**并发**:每个任务的 work agent 数在「系统设置」里配置(默认 3)。
|
||||
|
||||
**常用参数**:`./start.sh -addr :8787 -proxy :8788`(`-addr` 前端+API,`-proxy` 流量录制代理)。启动脚本会把参数原样透传给 `artex`。
|
||||
@@ -441,6 +465,22 @@ server {
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
}
|
||||
|
||||
# 一键备份 / 恢复:允许大体积备份流式上传与下载,避免触发默认 1m 限制与请求体缓冲
|
||||
location /api/system/backup/ {
|
||||
proxy_pass http://127.0.0.1:8787;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
client_max_body_size 0;
|
||||
proxy_request_buffering off;
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ import (
|
||||
// contiguous run is coalesced into a single record so the trace shows whole
|
||||
// messages, not dozens of fragments.
|
||||
func captureRun(ctx context.Context, opts agentcore.Options, input string, emit func(db.Activity)) (string, harness.TerminalReason, error) {
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close() // release the session's background-task manager (temp dir + processes)
|
||||
return captureRunSession(ctx, s, input, emit)
|
||||
}
|
||||
|
||||
+1
-1
@@ -173,7 +173,7 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
|
||||
// 实验功能:开启后由 noa 接管上下文压缩(归档集中在 <workDir>/noa/<SessionID> 下,持久)。
|
||||
enableNoa(&opts, c.noaEnabledFn, c.workDir, "chat-"+sessionID, noaWarn("chat-"+sessionID))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close()
|
||||
// reload prior conversation so the agent has context across turns (each Chat is
|
||||
// a fresh session). First turn: no file yet → Resume loads nothing and proceeds.
|
||||
|
||||
+1
-1
@@ -184,7 +184,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
|
||||
}
|
||||
enableNoa(&opts, m.noaEnabledFn, m.workDir, noaSession, noaWarn(noaSession))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close()
|
||||
// reload the prior conversation from the transcript so the agent has context
|
||||
// across turns (each Chat is a fresh session; without this it can't see earlier
|
||||
|
||||
+2
-2
@@ -21,9 +21,9 @@ const opsecPromptBlock = `【OPSEC 准则(硬约束,仅次于授权范围和
|
||||
|
||||
- 先复用已有流量、浏览器会话和已验证证据;能通过 traffic_search/traffic_get 或浏览器完成时,不要重复直接请求。
|
||||
- 所有目标交互必须经过平台配置的记录代理或浏览器工具,确保请求可追溯。
|
||||
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。命令行请求必须尽量复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;没有真实依据时不要伪造身份、设备或浏览器指纹。
|
||||
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。优先复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;平台 HTTP 工具和采集模板的默认 UA 由系统随机策略提供,同一次请求的重定向、重试必须保持一致,明确配置的 UA 优先。
|
||||
- 不得使用 -k、--ignore-https-errors、verify=False 等方式绕过证书校验;平台代理 CA 已配置时应正常验证。
|
||||
- 不得随机轮换 User-Agent、批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向。
|
||||
- 不得批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向,不得通过更换 UA 继续当前方向。
|
||||
- OPSEC 不扩大授权范围;目标、主机、端口、路径或账号不在授权范围内时不得访问。
|
||||
- 请求和响应是否留痕由平台代理和浏览器负责;不要额外复制完整请求头、Cookie 或响应正文。`
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"github.com/Autumn-27/artex/internal/webfetch"
|
||||
"github.com/Autumn-27/norma/agentcore"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
|
||||
// newAgentSession installs the configurable WebFetch before the SDK builds the
|
||||
// registry, so every agent role gets one tool and the same proxy/TLS settings.
|
||||
func newAgentSession(opts agentcore.Options) *agentcore.Session {
|
||||
if opts.EnableWebFetch {
|
||||
opts.EnableWebFetch = false
|
||||
if opts.Tools == nil {
|
||||
opts.Tools = actool.DefaultTools()
|
||||
}
|
||||
opts.Tools = append(opts.Tools, webfetch.New(actool.WebFetchConfig{
|
||||
Proxy: opts.WebFetchProxy,
|
||||
CACert: opts.WebFetchCACert,
|
||||
InsecureTLS: opts.WebFetchInsecureTLS,
|
||||
}))
|
||||
}
|
||||
return agentcore.NewSession(opts)
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Autumn-27/norma/agentcore"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
"github.com/Autumn-27/norma/permission"
|
||||
)
|
||||
|
||||
type fetchFixtureProvider struct {
|
||||
autoTitleProvider
|
||||
t *testing.T
|
||||
url string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (p *fetchFixtureProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
p.calls++
|
||||
if p.calls == 1 {
|
||||
counts := map[string]int{}
|
||||
for _, tool := range req.Tools {
|
||||
counts[tool.Name]++
|
||||
}
|
||||
if counts["WebFetch"] != 1 || counts["Read"] != 1 || counts["Bash"] != 1 {
|
||||
p.t.Errorf("session tool registration = %v", counts)
|
||||
}
|
||||
input, _ := json.Marshal(map[string]string{"url": p.url})
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{{Type: llm.BlockToolUse, ID: "fetch-fixture", Name: "WebFetch", Input: input}}}, "tool_use", llm.Usage{}, nil
|
||||
}
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("finished")}}, "end_turn", llm.Usage{}, nil
|
||||
}
|
||||
|
||||
func TestAgentSessionUsesConfiguredWebFetch(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
calls := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if ua == "" && !strings.HasPrefix(r.UserAgent(), "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", r.UserAgent())
|
||||
} else if ua != "" && r.UserAgent() != ua {
|
||||
t.Errorf("session sent User-Agent %q, want %q", r.UserAgent(), ua)
|
||||
}
|
||||
w.Write([]byte("fixture"))
|
||||
}))
|
||||
defer server.Close()
|
||||
provider := &fetchFixtureProvider{t: t, url: server.URL}
|
||||
session := newAgentSession(agentcore.Options{
|
||||
Provider: provider, NonStreaming: true, EnableWebFetch: true,
|
||||
PermissionMode: permission.ModeBypass, MaxTurns: 3,
|
||||
})
|
||||
defer session.Close()
|
||||
for _, err := range session.Prompt(t.Context(), "fetch local fixture") {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("WebFetch calls = %d, want 1", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -488,7 +488,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
|
||||
noaSession := WorkerSessionID(ts.ID(), intent.ID)
|
||||
enableNoa(&opts, w.noaEnabledFn, w.workDir, noaSession, noaWarn(noaSession))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close() // release the session's background-task manager (temp dir + processes)
|
||||
|
||||
// Resume prior conversation if this intent was paused/blocked/exhausted and is
|
||||
|
||||
+12
-4
@@ -90,6 +90,18 @@ func run() int {
|
||||
log.Printf("[config] 配置文件: %s (不存在 — 将仅尝试环境变量 ARTEX_PG_DSN)", cfgPath)
|
||||
}
|
||||
|
||||
// 页面「恢复备份」暂存的数据必须在任何组件打开数据目录与数据库之前装好。
|
||||
skillDir := config.SkillDir()
|
||||
if abs, err := filepath.Abs(skillDir); err == nil {
|
||||
skillDir = abs
|
||||
}
|
||||
restoreResult, err := server.ApplyPendingRestore(*dataDir, skillDir)
|
||||
if err != nil {
|
||||
log.Printf("[backup] 恢复文件替换未完成,终止本次启动等待重试:%v", err)
|
||||
return 1
|
||||
}
|
||||
server.SetBootRestoreResult(restoreResult)
|
||||
|
||||
sigCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
ctx, shutdown := shutdownContext(sigCtx)
|
||||
@@ -107,10 +119,6 @@ func run() int {
|
||||
settle := time.AfterFunc(selfupdate.SettleDelay, selfupdate.Settle)
|
||||
defer settle.Stop()
|
||||
|
||||
skillDir := config.SkillDir()
|
||||
if abs, err := filepath.Abs(skillDir); err == nil {
|
||||
skillDir = abs
|
||||
}
|
||||
log.Printf("[config] skill 目录: %s", skillDir)
|
||||
srv := server.New(ctx, mgr, skillDir, *dataDir, config.BaseDir())
|
||||
defer srv.Close()
|
||||
|
||||
+362
@@ -0,0 +1,362 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
// BackupTable describes one table's columns, in the order its COPY data uses,
|
||||
// and how many rows the data holds.
|
||||
type BackupTable struct {
|
||||
Name string `json:"name"`
|
||||
Columns []string `json:"columns"`
|
||||
Rows int64 `json:"rows"`
|
||||
}
|
||||
|
||||
// RestoreTable is one table's COPY text-format data to load during a restore.
|
||||
type RestoreTable struct {
|
||||
BackupTable
|
||||
Open func() (io.ReadCloser, error)
|
||||
}
|
||||
|
||||
// PathPrefixMove rewrites absolute paths recorded under an old data directory
|
||||
// (task_archives.archive_path) so they point at the restoring instance's one.
|
||||
// Separators are carried separately so a Windows backup lands on Linux intact.
|
||||
type PathPrefixMove struct{ From, FromSeparator, To, ToSeparator string }
|
||||
|
||||
// backupSkippedTables are neither exported nor replaced: server_logs is this
|
||||
// process's own runtime log ring, not data worth migrating.
|
||||
var backupSkippedTables = map[string]bool{"server_logs": true}
|
||||
|
||||
// tableColumnsSQL lists every ordinary table of the application schema with its
|
||||
// writable columns. Generated columns cannot be COPY'd in, so they are left out.
|
||||
const tableColumnsSQL = `
|
||||
SELECT c.relname::text, array_agg(a.attname::text ORDER BY a.attnum)
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum > 0 AND NOT a.attisdropped AND a.attgenerated = ''
|
||||
WHERE n.nspname = current_schema() AND c.relkind = 'r'
|
||||
GROUP BY c.relname
|
||||
ORDER BY c.relname`
|
||||
|
||||
func ident(name string) string { return pgx.Identifier{name}.Sanitize() }
|
||||
|
||||
func columnList(columns []string) string {
|
||||
quoted := make([]string, len(columns))
|
||||
for i, column := range columns {
|
||||
quoted[i] = ident(column)
|
||||
}
|
||||
return strings.Join(quoted, ", ")
|
||||
}
|
||||
|
||||
// withPgx runs fn on one pooled connection's native pgx handle; COPY is only
|
||||
// reachable through it.
|
||||
func (d *DB) withPgx(ctx context.Context, fn func(*pgx.Conn) error) error {
|
||||
conn, err := d.Conn(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Close()
|
||||
return conn.Raw(func(driverConn any) error {
|
||||
native, ok := driverConn.(*stdlib.Conn)
|
||||
if !ok {
|
||||
return fmt.Errorf("unexpected driver connection %T", driverConn)
|
||||
}
|
||||
return fn(native.Conn())
|
||||
})
|
||||
}
|
||||
|
||||
func queryTables(ctx context.Context, tx pgx.Tx) ([]BackupTable, error) {
|
||||
rows, err := tx.Query(ctx, tableColumnsSQL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var tables []BackupTable
|
||||
for rows.Next() {
|
||||
var table BackupTable
|
||||
if err := rows.Scan(&table.Name, &table.Columns); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !backupSkippedTables[table.Name] {
|
||||
tables = append(tables, table)
|
||||
}
|
||||
}
|
||||
return tables, rows.Err()
|
||||
}
|
||||
|
||||
// CountActiveTasks returns how many non-deleted, non-terminal tasks are not
|
||||
// paused (running, created/unpaused, or queued). Exporting while any exist
|
||||
// would race with live disk writes and put active tasks into the backup.
|
||||
func (d *DB) CountActiveTasks(ctx context.Context) (int, error) {
|
||||
var count int
|
||||
err := d.QueryRowContext(ctx, `
|
||||
SELECT COUNT(*)
|
||||
FROM tasks
|
||||
WHERE deleted_at IS NULL
|
||||
AND paused = false
|
||||
AND status NOT IN ('done', 'failed', 'timeout')`).Scan(&count)
|
||||
return count, err
|
||||
}
|
||||
|
||||
// DumpTables exports every application table from a single read-only
|
||||
// REPEATABLE READ snapshot, so rows written while the dump runs never produce a
|
||||
// backup whose tables disagree with each other. emit is called once per table
|
||||
// with a function that writes the table's COPY text data and returns its row
|
||||
// count; emit records the count itself.
|
||||
func (d *DB) DumpTables(ctx context.Context, emit func(BackupTable, func(io.Writer) (int64, error)) error) error {
|
||||
return d.withPgx(ctx, func(conn *pgx.Conn) error {
|
||||
tx, err := conn.BeginTx(ctx, pgx.TxOptions{IsoLevel: pgx.RepeatableRead, AccessMode: pgx.ReadOnly})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx) //nolint:errcheck // read-only; nothing to keep
|
||||
tables, err := queryTables(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, table := range tables {
|
||||
statement := fmt.Sprintf("COPY %s (%s) TO STDOUT", ident(table.Name), columnList(table.Columns))
|
||||
copyTo := func(w io.Writer) (int64, error) {
|
||||
tag, err := tx.Conn().PgConn().CopyTo(ctx, w, statement)
|
||||
return tag.RowsAffected(), err
|
||||
}
|
||||
if err := emit(table, copyTo); err != nil {
|
||||
return fmt.Errorf("备份表 %s: %w", table.Name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// CheckRestoreTables reports whether every table and column of a backup exists
|
||||
// in the current schema. A backup taken by a newer ARTEX may carry columns this
|
||||
// build does not know; loading it would fail halfway, so it is refused upfront.
|
||||
func (d *DB) CheckRestoreTables(ctx context.Context, tables []BackupTable) error {
|
||||
return d.withPgx(ctx, func(conn *pgx.Conn) error {
|
||||
tx, err := conn.BeginTx(ctx, pgx.TxOptions{AccessMode: pgx.ReadOnly})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx) //nolint:errcheck // read-only
|
||||
current, err := queryTables(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return checkRestoreTables(current, tables)
|
||||
})
|
||||
}
|
||||
|
||||
func checkRestoreTables(current, tables []BackupTable) error {
|
||||
known := make(map[string]map[string]bool, len(current))
|
||||
for _, table := range current {
|
||||
columns := make(map[string]bool, len(table.Columns))
|
||||
for _, column := range table.Columns {
|
||||
columns[column] = true
|
||||
}
|
||||
known[table.Name] = columns
|
||||
}
|
||||
seen := make(map[string]bool, len(tables))
|
||||
for _, table := range tables {
|
||||
if seen[table.Name] {
|
||||
return fmt.Errorf("备份中表 %s 重复", table.Name)
|
||||
}
|
||||
seen[table.Name] = true
|
||||
columns, ok := known[table.Name]
|
||||
if !ok {
|
||||
return fmt.Errorf("备份包含当前版本不存在的表 %s(备份可能来自更新的 ARTEX 版本)", table.Name)
|
||||
}
|
||||
for _, column := range table.Columns {
|
||||
if !columns[column] {
|
||||
return fmt.Errorf("备份表 %s 含当前版本不存在的列 %s(备份可能来自更新的 ARTEX 版本)", table.Name, column)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type foreignKey struct{ table, name, definition string }
|
||||
|
||||
// RestoreTables replaces the contents of every application table with the given
|
||||
// data in one transaction: either the whole backup lands or nothing changes.
|
||||
// Tables of the current schema that the backup does not carry end up empty;
|
||||
// columns it does not carry take their defaults.
|
||||
//
|
||||
// Foreign keys are dropped for the load and re-added afterwards (which
|
||||
// re-validates them), and user triggers are disabled so derived rows such as
|
||||
// task_asset_links come from the backup instead of being regenerated on insert.
|
||||
// Both only need table ownership, unlike session_replication_role, which needs
|
||||
// superuser. Serial sequences are moved past the restored ids.
|
||||
func (d *DB) RestoreTables(ctx context.Context, tables []RestoreTable, move PathPrefixMove) error {
|
||||
return d.withPgx(ctx, func(conn *pgx.Conn) error {
|
||||
tx, err := conn.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx) //nolint:errcheck // no-op once committed
|
||||
if _, err := tx.Exec(ctx, `SELECT pg_advisory_xact_lock($1)`, schemaMigrationLockKey); err != nil {
|
||||
return fmt.Errorf("coordinate with schema migration: %w", err)
|
||||
}
|
||||
current, err := queryTables(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
backup := make([]BackupTable, len(tables))
|
||||
for i, table := range tables {
|
||||
backup[i] = table.BackupTable
|
||||
}
|
||||
if err := checkRestoreTables(current, backup); err != nil {
|
||||
return err
|
||||
}
|
||||
names := make([]string, len(current))
|
||||
for i, table := range current {
|
||||
names[i] = ident(table.Name)
|
||||
}
|
||||
|
||||
keys, err := listForeignKeys(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, key := range keys {
|
||||
if _, err := tx.Exec(ctx, fmt.Sprintf("ALTER TABLE %s DROP CONSTRAINT %s", key.table, ident(key.name))); err != nil {
|
||||
return fmt.Errorf("暂时移除外键 %s: %w", key.name, err)
|
||||
}
|
||||
}
|
||||
for _, name := range names {
|
||||
if _, err := tx.Exec(ctx, "ALTER TABLE "+name+" DISABLE TRIGGER USER"); err != nil {
|
||||
return fmt.Errorf("暂停触发器 %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
if len(names) > 0 {
|
||||
if _, err := tx.Exec(ctx, "TRUNCATE "+strings.Join(names, ", ")); err != nil {
|
||||
return fmt.Errorf("清空当前数据: %w", err)
|
||||
}
|
||||
}
|
||||
for _, table := range tables {
|
||||
if err := copyIn(ctx, tx, table); err != nil {
|
||||
return fmt.Errorf("恢复表 %s: %w", table.Name, err)
|
||||
}
|
||||
}
|
||||
if move.From != "" && (move.From != move.To || move.FromSeparator != move.ToSeparator) {
|
||||
// Runs while triggers are still off so updated_at keeps the backup's value.
|
||||
from := strings.TrimSuffix(move.From, move.FromSeparator) + move.FromSeparator
|
||||
to := strings.TrimSuffix(move.To, move.ToSeparator) + move.ToSeparator
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE task_archives SET archive_path = $2 || replace(substr(archive_path, length($1) + 1), $3, $4)
|
||||
WHERE left(archive_path, length($1)) = $1`, from, to, move.FromSeparator, move.ToSeparator); err != nil {
|
||||
return fmt.Errorf("改写归档包路径: %w", err)
|
||||
}
|
||||
}
|
||||
// Any non-terminal task that was not paused when backed up (for example,
|
||||
// from an older backup) is forced into a manual pause so restoring onto a
|
||||
// new host never resumes scanning targets on its own.
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE tasks
|
||||
SET paused = true,
|
||||
pause_reason = CASE WHEN COALESCE(pause_reason, '') = '' THEN 'paused_by_user' ELSE pause_reason END,
|
||||
queued = false,
|
||||
queued_at = NULL,
|
||||
schedule_paused = false,
|
||||
schedule_manual_until = NULL,
|
||||
active_run_seconds = active_run_seconds + CASE WHEN run_segment_started_at IS NOT NULL THEN GREATEST(0, FLOOR(EXTRACT(EPOCH FROM (now() - run_segment_started_at)))::bigint) ELSE 0 END,
|
||||
run_segment_started_at = NULL
|
||||
WHERE deleted_at IS NULL
|
||||
AND paused = false
|
||||
AND status NOT IN ('done', 'failed', 'timeout')`); err != nil {
|
||||
return fmt.Errorf("暂停恢复后的未完成任务: %w", err)
|
||||
}
|
||||
for _, key := range keys {
|
||||
if _, err := tx.Exec(ctx, fmt.Sprintf("ALTER TABLE %s ADD CONSTRAINT %s %s", key.table, ident(key.name), key.definition)); err != nil {
|
||||
return fmt.Errorf("恢复外键 %s: %w", key.name, err)
|
||||
}
|
||||
}
|
||||
for _, name := range names {
|
||||
if _, err := tx.Exec(ctx, "ALTER TABLE "+name+" ENABLE TRIGGER USER"); err != nil {
|
||||
return fmt.Errorf("恢复触发器 %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
if err := resetSequences(ctx, tx); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
})
|
||||
}
|
||||
|
||||
func listForeignKeys(ctx context.Context, tx pgx.Tx) ([]foreignKey, error) {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT conrelid::regclass::text, conname::text, pg_get_constraintdef(oid)
|
||||
FROM pg_constraint
|
||||
WHERE contype = 'f' AND connamespace = current_schema()::regnamespace
|
||||
ORDER BY 1, 2`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var keys []foreignKey
|
||||
for rows.Next() {
|
||||
var key foreignKey
|
||||
if err := rows.Scan(&key.table, &key.name, &key.definition); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keys = append(keys, key)
|
||||
}
|
||||
return keys, rows.Err()
|
||||
}
|
||||
|
||||
func copyIn(ctx context.Context, tx pgx.Tx, table RestoreTable) error {
|
||||
data, err := table.Open()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer data.Close()
|
||||
statement := fmt.Sprintf("COPY %s (%s) FROM STDIN", ident(table.Name), columnList(table.Columns))
|
||||
tag, err := tx.Conn().PgConn().CopyFrom(ctx, data, statement)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() != table.Rows {
|
||||
return fmt.Errorf("写入 %d 行,备份清单记录 %d 行", tag.RowsAffected(), table.Rows)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resetSequences moves every column-owned sequence past the restored maximum;
|
||||
// otherwise the next insert would collide with a restored primary key.
|
||||
func resetSequences(ctx context.Context, tx pgx.Tx) error {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT c.relname::text, a.attname::text, s.seq
|
||||
FROM pg_class c
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum > 0 AND NOT a.attisdropped
|
||||
CROSS JOIN LATERAL (SELECT pg_get_serial_sequence(format('%I.%I', n.nspname, c.relname), a.attname) AS seq) s
|
||||
WHERE n.nspname = current_schema() AND c.relkind = 'r' AND s.seq IS NOT NULL`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
type serial struct{ table, column, sequence string }
|
||||
var serials []serial
|
||||
for rows.Next() {
|
||||
var item serial
|
||||
if err := rows.Scan(&item.table, &item.column, &item.sequence); err != nil {
|
||||
rows.Close()
|
||||
return err
|
||||
}
|
||||
serials = append(serials, item)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, item := range serials {
|
||||
statement := fmt.Sprintf("SELECT setval($1, COALESCE((SELECT max(%s) FROM %s), 0) + 1, false)", ident(item.column), ident(item.table))
|
||||
if _, err := tx.Exec(ctx, statement, item.sequence); err != nil {
|
||||
return fmt.Errorf("重置序列 %s: %w", item.sequence, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -28,6 +28,7 @@ services:
|
||||
ARTEX_LLM_MODEL: ${ARTEX_LLM_MODEL:-}
|
||||
ARTEX_LLM_BASE_URL: ${ARTEX_LLM_BASE_URL:-}
|
||||
ARTEX_LLM_PROXY: ${ARTEX_LLM_PROXY:-}
|
||||
ARTEX_HTTP_USER_AGENT: ${ARTEX_HTTP_USER_AGENT:-}
|
||||
ports:
|
||||
- "8787:8787"
|
||||
- "127.0.0.1:8788:8788"
|
||||
|
||||
+2
-1
@@ -20,6 +20,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
|
||||
"github.com/miekg/dns"
|
||||
"github.com/projectdiscovery/dnsx/libs/dnsx"
|
||||
@@ -230,7 +231,7 @@ func (e *Engine) doHTTP(id int64, rawURL string) {
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-enrich/1.0")
|
||||
httpidentity.Apply(req)
|
||||
resp, err := e.client.Do(req)
|
||||
if err != nil {
|
||||
return
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package enrich
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Stop after receiving the probe so this request regression needs no database.
|
||||
func TestProbeUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
calls := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
http.Redirect(w, r, "/stop", http.StatusFound)
|
||||
}))
|
||||
defer server.Close()
|
||||
client := buildClient(nil)
|
||||
client.CheckRedirect = func(*http.Request, []*http.Request) error {
|
||||
return errors.New("probe captured")
|
||||
}
|
||||
e := &Engine{client: client}
|
||||
e.doHTTP(1, server.URL)
|
||||
if calls != 1 {
|
||||
t.Fatalf("probe calls = %d, want 1", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -3,19 +3,19 @@ module github.com/Autumn-27/artex
|
||||
go 1.26.10
|
||||
|
||||
require (
|
||||
mvdan.cc/sh/v3 v3.10.0
|
||||
github.com/Autumn-27/norma v0.4.3
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/jackc/pgx/v5 v5.10.0
|
||||
github.com/klauspost/compress v1.18.6
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5
|
||||
github.com/miekg/dns v1.1.72
|
||||
github.com/projectdiscovery/dnsx v1.2.3
|
||||
golang.org/x/crypto v0.55.0
|
||||
golang.org/x/net v0.58.0
|
||||
golang.org/x/text v0.41.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/net v0.61.0
|
||||
golang.org/x/text v0.43.0
|
||||
modernc.org/sqlite v1.52.0
|
||||
mvdan.cc/sh/v3 v3.10.0
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -52,10 +52,10 @@ require (
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
|
||||
golang.org/x/mod v0.40.0 // indirect
|
||||
golang.org/x/sync v0.22.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/tools v0.49.0 // indirect
|
||||
golang.org/x/mod v0.41.0 // indirect
|
||||
golang.org/x/sync v0.24.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/tools v0.51.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
modernc.org/libc v1.72.3 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
|
||||
@@ -48,8 +48,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2 h1:+QTbDyUUaVRmEavPq0/eWzJ2K+K4JdL4ff91s/adnxk=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2/go.mod h1:6H5PR+nbVsiGQJC/Y98GHJItyFcmP/Z0txAgjhG2a80=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5 h1:4ar1qWYlbXF+7HhG06AVjKkDILXX1FS0mNy+hkX4uX4=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5/go.mod h1:74Z1IyzuUu/pq1IpdDd555f1xXM7Gl9KtUhWn/mzU7E=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
|
||||
@@ -108,21 +108,35 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
|
||||
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
|
||||
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
|
||||
golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
|
||||
golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
|
||||
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
|
||||
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
|
||||
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
|
||||
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
|
||||
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
|
||||
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
|
||||
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
|
||||
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
|
||||
golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
|
||||
golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
|
||||
+50
-9
@@ -572,8 +572,9 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
|
||||
if err != nil || len(rules) == 0 {
|
||||
return Decision{}, false
|
||||
}
|
||||
values := inputStrings(input)
|
||||
for _, r := range rules {
|
||||
if ruleMatches(r, toolName, input) {
|
||||
if ruleMatches(r, toolName, input, values) {
|
||||
msg := r.Message
|
||||
if msg == "" {
|
||||
msg = defaultMessage(r.Action, r.Name)
|
||||
@@ -593,20 +594,60 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
|
||||
return Decision{}, false
|
||||
}
|
||||
|
||||
func ruleMatches(r compiledRule, toolName string, input []byte) bool {
|
||||
var subject string
|
||||
// ruleMatches reports whether r applies to the call. tool_input rules are tried
|
||||
// against the raw JSON first, then against each decoded string value in values.
|
||||
// The raw form alone is not enough: JSON escapes newlines, tabs and quotes, and
|
||||
// Go's encoder also escapes & < >, so a rule like `\brm\b` never fires on
|
||||
// "cd /tmp\nrm -rf /var/www" (the raw bytes read `\nrm`, with no word boundary).
|
||||
func ruleMatches(r compiledRule, toolName string, input []byte, values []string) bool {
|
||||
switch r.MatchTarget {
|
||||
case "tool_name":
|
||||
subject = toolName
|
||||
return r.matchString(toolName)
|
||||
case "tool_input":
|
||||
subject = string(input)
|
||||
default:
|
||||
return false
|
||||
if r.matchString(string(input)) {
|
||||
return true
|
||||
}
|
||||
for _, v := range values {
|
||||
if r.matchString(v) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r compiledRule) matchString(s string) bool {
|
||||
if r.MatchType == "regex" {
|
||||
return r.re != nil && r.re.MatchString(subject)
|
||||
return r.re != nil && r.re.MatchString(s)
|
||||
}
|
||||
return strings.Contains(subject, r.Pattern)
|
||||
return strings.Contains(s, r.Pattern)
|
||||
}
|
||||
|
||||
// inputStrings returns every string value in a JSON tool input, at any depth.
|
||||
// Object keys are skipped. It returns nil if the input is not valid JSON.
|
||||
func inputStrings(input []byte) []string {
|
||||
var doc any
|
||||
if json.Unmarshal(input, &doc) != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
var walk func(any)
|
||||
walk = func(v any) {
|
||||
switch v := v.(type) {
|
||||
case string:
|
||||
out = append(out, v)
|
||||
case []any:
|
||||
for _, e := range v {
|
||||
walk(e)
|
||||
}
|
||||
case map[string]any:
|
||||
for _, e := range v {
|
||||
walk(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(doc)
|
||||
return out
|
||||
}
|
||||
|
||||
func defaultMessage(action, name string) string {
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
package intercept
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"regexp"
|
||||
"testing"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
)
|
||||
|
||||
// Patterns copied from the built-in rules seeded in db/db.go.
|
||||
const (
|
||||
builtinRmRf = `(?i)\brm\b.{0,80}(?:-[a-z]*r[a-z]*f[a-z]*|-[a-z]*f[a-z]*r[a-z]*|--recursive|--no-preserve-root)`
|
||||
builtinMkfs = `\bmkfs\b`
|
||||
builtinDrop = `(?i)\bDROP\s+(?:DATABASE|TABLE|SCHEMA|INDEX|VIEW|TABLESPACE|USER|ROLE)\b`
|
||||
builtinFlush = `(?i)\b(?:FLUSHALL|FLUSHDB)\b`
|
||||
builtinV2 = `(?i)\b(rm\s+-rf\s+/|mkfs|dd\s+if=|:\(\)\s*\{|shutdown|reboot|>\s*/dev/sd)`
|
||||
)
|
||||
|
||||
func testRule(target, matchType, pattern string) compiledRule {
|
||||
r := compiledRule{InterceptRule: db.InterceptRule{
|
||||
ID: 1, Name: "test", Enabled: true, Action: "deny",
|
||||
MatchTarget: target, MatchType: matchType, Pattern: pattern,
|
||||
}}
|
||||
if matchType == "regex" {
|
||||
r.re = regexp.MustCompile(pattern)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// interceptorWith returns an Interceptor whose rule cache is pre-filled, so
|
||||
// Match runs without a database.
|
||||
func interceptorWith(rules ...compiledRule) *Interceptor {
|
||||
return &Interceptor{cached: rules}
|
||||
}
|
||||
|
||||
func toolInput(t *testing.T, v any) []byte {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func TestMatchToolInputSeesDecodedValues(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
pattern string
|
||||
tool string
|
||||
input any
|
||||
}{
|
||||
{"rm -rf on its own line", builtinRmRf, "Bash", map[string]string{"command": "cd /tmp\nrm -rf /var/www"}},
|
||||
{"rm -rf after a tab", builtinRmRf, "Bash", map[string]string{"command": "echo done\trm -rf /var/www"}},
|
||||
{"mkfs on its own line", builtinMkfs, "Bash", map[string]string{"command": "lsblk\nmkfs.ext4 /dev/sdb"}},
|
||||
{"DROP inside a heredoc", builtinDrop, "Bash", map[string]string{"command": "psql <<EOF\nDROP TABLE users;\nEOF"}},
|
||||
{"FLUSHALL inside a heredoc", builtinFlush, "Bash", map[string]string{"command": "redis-cli <<EOF\nFLUSHALL\nEOF"}},
|
||||
{"shutdown on its own line", builtinV2, "Bash", map[string]string{"command": "sync\nshutdown -h now"}},
|
||||
// encoding/json writes > as >, so this only matches the decoded value.
|
||||
{"redirect to a disk", `>\s*/dev/sd`, "Bash", map[string]string{"command": "cat img > /dev/sda"}},
|
||||
{"interactive shell text", builtinRmRf, "shell_send", map[string]string{"text": "ls\nrm -rf /srv/data\n"}},
|
||||
{"nested edit value", builtinDrop, "MultiEdit", map[string]any{
|
||||
"file_path": "/tmp/m.sql",
|
||||
"edits": []map[string]string{{"old_string": "-- x", "new_string": "-- x\nDROP TABLE users;"}},
|
||||
}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", tc.pattern))
|
||||
in := toolInput(t, tc.input)
|
||||
if _, ok := ic.Match(tc.tool, in); !ok {
|
||||
t.Fatalf("rule %q did not match input %s", tc.pattern, in)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputStringRuleSeesDecodedValues(t *testing.T) {
|
||||
// The raw JSON holds rm -rf \"/var/www\", so the quotes only line up once decoded.
|
||||
ic := interceptorWith(testRule("tool_input", "string", `rm -rf "/var/www"`))
|
||||
in := toolInput(t, map[string]string{"command": `rm -rf "/var/www"`})
|
||||
if _, ok := ic.Match("Bash", in); !ok {
|
||||
t.Fatalf("string rule did not match input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
// Rules written against the raw JSON form must keep working.
|
||||
func TestMatchToolInputStillMatchesRawJSON(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", `"command":"rm `))
|
||||
in := toolInput(t, map[string]string{"command": "rm notes.txt"})
|
||||
if _, ok := ic.Match("Bash", in); !ok {
|
||||
t.Fatalf("raw JSON rule did not match input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputInvalidJSONFallsBackToRaw(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", builtinMkfs))
|
||||
if _, ok := ic.Match("Bash", []byte(`not json: mkfs /dev/sdb`)); !ok {
|
||||
t.Fatal("rule did not match non-JSON input")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputNoFalsePositives(t *testing.T) {
|
||||
ic := interceptorWith(
|
||||
testRule("tool_input", "regex", builtinRmRf),
|
||||
testRule("tool_input", "regex", builtinMkfs),
|
||||
testRule("tool_input", "regex", builtinDrop),
|
||||
)
|
||||
for _, cmd := range []string{
|
||||
"cd /tmp\nls -la",
|
||||
"nmap -sV 10.0.0.5\ncurl -s http://10.0.0.5/",
|
||||
"echo firm\nrmdir build", // "firm" and "rmdir" must not look like rm
|
||||
} {
|
||||
in := toolInput(t, map[string]string{"command": cmd})
|
||||
if dec, ok := ic.Match("Bash", in); ok {
|
||||
t.Fatalf("unexpected match %q for input %s", dec.RuleName, in)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The match key ("command") must not be matched as a value.
|
||||
func TestMatchToolInputIgnoresObjectKeys(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", `^mkfs$`))
|
||||
in := toolInput(t, map[string]string{"mkfs": "ls"})
|
||||
if _, ok := ic.Match("Bash", in); ok {
|
||||
t.Fatalf("object key matched as a value for input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolNameUnchanged(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_name", "string", "WebFetch"))
|
||||
if _, ok := ic.Match("WebFetch", []byte(`{"url":"http://x"}`)); !ok {
|
||||
t.Fatal("tool_name rule did not match")
|
||||
}
|
||||
if _, ok := ic.Match("Bash", []byte(`{"command":"WebFetch"}`)); ok {
|
||||
t.Fatal("tool_name rule matched tool input")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
// Package httpidentity defines the explicit identity for target-facing HTTP requests.
|
||||
package httpidentity
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"io"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Keep these 20 standard desktop browser UA samples in sync with the standalone
|
||||
// Python templates. Reviewed 2026-11-11; see HTTP_USER_AGENT.md for versions and
|
||||
// vendor format references. Keep normal reduced platform/version fields and no
|
||||
// tool identifiers, device-specific details or custom suffixes.
|
||||
var userAgents = [...]string{
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15",
|
||||
}
|
||||
|
||||
// UserAgent returns the configured printable ASCII value, or a random browser
|
||||
// sample when unset, empty or malformed. Call once per logical request and reuse
|
||||
// the selected value for redirects and retries.
|
||||
func UserAgent() string {
|
||||
value := strings.Trim(os.Getenv("ARTEX_HTTP_USER_AGENT"), " ")
|
||||
for _, c := range value {
|
||||
if c < 32 || c > 126 {
|
||||
return randomUserAgent(rand.Reader)
|
||||
}
|
||||
}
|
||||
if value != "" {
|
||||
return value
|
||||
}
|
||||
return randomUserAgent(rand.Reader)
|
||||
}
|
||||
|
||||
func randomUserAgent(source io.Reader) string {
|
||||
index, err := rand.Int(source, big.NewInt(int64(len(userAgents))))
|
||||
if err != nil {
|
||||
return userAgents[0] // Keep a valid nonempty UA if system entropy fails.
|
||||
}
|
||||
return userAgents[index.Int64()]
|
||||
}
|
||||
|
||||
// Apply selects an identity for this request; net/http preserves it on redirects.
|
||||
func Apply(req *http.Request) {
|
||||
req.Header.Set("User-Agent", UserAgent())
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package httpidentity
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestApplyOnWire(t *testing.T) {
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{"empty", "", ""},
|
||||
{"configured", " Captured-UA/1.0 ", "Captured-UA/1.0"},
|
||||
{"whitespace", " ", ""},
|
||||
{"injection", "ok\r\nX-Injected: yes", ""},
|
||||
{"non-ascii", "客户端", ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.value)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got := r.UserAgent()
|
||||
if tc.want == "" && !slices.Contains(userAgents[:], got) {
|
||||
t.Errorf("default User-Agent is not a browser sample: %q", got)
|
||||
} else if tc.want != "" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
if r.Header.Get("X-Injected") != "" {
|
||||
t.Error("header injection")
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
req, err := http.NewRequest(http.MethodGet, server.URL, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
Apply(req)
|
||||
resp, err := server.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRandomUserAgentSelection(t *testing.T) {
|
||||
if len(userAgents) != 20 {
|
||||
t.Fatalf("pool size = %d, want 20", len(userAgents))
|
||||
}
|
||||
seen := make(map[string]bool, len(userAgents))
|
||||
// Exercise every choice deterministically without a statistical/flaky test.
|
||||
for index, want := range userAgents {
|
||||
if seen[want] {
|
||||
t.Errorf("duplicate sample: %q", want)
|
||||
}
|
||||
seen[want] = true
|
||||
if got := randomUserAgent(bytes.NewReader([]byte{byte(index)})); got != want {
|
||||
t.Errorf("choice %d = %q, want %q", index, got, want)
|
||||
}
|
||||
}
|
||||
if got := randomUserAgent(bytes.NewReader(nil)); got != userAgents[0] {
|
||||
t.Errorf("entropy failure fallback = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,467 @@
|
||||
// Adapted from github.com/Autumn-27/norma/tool/webfetch.go at v0.4.3.
|
||||
// Keep the SDK tool metadata and fetch/render behavior; use ARTEX's explicit
|
||||
// HTTP identity for both the proxy attempt and its direct fallback. This local
|
||||
// adapter is needed until the SDK exposes request headers in actool.WebFetchConfig.
|
||||
package webfetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
"golang.org/x/net/html"
|
||||
)
|
||||
|
||||
// New keeps the SDK's schema, permission checks and tool metadata.
|
||||
func New(cfg actool.WebFetchConfig) actool.CoreTool {
|
||||
base := actool.NewWebFetch(cfg)
|
||||
return actool.Build(actool.Spec{
|
||||
Name: base.Name(), Description: base.Description(), Prompt: base.Prompt(),
|
||||
Schema: base.InputSchema(), Permissions: base.CheckPermissions,
|
||||
ReadOnly: base.IsReadOnly, Concurrent: base.IsConcurrencySafe,
|
||||
Run: webFetchRunner(cfg),
|
||||
})
|
||||
}
|
||||
|
||||
const maxFetchBytes = 2 << 20 // 2 MiB
|
||||
|
||||
// newFetchClient returns an HTTP client honoring the proxy / TLS config.
|
||||
func newFetchClient(cfg actool.WebFetchConfig) *http.Client {
|
||||
tr := &http.Transport{}
|
||||
if cfg.Proxy != "" {
|
||||
if pu, err := url.Parse(cfg.Proxy); err == nil {
|
||||
tr.Proxy = http.ProxyURL(pu)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case cfg.InsecureTLS:
|
||||
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
|
||||
case cfg.CACert != "":
|
||||
// Trust the proxy's CA in addition to the system roots, so certs it
|
||||
// re-signs verify while every other connection stays fully checked.
|
||||
if pem, err := os.ReadFile(cfg.CACert); err == nil {
|
||||
pool, _ := x509.SystemCertPool()
|
||||
if pool == nil {
|
||||
pool = x509.NewCertPool()
|
||||
}
|
||||
if pool.AppendCertsFromPEM(pem) {
|
||||
tr.TLSClientConfig = &tls.Config{RootCAs: pool}
|
||||
}
|
||||
}
|
||||
}
|
||||
return &http.Client{Transport: tr}
|
||||
}
|
||||
|
||||
// proxyLikelyErr reports whether a fetch error is the kind a recording/MITM proxy
|
||||
// introduces — a closed tunnel (EOF / connection reset), a failed CONNECT, or a
|
||||
// TLS/HTTP2 mismatch — so a direct (no-proxy) retry may succeed. Name-resolution
|
||||
// failures and caller cancellation/timeout are target- or caller-side problems a
|
||||
// direct retry won't fix, so those return false (no retry).
|
||||
func proxyLikelyErr(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
s := strings.ToLower(err.Error())
|
||||
for _, n := range []string{"no such host", "context canceled", "context deadline", "timeout"} {
|
||||
if strings.Contains(s, n) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
for _, p := range []string{
|
||||
"eof", "connection reset", "connection refused", "proxyconnect",
|
||||
"broken pipe", "forcibly closed", "malformed", "http2", "http/2",
|
||||
"tls:", "protocol error",
|
||||
} {
|
||||
if strings.Contains(s, p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// staticAssetExts are file extensions WebFetch refuses: linked resources (scripts,
|
||||
// styles, source maps, images, fonts, media, wasm) that are not page content.
|
||||
var staticAssetExts = map[string]bool{
|
||||
".js": true, ".mjs": true, ".cjs": true, ".css": true, ".map": true,
|
||||
".png": true, ".jpg": true, ".jpeg": true, ".gif": true, ".svg": true,
|
||||
".webp": true, ".avif": true, ".ico": true, ".bmp": true,
|
||||
".woff": true, ".woff2": true, ".ttf": true, ".otf": true, ".eot": true,
|
||||
".wasm": true, ".mp4": true, ".webm": true, ".mp3": true, ".wav": true, ".ogg": true,
|
||||
}
|
||||
|
||||
// staticAssetExt returns the offending extension (with dot) if rawURL points at a
|
||||
// static asset, or "" otherwise. The query string and fragment are ignored, so
|
||||
// "/app.js?v=3" is still caught.
|
||||
func staticAssetExt(rawURL string) string {
|
||||
path := rawURL
|
||||
if u, err := url.Parse(rawURL); err == nil {
|
||||
path = u.Path
|
||||
}
|
||||
if i := strings.LastIndexByte(path, '/'); i >= 0 {
|
||||
path = path[i+1:] // last segment only
|
||||
}
|
||||
dot := strings.LastIndexByte(path, '.')
|
||||
if dot < 0 {
|
||||
return ""
|
||||
}
|
||||
if ext := strings.ToLower(path[dot:]); staticAssetExts[ext] {
|
||||
return ext
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func webFetchRunner(cfg actool.WebFetchConfig) func(context.Context, json.RawMessage, *actool.ToolContext) (actool.Result, error) {
|
||||
client := newFetchClient(cfg)
|
||||
// direct is the no-proxy fallback used when a proxied fetch fails for a
|
||||
// proxy-caused reason (closed tunnel / failed CONNECT / TLS or HTTP2 mismatch):
|
||||
// it bypasses the configured proxy so a recording/MITM hiccup doesn't fail an
|
||||
// otherwise-reachable URL. nil when no proxy is configured (nothing to bypass).
|
||||
var direct *http.Client
|
||||
if cfg.Proxy != "" {
|
||||
direct = newFetchClient(actool.WebFetchConfig{InsecureTLS: cfg.InsecureTLS})
|
||||
}
|
||||
return func(ctx context.Context, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
|
||||
var in struct {
|
||||
URL string `json:"url"`
|
||||
Extract bool `json:"extract"`
|
||||
}
|
||||
if err := json.Unmarshal(input, &in); err != nil {
|
||||
return actool.Result{}, err
|
||||
}
|
||||
if !strings.HasPrefix(in.URL, "http://") && !strings.HasPrefix(in.URL, "https://") {
|
||||
return actool.Errorf("Error: url must start with http:// or https://"), nil
|
||||
}
|
||||
if ext := staticAssetExt(in.URL); ext != "" {
|
||||
return actool.Errorf("Error: WebFetch does not fetch static assets (" + ext + " file). It reads HTML pages, documentation, and API responses — not linked resources like scripts, styles, images, or fonts. Fetch the page that references this asset instead."), nil
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil)
|
||||
if err != nil {
|
||||
return actool.Errorf("Error: " + err.Error()), nil
|
||||
}
|
||||
userAgent := httpidentity.UserAgent()
|
||||
req.Header.Set("User-Agent", userAgent)
|
||||
resp, err := client.Do(req)
|
||||
// One direct retry when the proxied attempt failed for a proxy-caused reason.
|
||||
// A non-proxy error (bad host, caller timeout/cancel) is returned as-is — a
|
||||
// direct retry wouldn't fix it. The direct response is NOT captured by the proxy.
|
||||
if err != nil && direct != nil && proxyLikelyErr(err) {
|
||||
if r2, e2 := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil); e2 == nil {
|
||||
r2.Header.Set("User-Agent", userAgent)
|
||||
if resp2, err2 := direct.Do(r2); err2 == nil {
|
||||
resp, err = resp2, nil
|
||||
} else {
|
||||
err = err2 // surface the direct failure (more representative of the target)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return actool.Errorf("Error fetching URL: " + err.Error()), nil
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, maxFetchBytes))
|
||||
|
||||
header := fmt.Sprintf("[%d %s] %s\n", resp.StatusCode, http.StatusText(resp.StatusCode), in.URL)
|
||||
// Cheap fingerprint signal that HTML→Markdown would otherwise hide.
|
||||
if sv := resp.Header.Get("Server"); sv != "" {
|
||||
header += "Server: " + sv + "\n"
|
||||
}
|
||||
if xp := resp.Header.Get("X-Powered-By"); xp != "" {
|
||||
header += "X-Powered-By: " + xp + "\n"
|
||||
}
|
||||
|
||||
var out string
|
||||
if strings.Contains(strings.ToLower(resp.Header.Get("content-type")), "html") {
|
||||
doc, perr := html.Parse(strings.NewReader(string(body)))
|
||||
if perr != nil { // html.Parse is lenient; this is effectively unreachable
|
||||
out = string(body)
|
||||
} else {
|
||||
out = htmlToMarkdown(doc)
|
||||
if in.Extract {
|
||||
if recon := extractRecon(doc); recon != "" {
|
||||
out += "\n\n" + recon
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
out = string(body)
|
||||
}
|
||||
return actool.Text(truncate(header+strings.TrimSpace(out), 50000)), nil
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
reWS = regexp.MustCompile(`[ \t]+`)
|
||||
reLineTrim = regexp.MustCompile(` *\n *`) // strip spaces hugging a newline
|
||||
reBlankLines = regexp.MustCompile(`\n{2,}`) // collapse blank lines → one newline
|
||||
// reEndpoint matches quoted absolute URLs and root-relative paths in inline JS.
|
||||
reEndpoint = regexp.MustCompile("[\"'`](https?://[^\"'`\\s]+|/[A-Za-z0-9._~$&'()*+,;=:@%/?#\\[\\]-]{2,})[\"'`]")
|
||||
)
|
||||
|
||||
// skipMarkdown are elements whose subtree carries no readable prose.
|
||||
var skipMarkdown = map[string]bool{"script": true, "style": true, "noscript": true, "svg": true, "template": true}
|
||||
|
||||
// htmlToMarkdown renders the visible content of a parsed document as Markdown.
|
||||
// Text nodes are already entity-decoded by the parser, so no unescaping is needed.
|
||||
func htmlToMarkdown(doc *html.Node) string {
|
||||
var b strings.Builder
|
||||
renderNode(&b, doc)
|
||||
s := reWS.ReplaceAllString(b.String(), " ")
|
||||
s = reLineTrim.ReplaceAllString(s, "\n") // drop indentation / trailing spaces per line
|
||||
s = reBlankLines.ReplaceAllString(s, "\n") // no blank lines — compact, token-lean output
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
func renderNode(b *strings.Builder, n *html.Node) {
|
||||
switch n.Type {
|
||||
case html.TextNode:
|
||||
b.WriteString(strings.ReplaceAll(n.Data, "\n", " "))
|
||||
return
|
||||
case html.ElementNode:
|
||||
if skipMarkdown[n.Data] {
|
||||
return
|
||||
}
|
||||
default:
|
||||
if n.Type != html.DocumentNode {
|
||||
return
|
||||
}
|
||||
}
|
||||
tag := ""
|
||||
if n.Type == html.ElementNode {
|
||||
tag = n.Data
|
||||
}
|
||||
switch tag {
|
||||
case "h1", "h2", "h3", "h4", "h5", "h6":
|
||||
b.WriteString("\n\n" + strings.Repeat("#", int(tag[1]-'0')) + " ")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "title":
|
||||
b.WriteString("\n\n# ")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "p", "div", "section", "article", "header", "footer", "main", "ul", "ol", "table", "tr", "blockquote":
|
||||
b.WriteString("\n\n")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "br":
|
||||
b.WriteString("\n")
|
||||
case "hr":
|
||||
b.WriteString("\n\n---\n\n")
|
||||
case "li":
|
||||
b.WriteString("\n- ")
|
||||
renderChildren(b, n)
|
||||
case "a":
|
||||
var inner strings.Builder
|
||||
renderChildren(&inner, n)
|
||||
text := strings.TrimSpace(inner.String())
|
||||
if href := attr(n, "href"); href != "" && text != "" {
|
||||
fmt.Fprintf(b, "[%s](%s)", text, href)
|
||||
} else {
|
||||
b.WriteString(text)
|
||||
}
|
||||
case "img":
|
||||
if src := attr(n, "src"); src != "" {
|
||||
fmt.Fprintf(b, "", attr(n, "alt"), src)
|
||||
}
|
||||
case "strong", "b":
|
||||
b.WriteString("**")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("**")
|
||||
case "em", "i":
|
||||
b.WriteString("*")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("*")
|
||||
case "code":
|
||||
b.WriteString("`")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("`")
|
||||
case "pre":
|
||||
var inner strings.Builder
|
||||
textOnly(&inner, n)
|
||||
b.WriteString("\n\n```\n" + strings.TrimSpace(inner.String()) + "\n```\n\n")
|
||||
default:
|
||||
renderChildren(b, n)
|
||||
}
|
||||
}
|
||||
|
||||
func renderChildren(b *strings.Builder, n *html.Node) {
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
renderNode(b, c)
|
||||
}
|
||||
}
|
||||
|
||||
// extractRecon walks the document for security-relevant signal that Markdown
|
||||
// rendering drops: comments, script sources, inline endpoints, forms, meta, links.
|
||||
func extractRecon(doc *html.Node) string {
|
||||
var comments, scriptSrcs, inlineEndpoints, forms, metas, links []string
|
||||
seenLink := map[string]bool{}
|
||||
|
||||
var walk func(n *html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
switch n.Type {
|
||||
case html.CommentNode:
|
||||
if c := oneLine(n.Data, 300); c != "" {
|
||||
comments = append(comments, c)
|
||||
}
|
||||
case html.ElementNode:
|
||||
switch n.Data {
|
||||
case "script":
|
||||
if src := attr(n, "src"); src != "" {
|
||||
scriptSrcs = append(scriptSrcs, src)
|
||||
} else {
|
||||
var body strings.Builder
|
||||
textOnly(&body, n)
|
||||
for _, m := range reEndpoint.FindAllStringSubmatch(body.String(), 40) {
|
||||
inlineEndpoints = append(inlineEndpoints, m[1])
|
||||
}
|
||||
}
|
||||
case "form":
|
||||
forms = append(forms, describeForm(n))
|
||||
case "meta":
|
||||
name := attr(n, "name")
|
||||
if name == "" {
|
||||
name = attr(n, "property")
|
||||
}
|
||||
if content := attr(n, "content"); name != "" && content != "" {
|
||||
metas = append(metas, name+"="+oneLine(content, 120))
|
||||
}
|
||||
case "a", "link":
|
||||
if h := attr(n, "href"); h != "" && !seenLink[h] {
|
||||
seenLink[h] = true
|
||||
links = append(links, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(doc)
|
||||
|
||||
var b strings.Builder
|
||||
b.WriteString("=== RECON ===")
|
||||
reconSection(&b, "COMMENTS", comments)
|
||||
reconSection(&b, "SCRIPTS (src)", dedup(scriptSrcs))
|
||||
reconSection(&b, "INLINE ENDPOINTS", dedup(inlineEndpoints))
|
||||
reconSection(&b, "FORMS", forms)
|
||||
reconSection(&b, "META", metas)
|
||||
reconSection(&b, "LINKS", capSlice(links, 100))
|
||||
if b.Len() == len("=== RECON ===") {
|
||||
return "" // nothing worth reporting
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// describeForm summarizes a <form> as "METHOD action → name(type[=value]) …".
|
||||
// Hidden inputs keep their value (CSRF tokens, parameter defaults matter in recon).
|
||||
func describeForm(form *html.Node) string {
|
||||
method := strings.ToUpper(attr(form, "method"))
|
||||
if method == "" {
|
||||
method = "GET"
|
||||
}
|
||||
var inputs []string
|
||||
var walk func(n *html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
if n.Type == html.ElementNode {
|
||||
switch n.Data {
|
||||
case "input", "textarea", "select":
|
||||
typ := attr(n, "type")
|
||||
if typ == "" {
|
||||
typ = n.Data
|
||||
}
|
||||
desc := attr(n, "name") + "(" + typ
|
||||
if typ == "hidden" {
|
||||
if v := attr(n, "value"); v != "" {
|
||||
desc += "=" + oneLine(v, 60)
|
||||
}
|
||||
}
|
||||
inputs = append(inputs, desc+")")
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(form)
|
||||
return fmt.Sprintf("%s %s → %s", method, attr(form, "action"), strings.Join(inputs, " "))
|
||||
}
|
||||
|
||||
// --- small helpers ---
|
||||
|
||||
func attr(n *html.Node, key string) string {
|
||||
for _, a := range n.Attr {
|
||||
if a.Key == key {
|
||||
return a.Val
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func textOnly(b *strings.Builder, n *html.Node) {
|
||||
if n.Type == html.TextNode {
|
||||
b.WriteString(n.Data)
|
||||
return
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
textOnly(b, c)
|
||||
}
|
||||
}
|
||||
|
||||
func oneLine(s string, max int) string {
|
||||
s = strings.Join(strings.Fields(s), " ")
|
||||
if len(s) > max {
|
||||
return s[:max] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func reconSection(b *strings.Builder, title string, items []string) {
|
||||
if len(items) == 0 {
|
||||
return
|
||||
}
|
||||
b.WriteString("\n\n## " + title + "\n")
|
||||
for _, it := range items {
|
||||
b.WriteString("- " + it + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
func dedup(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := in[:0:0]
|
||||
for _, s := range in {
|
||||
if s == "" || seen[s] {
|
||||
continue
|
||||
}
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func capSlice(in []string, n int) []string {
|
||||
if len(in) <= n {
|
||||
return in
|
||||
}
|
||||
return append(in[:n:n], fmt.Sprintf("… (%d more)", len(in)-n))
|
||||
}
|
||||
|
||||
// truncate retains the SDK's head/tail output limit.
|
||||
func truncate(s string, max int) string {
|
||||
if max <= 0 || len(s) <= max {
|
||||
return s
|
||||
}
|
||||
half := max / 2
|
||||
return s[:half] + fmt.Sprintf("\n\n... [%d characters truncated] ...\n\n", len(s)-max) + s[len(s)-half:]
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
package webfetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
|
||||
func callFetch(t *testing.T, tool actool.CoreTool, target string, extract bool) actool.Result {
|
||||
t.Helper()
|
||||
input, err := json.Marshal(map[string]any{"url": target, "extract": extract})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := tool.Call(context.Background(), input, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestIdentityAcrossRoutes(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
for _, route := range []string{"direct", "proxy", "fallback"} {
|
||||
t.Run(route+"/"+ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
var targetCalls, proxyCalls atomic.Int32
|
||||
var mu sync.Mutex
|
||||
var selected string
|
||||
checkUA := func(r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
got := r.UserAgent()
|
||||
if ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
if selected != "" && selected != got {
|
||||
t.Errorf("redirect/retry UA changed from %q to %q", selected, got)
|
||||
}
|
||||
selected = got
|
||||
}
|
||||
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
targetCalls.Add(1)
|
||||
checkUA(r)
|
||||
if r.URL.Path == "/start" {
|
||||
http.Redirect(w, r, "/page", http.StatusFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
fmt.Fprint(w, `<h1>Page</h1><!--keep--><script src="/app.js"></script><form action="/submit"><input name="csrf" type="hidden" value="token"></form>`)
|
||||
}))
|
||||
defer target.Close()
|
||||
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
proxyCalls.Add(1)
|
||||
checkUA(r)
|
||||
if route == "fallback" {
|
||||
conn, _, err := w.(http.Hijacker).Hijack()
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
conn.Close() // Deterministic EOF, causing the SDK-compatible retry.
|
||||
return
|
||||
}
|
||||
req := r.Clone(r.Context())
|
||||
req.RequestURI = ""
|
||||
// A forwarding proxy must preserve absence, like ARTEX's sink.
|
||||
if _, present := req.Header["User-Agent"]; !present {
|
||||
req.Header.Set("User-Agent", "")
|
||||
}
|
||||
resp, err := target.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
for key, values := range resp.Header {
|
||||
w.Header()[key] = values
|
||||
}
|
||||
w.WriteHeader(resp.StatusCode)
|
||||
io.Copy(w, resp.Body)
|
||||
}))
|
||||
defer proxy.Close()
|
||||
cfg := actool.WebFetchConfig{}
|
||||
if route != "direct" {
|
||||
cfg.Proxy = proxy.URL
|
||||
}
|
||||
result := callFetch(t, New(cfg), target.URL+"/start", true)
|
||||
for _, want := range []string{"[200 OK]", "# Page", "=== RECON ===", "keep", "/app.js", "csrf(hidden=token)"} {
|
||||
if result.IsError || !strings.Contains(result.Flatten(), want) {
|
||||
t.Fatalf("missing %q in result: %+v", want, result)
|
||||
}
|
||||
}
|
||||
if targetCalls.Load() != 2 {
|
||||
t.Errorf("target calls = %d, want 2", targetCalls.Load())
|
||||
}
|
||||
if route != "direct" && proxyCalls.Load() == 0 {
|
||||
t.Error("configured proxy was bypassed")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTLSAndStaticAssetGuard(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", "Captured-UA/1.0")
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
if r.UserAgent() != "Captured-UA/1.0" {
|
||||
t.Errorf("User-Agent = %q", r.UserAgent())
|
||||
}
|
||||
fmt.Fprint(w, "verified")
|
||||
}))
|
||||
defer server.Close()
|
||||
certPath := filepath.Join(t.TempDir(), "ca.pem")
|
||||
cert := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})
|
||||
if err := os.WriteFile(certPath, cert, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tool := New(actool.WebFetchConfig{CACert: certPath})
|
||||
if result := callFetch(t, tool, server.URL, false); result.IsError || !strings.Contains(result.Flatten(), "verified") {
|
||||
t.Fatalf("CA trust lost: %+v", result)
|
||||
}
|
||||
if result := callFetch(t, tool, server.URL+"/app.js?v=1", false); !result.IsError {
|
||||
t.Fatalf("static asset guard lost: %+v", result)
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("static asset triggered network request: calls = %d", calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentitySelectedForEachCall(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", "Before-Tool-Creation")
|
||||
tool := New(actool.WebFetchConfig{})
|
||||
seen := make(chan string, 2)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seen <- r.UserAgent()
|
||||
fmt.Fprint(w, "ok")
|
||||
}))
|
||||
defer server.Close()
|
||||
for _, want := range []string{"First-Call", "Second-Call"} {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", want)
|
||||
if result := callFetch(t, tool, server.URL, false); result.IsError {
|
||||
t.Fatal(result.Flatten())
|
||||
}
|
||||
if got := <-seen; got != want {
|
||||
t.Fatalf("identity cached across calls: got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSDKMetadataPreserved(t *testing.T) {
|
||||
base := actool.NewWebFetch(actool.WebFetchConfig{})
|
||||
local := New(actool.WebFetchConfig{})
|
||||
if local.Name() != base.Name() || local.Description() != base.Description() || !reflect.DeepEqual(local.InputSchema(), base.InputSchema()) {
|
||||
t.Fatal("SDK tool contract changed")
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
// Repo 是发布源。写死而不是做成配置项:更新源可配等于给任何能改配置的人一条
|
||||
@@ -125,7 +127,7 @@ func FetchLatest(ctx context.Context, c *http.Client) (*Release, error) {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Accept", "application/vnd.github+json")
|
||||
req.Header.Set("User-Agent", "artex-selfupdate")
|
||||
httpidentity.Apply(req)
|
||||
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
|
||||
@@ -56,3 +56,38 @@ func TestFetchLatestMissingRelease(t *testing.T) {
|
||||
t.Fatalf("unexpected missing-release error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpdateUserAgent(t *testing.T) {
|
||||
for _, tc := range []struct{ env, want string }{
|
||||
{"", ""},
|
||||
{"Captured-UA/1.0", "Captured-UA/1.0"},
|
||||
} {
|
||||
t.Run(tc.env, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
|
||||
calls := 0
|
||||
client := &http.Client{Transport: releaseTransport(func(req *http.Request) (*http.Response, error) {
|
||||
calls++
|
||||
if got := req.UserAgent(); tc.want == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if tc.want != "" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
body := `{"tag_name":"v0.3.17"}`
|
||||
return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(body))}, nil
|
||||
})}
|
||||
if _, err := FetchLatest(context.Background(), client); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, name := range []string{"SHA256SUMS", "artex.zip"} {
|
||||
body, err := get(context.Background(), client, "https://github.com/RuoJi6/ARTEX/releases/download/v0.3.17/"+name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body.Close()
|
||||
}
|
||||
if calls != 3 {
|
||||
t.Fatalf("calls = %d, want 3", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -13,6 +13,8 @@ import (
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
// sumsAsset 是 release.yml 生成的校验和清单,覆盖 Release 里全部 zip。
|
||||
@@ -229,7 +231,7 @@ func get(ctx context.Context, c *http.Client, rawURL string) (io.ReadCloser, err
|
||||
if err := checkURL(req.URL); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-selfupdate")
|
||||
httpidentity.Apply(req)
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -18,6 +18,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
"github.com/Autumn-27/norma/permission"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
@@ -438,6 +439,7 @@ func (s *Server) runHTTPTool(ctx context.Context, execRaw json.RawMessage, param
|
||||
if err != nil {
|
||||
return actool.Errorf(maskSecrets(err.Error())), nil
|
||||
}
|
||||
httpidentity.Apply(req)
|
||||
for k, v := range spec.Headers {
|
||||
req.Header.Set(k, renderTemplate(v, params, identity))
|
||||
}
|
||||
|
||||
@@ -23,6 +23,7 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
"github.com/Autumn-27/artex/traffic"
|
||||
)
|
||||
|
||||
@@ -305,7 +306,7 @@ func probeOne(ctx context.Context, client *http.Client, target string) (string,
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-egress-probe/1.0")
|
||||
httpidentity.Apply(req)
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
|
||||
@@ -27,6 +27,27 @@ func fakeIPServer(ip string) *httptest.Server {
|
||||
}))
|
||||
}
|
||||
|
||||
func TestProbeOneUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
fmt.Fprint(w, `{"ip":"203.0.113.7"}`)
|
||||
}))
|
||||
defer server.Close()
|
||||
ip, err := probeOne(context.Background(), server.Client(), server.URL)
|
||||
if err != nil || ip != "203.0.113.7" {
|
||||
t.Fatalf("ip = %q, err = %v", ip, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestProbeEgress 覆盖 probeEgress 的输入/响应形态:成功、非法协议、死代理、坏响应、回落。
|
||||
// 每条用例自带服务端与断言,失败时 t.Run 的名字直接指出是哪一条。
|
||||
func TestProbeEgress(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTargetHTTPIdentityOverrides(t *testing.T) {
|
||||
t.Setenv(repeaterAllowLocalEnv, "1")
|
||||
for _, route := range []string{"custom", "repeater"} {
|
||||
for _, tc := range []struct {
|
||||
name, env, want string
|
||||
headers map[string]string
|
||||
}{
|
||||
{name: "default"},
|
||||
{name: "environment", env: "Captured-UA/1.0", want: "Captured-UA/1.0"},
|
||||
{name: "override", env: "Environment-UA/1.0", want: "Explicit-UA/1.0", headers: map[string]string{"user-agent": "Explicit-UA/1.0"}},
|
||||
{name: "explicit-empty", env: "Environment-UA/1.0", headers: map[string]string{"User-Agent": ""}},
|
||||
} {
|
||||
t.Run(route+"/"+tc.name, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); tc.name == "default" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if tc.name != "default" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
if _, present := r.Header["User-Agent"]; tc.name == "explicit-empty" && present {
|
||||
t.Error("empty User-Agent present on wire")
|
||||
}
|
||||
w.Write([]byte("ok"))
|
||||
}))
|
||||
defer server.Close()
|
||||
payload := map[string]any{"url": server.URL, "headers": tc.headers, "use_recording_proxy": false}
|
||||
if route == "custom" {
|
||||
raw, _ := json.Marshal(payload)
|
||||
result, err := (&Server{}).runHTTPTool(context.Background(), raw, nil, nil)
|
||||
if err != nil || result.IsError {
|
||||
t.Fatalf("custom tool: %v %+v", err, result)
|
||||
}
|
||||
} else {
|
||||
status, result, body := repeaterTestSend(t, &Server{}, payload)
|
||||
if status != http.StatusOK || result.Status != http.StatusOK {
|
||||
t.Fatalf("repeater: %d %s", status, body)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,8 @@ import (
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -141,6 +143,7 @@ func (s *Server) sendRepeater(w http.ResponseWriter, r *http.Request) {
|
||||
writeErr(w, http.StatusBadRequest, "构造请求失败")
|
||||
return
|
||||
}
|
||||
httpidentity.Apply(req)
|
||||
for key, value := range in.Headers {
|
||||
name := strings.TrimSpace(key)
|
||||
if name == "" {
|
||||
|
||||
@@ -825,6 +825,11 @@ func (s *Server) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /api/update/rollback", s.updateRollback)
|
||||
mux.HandleFunc("GET /api/update/stream", s.updateStream)
|
||||
|
||||
// 一键备份 / 恢复(迁移数据)。恢复会替换全部数据并重启,同样只对已登录管理员开放。
|
||||
mux.HandleFunc("GET /api/system/backup/status", s.backupStatus)
|
||||
mux.HandleFunc("GET /api/system/backup/export", s.backupExport)
|
||||
mux.HandleFunc("POST /api/system/backup/restore", s.backupRestore)
|
||||
|
||||
mux.HandleFunc("GET /api/tasks", s.listTasks)
|
||||
mux.HandleFunc("POST /api/tasks", s.createTask)
|
||||
mux.HandleFunc("GET /api/schedules", s.listSchedules)
|
||||
|
||||
@@ -0,0 +1,855 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
pgdb "github.com/Autumn-27/artex/db"
|
||||
"github.com/klauspost/compress/zstd"
|
||||
)
|
||||
|
||||
// 一键备份 / 恢复(迁移数据)。
|
||||
//
|
||||
// 备份包是 tar.zst,结构:
|
||||
//
|
||||
// db/<table>.copy PostgreSQL 各表的 COPY 文本数据(大表按分片追加 .000001…)
|
||||
// files/<rel> 数据目录(任务工作区、会话、流量、证据、归档包、MITM CA…)
|
||||
// skills/<name>/… skill 目录
|
||||
// manifest.json 最后写入:表结构、分片与行数、来源数据目录、顶层条目清单
|
||||
//
|
||||
// 恢复分两步:上传时只解包到 <data>/.restore-pending 并对照当前 schema 校验,
|
||||
// 然后请求重启;真正的替换在下次启动、任何组件打开数据之前由 ApplyPendingRestore
|
||||
// 完成——运行中的 Agent、流量代理和 SQLite 索引都还没碰数据目录,替换才安全。
|
||||
// 数据库在一个事务内整体替换,失败则保持原数据不动。
|
||||
|
||||
const (
|
||||
backupFormat = 1
|
||||
backupManifestName = "manifest.json"
|
||||
backupPendingDir = ".restore-pending"
|
||||
backupUploadPrefix = ".restore-upload-"
|
||||
backupPreviousDir = ".restore-previous"
|
||||
backupFailedDir = ".restore-failed"
|
||||
backupDBDoneMarker = ".db-restored"
|
||||
backupRestoreIDFile = ".restore-id"
|
||||
backupMaxFiles = 20_000_000
|
||||
backupTrafficIndex = "traffic/_index" // live SQLite; exported via SnapshotIndex
|
||||
)
|
||||
|
||||
// backupCopyChunkBytes bounds each streamed DB COPY tar entry so large tables
|
||||
// flush compressed bytes to the client without disk spooling or proxy timeouts.
|
||||
var backupCopyChunkBytes = 8 << 20
|
||||
|
||||
// backupSkipped lists data-dir paths (slash form, relative) that hold transient
|
||||
// staging state rather than data. Top-level dot entries are skipped as well.
|
||||
var backupSkipped = map[string]bool{
|
||||
"traffic/_delete_staging": true,
|
||||
"archives/tasks/.staging": true,
|
||||
"archives/tasks/.restore": true,
|
||||
"evidence/.staging": true,
|
||||
"traffic/_index/.snapshot": true,
|
||||
}
|
||||
|
||||
type backupTableEntry struct {
|
||||
pgdb.BackupTable
|
||||
Parts int `json:"parts,omitempty"`
|
||||
}
|
||||
|
||||
type backupManifest struct {
|
||||
Format int `json:"format"`
|
||||
Version string `json:"version"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
DataDir string `json:"data_dir"`
|
||||
Separator string `json:"separator"`
|
||||
Tables []backupTableEntry `json:"tables"`
|
||||
Entries []string `json:"entries"`
|
||||
Skills []string `json:"skills"`
|
||||
}
|
||||
|
||||
// backupMu serializes export and restore staging: two concurrent restores
|
||||
// would race on the pending directory, and an export racing a restore upload
|
||||
// is never what the operator meant.
|
||||
var backupMu sync.Mutex
|
||||
|
||||
// BackupRestoreResult is the outcome of the boot-time restore, reported to the
|
||||
// UI after the restart that applied it.
|
||||
type BackupRestoreResult struct {
|
||||
ID string `json:"id,omitempty"`
|
||||
Applied bool `json:"applied"`
|
||||
Error string `json:"error,omitempty"`
|
||||
At time.Time `json:"at"`
|
||||
Source string `json:"source_version,omitempty"`
|
||||
Created time.Time `json:"backup_created_at,omitempty"`
|
||||
}
|
||||
|
||||
var (
|
||||
bootRestoreMu sync.Mutex
|
||||
bootRestore *BackupRestoreResult
|
||||
)
|
||||
|
||||
// SetBootRestoreResult is called once by main with ApplyPendingRestore's result.
|
||||
func SetBootRestoreResult(result *BackupRestoreResult) {
|
||||
bootRestoreMu.Lock()
|
||||
defer bootRestoreMu.Unlock()
|
||||
bootRestore = result
|
||||
}
|
||||
|
||||
func bootRestoreResult() *BackupRestoreResult {
|
||||
bootRestoreMu.Lock()
|
||||
defer bootRestoreMu.Unlock()
|
||||
return bootRestore
|
||||
}
|
||||
|
||||
// GET /api/system/backup/status
|
||||
func (s *Server) backupStatus(w http.ResponseWriter, r *http.Request) {
|
||||
_, err := os.Stat(filepath.Join(s.m.dir, backupPendingDir))
|
||||
writeJSON(w, 200, map[string]any{"pending": err == nil, "last_restore": bootRestoreResult()})
|
||||
}
|
||||
|
||||
// GET|HEAD /api/system/backup/export streams the backup package. Errors after
|
||||
// the first byte abort the connection so a partial file never looks complete.
|
||||
func (s *Server) backupExport(w http.ResponseWriter, r *http.Request) {
|
||||
if s.pg(w) == nil {
|
||||
return
|
||||
}
|
||||
if !backupMu.TryLock() {
|
||||
w.Header().Set("X-Artex-Error-Code", "backup_busy")
|
||||
writeProductError(w, 409, "backup_busy", "已有备份或恢复正在进行", nil)
|
||||
return
|
||||
}
|
||||
defer backupMu.Unlock()
|
||||
active, err := s.m.pg.CountActiveTasks(r.Context())
|
||||
if err != nil {
|
||||
log.Printf("[backup] 检查运行中任务失败:%v", err)
|
||||
writeProductError(w, 500, "data_unavailable", "检查运行中任务失败", nil)
|
||||
return
|
||||
}
|
||||
if active > 0 {
|
||||
w.Header().Set("X-Artex-Error-Code", "backup_active_tasks")
|
||||
writeProductError(w, 409, "backup_active_tasks", "存在未暂停的运行中或排队中任务,请先暂停或等待任务完成后再下载备份", map[string]any{"count": active})
|
||||
return
|
||||
}
|
||||
name := "artex-backup-" + time.Now().Format("20060102-150405") + ".tar.zst"
|
||||
w.Header().Set("Content-Type", "application/zstd")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
setFileDisposition(w, "attachment", name)
|
||||
if r.Method == http.MethodHead {
|
||||
return
|
||||
}
|
||||
started := time.Now()
|
||||
if err := s.writeBackup(r.Context(), w); err != nil {
|
||||
log.Printf("[backup] 导出失败:%v", err)
|
||||
panic(http.ErrAbortHandler)
|
||||
}
|
||||
log.Printf("[backup] 已导出备份 %s(%s)", name, time.Since(started).Round(time.Millisecond))
|
||||
}
|
||||
|
||||
func (s *Server) writeBackup(ctx context.Context, out io.Writer) error {
|
||||
encoder, err := zstd.NewWriter(out, zstd.WithEncoderLevel(zstd.SpeedDefault), zstd.WithEncoderCRC(true))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
flusher, _ := out.(http.Flusher)
|
||||
tw := tar.NewWriter(encoder)
|
||||
manifest := backupManifest{
|
||||
Format: backupFormat, Version: BuildVersion, CreatedAt: time.Now().UTC(),
|
||||
DataDir: s.m.dir, Separator: string(filepath.Separator), Entries: []string{}, Skills: []string{},
|
||||
}
|
||||
chunkMax := backupCopyChunkBytes
|
||||
if chunkMax <= 0 {
|
||||
chunkMax = 8 << 20
|
||||
}
|
||||
var copyBuf []byte
|
||||
err = s.m.pg.DumpTables(ctx, func(table pgdb.BackupTable, copyTo func(io.Writer) (int64, error)) error {
|
||||
cw := copyChunkWriter{
|
||||
tw: tw,
|
||||
encoder: encoder,
|
||||
flusher: flusher,
|
||||
table: table.Name,
|
||||
modTime: manifest.CreatedAt,
|
||||
buf: copyBuf[:0],
|
||||
chunkMax: chunkMax,
|
||||
}
|
||||
rows, copyErr := copyTo(&cw)
|
||||
if copyErr != nil {
|
||||
return copyErr
|
||||
}
|
||||
parts, err := cw.finish()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
copyBuf = cw.buf[:0]
|
||||
table.Rows = rows
|
||||
entry := backupTableEntry{BackupTable: table}
|
||||
if parts > 1 {
|
||||
entry.Parts = parts
|
||||
}
|
||||
manifest.Tables = append(manifest.Tables, entry)
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
snapshot := ""
|
||||
if tr := s.m.Traffic(); tr != nil {
|
||||
spool, err := os.MkdirTemp(s.m.dir, ".backup-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(spool)
|
||||
snapshot = filepath.Join(spool, "index.sqlite")
|
||||
if err := tr.SnapshotIndex(snapshot); err != nil {
|
||||
return fmt.Errorf("快照流量索引: %w", err)
|
||||
}
|
||||
if err := addBackupFile(tw, snapshot, "files/"+backupTrafficIndex+"/index.sqlite"); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := encoder.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if flusher != nil {
|
||||
flusher.Flush()
|
||||
}
|
||||
}
|
||||
err = walkBackupTree(s.m.dir, func(relative string, info fs.FileInfo) (bool, error) {
|
||||
if !strings.Contains(relative, "/") {
|
||||
if strings.HasPrefix(relative, ".") {
|
||||
return false, nil
|
||||
}
|
||||
manifest.Entries = append(manifest.Entries, relative)
|
||||
}
|
||||
if backupSkipped[relative] || (snapshot != "" && relative == backupTrafficIndex) {
|
||||
return false, nil
|
||||
}
|
||||
return true, addBackupEntry(tw, filepath.Join(s.m.dir, filepath.FromSlash(relative)), "files/"+relative, info)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.skillDir != "" {
|
||||
err = walkBackupTree(s.skillDir, func(relative string, info fs.FileInfo) (bool, error) {
|
||||
if strings.HasPrefix(relative, ".") {
|
||||
return false, nil
|
||||
}
|
||||
if !strings.Contains(relative, "/") {
|
||||
if !info.IsDir() {
|
||||
return false, nil
|
||||
}
|
||||
manifest.Skills = append(manifest.Skills, relative)
|
||||
}
|
||||
return true, addBackupEntry(tw, filepath.Join(s.skillDir, filepath.FromSlash(relative)), "skills/"+relative, info)
|
||||
})
|
||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
}
|
||||
raw, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: backupManifestName, Mode: archiveFileMode, Size: int64(len(raw)), ModTime: manifest.CreatedAt, Typeflag: tar.TypeReg}); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tw.Write(raw); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return encoder.Close()
|
||||
}
|
||||
|
||||
func backupTableCopyPath(table string, part int) string {
|
||||
if part == 0 {
|
||||
return "db/" + table + ".copy"
|
||||
}
|
||||
return fmt.Sprintf("db/%s.copy.%06d", table, part)
|
||||
}
|
||||
|
||||
func backupTablesOnly(entries []backupTableEntry) []pgdb.BackupTable {
|
||||
tables := make([]pgdb.BackupTable, len(entries))
|
||||
for i, entry := range entries {
|
||||
tables[i] = entry.BackupTable
|
||||
}
|
||||
return tables
|
||||
}
|
||||
|
||||
type copyChunkWriter struct {
|
||||
tw *tar.Writer
|
||||
encoder *zstd.Encoder
|
||||
flusher http.Flusher
|
||||
table string
|
||||
modTime time.Time
|
||||
buf []byte
|
||||
chunkMax int
|
||||
parts int
|
||||
}
|
||||
|
||||
func (w *copyChunkWriter) Write(p []byte) (int, error) {
|
||||
total := len(p)
|
||||
for len(p) > 0 {
|
||||
space := w.chunkMax - len(w.buf)
|
||||
n := min(space, len(p))
|
||||
w.buf = append(w.buf, p[:n]...)
|
||||
p = p[n:]
|
||||
if len(w.buf) == w.chunkMax {
|
||||
if err := w.flushChunk(); err != nil {
|
||||
return total - len(p), err
|
||||
}
|
||||
}
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
|
||||
func (w *copyChunkWriter) finish() (int, error) {
|
||||
if len(w.buf) > 0 || w.parts == 0 {
|
||||
if err := w.flushChunk(); err != nil {
|
||||
return w.parts, err
|
||||
}
|
||||
}
|
||||
return w.parts, nil
|
||||
}
|
||||
|
||||
func (w *copyChunkWriter) flushChunk() error {
|
||||
header := &tar.Header{
|
||||
Name: backupTableCopyPath(w.table, w.parts),
|
||||
Mode: archiveFileMode,
|
||||
Size: int64(len(w.buf)),
|
||||
ModTime: w.modTime,
|
||||
Typeflag: tar.TypeReg,
|
||||
}
|
||||
if err := w.tw.WriteHeader(header); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(w.buf) > 0 {
|
||||
if _, err := w.tw.Write(w.buf); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
w.buf = w.buf[:0]
|
||||
w.parts++
|
||||
if err := w.encoder.Flush(); err != nil {
|
||||
return err
|
||||
}
|
||||
if w.flusher != nil {
|
||||
w.flusher.Flush()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// walkBackupTree visits root's descendants in lexical order with slash-form
|
||||
// relative paths. visit returns false to skip an entry (and its subtree).
|
||||
// Symlinks are skipped: the package format only carries files and directories.
|
||||
func walkBackupTree(root string, visit func(relative string, info fs.FileInfo) (bool, error)) error {
|
||||
return filepath.WalkDir(root, func(current string, entry fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
if current != root && errors.Is(walkErr, fs.ErrNotExist) {
|
||||
return nil // removed while walking (an agent cleaned up)
|
||||
}
|
||||
return walkErr
|
||||
}
|
||||
relative, err := filepath.Rel(root, current)
|
||||
if err != nil || relative == "." {
|
||||
return err
|
||||
}
|
||||
if entry.Type()&fs.ModeSymlink != 0 {
|
||||
log.Printf("[backup] 跳过符号链接:%s", current)
|
||||
return nil
|
||||
}
|
||||
info, err := entry.Info()
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !info.IsDir() && !info.Mode().IsRegular() {
|
||||
return nil
|
||||
}
|
||||
keep, err := visit(filepath.ToSlash(relative), info)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !keep && info.IsDir() {
|
||||
return filepath.SkipDir
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func addBackupFile(tw *tar.Writer, path, name string) error {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return addBackupEntry(tw, path, name, info)
|
||||
}
|
||||
|
||||
func addBackupEntry(tw *tar.Writer, path, name string, info fs.FileInfo) error {
|
||||
header, err := tar.FileInfoHeader(info, "")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
header.Name = name
|
||||
header.Uname, header.Gname = "", ""
|
||||
if info.IsDir() {
|
||||
header.Name += "/"
|
||||
return tw.WriteHeader(header)
|
||||
}
|
||||
input, err := os.Open(path)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil // removed between listing and opening
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer input.Close()
|
||||
if err := tw.WriteHeader(header); err != nil {
|
||||
return err
|
||||
}
|
||||
// Exactly the stat'ed size: a file still being appended to is cut at that
|
||||
// point instead of overflowing its tar entry.
|
||||
if _, err := io.CopyN(tw, input, header.Size); err != nil {
|
||||
return fmt.Errorf("读取 %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// POST /api/system/backup/restore — body is the raw .tar.zst package.
|
||||
func (s *Server) backupRestore(w http.ResponseWriter, r *http.Request) {
|
||||
if s.pg(w) == nil {
|
||||
return
|
||||
}
|
||||
if _, running := updHub.snapshot(); running {
|
||||
writeProductError(w, 409, "backup_update_running", "更新正在进行中,无法恢复备份", nil)
|
||||
return
|
||||
}
|
||||
if !backupMu.TryLock() {
|
||||
writeProductError(w, 409, "backup_busy", "已有备份或恢复正在进行", nil)
|
||||
return
|
||||
}
|
||||
defer backupMu.Unlock()
|
||||
manifest, restoreID, err := s.stageBackupUpload(r.Context(), r.Body)
|
||||
if err != nil {
|
||||
log.Printf("[backup] 恢复包校验失败:%v", err)
|
||||
writeProductError(w, 400, "backup_invalid", "备份包无效:"+err.Error(), map[string]any{"message": err.Error()})
|
||||
return
|
||||
}
|
||||
log.Printf("[backup] 已接收备份(版本 %s,创建于 %s),即将重启以完成恢复", manifest.Version, manifest.CreatedAt.Format(time.RFC3339))
|
||||
writeJSON(w, 202, map[string]any{"ok": true, "restore_id": restoreID, "version": manifest.Version, "created_at": manifest.CreatedAt})
|
||||
go func() {
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
requestRestart()
|
||||
}()
|
||||
}
|
||||
|
||||
func newBackupRestoreID() string {
|
||||
var raw [16]byte
|
||||
if _, err := rand.Read(raw[:]); err == nil {
|
||||
return hex.EncodeToString(raw[:])
|
||||
}
|
||||
return fmt.Sprintf("%d", time.Now().UnixNano())
|
||||
}
|
||||
|
||||
func (s *Server) stageBackupUpload(ctx context.Context, body io.Reader) (*backupManifest, string, error) {
|
||||
upload, err := os.MkdirTemp(s.m.dir, backupUploadPrefix)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
staged := false
|
||||
defer func() {
|
||||
if !staged {
|
||||
_ = os.RemoveAll(upload)
|
||||
}
|
||||
}()
|
||||
if err := extractTarZst(body, upload, backupMaxFiles); err != nil {
|
||||
return nil, "", fmt.Errorf("解包失败: %w", err)
|
||||
}
|
||||
manifest, err := readBackupManifest(upload)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
if err := s.m.pg.CheckRestoreTables(ctx, backupTablesOnly(manifest.Tables)); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
restoreID := newBackupRestoreID()
|
||||
if err := os.WriteFile(filepath.Join(upload, backupRestoreIDFile), []byte(restoreID), archiveFileMode); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
pending := filepath.Join(s.m.dir, backupPendingDir)
|
||||
oldPending := upload + "-old"
|
||||
hadPending := false
|
||||
if _, err := os.Lstat(pending); err == nil {
|
||||
if err := os.Rename(pending, oldPending); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
hadPending = true
|
||||
} else if !errors.Is(err, fs.ErrNotExist) {
|
||||
return nil, "", err
|
||||
}
|
||||
if err := os.Rename(upload, pending); err != nil {
|
||||
if hadPending {
|
||||
_ = os.Rename(oldPending, pending)
|
||||
}
|
||||
return nil, "", err
|
||||
}
|
||||
staged = true
|
||||
if hadPending {
|
||||
_ = os.RemoveAll(oldPending)
|
||||
}
|
||||
return manifest, restoreID, nil
|
||||
}
|
||||
|
||||
// readBackupManifest loads and validates the manifest of an extracted package:
|
||||
// every name it lists must be a single local path element that actually exists,
|
||||
// so a crafted manifest cannot point the restore outside the package.
|
||||
func readBackupManifest(root string) (*backupManifest, error) {
|
||||
raw, err := os.ReadFile(filepath.Join(root, backupManifestName))
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return nil, errors.New("缺少 manifest.json(不是 ARTEX 备份包,或文件不完整)")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var manifest backupManifest
|
||||
if err := json.Unmarshal(raw, &manifest); err != nil {
|
||||
return nil, fmt.Errorf("manifest.json 格式错误: %w", err)
|
||||
}
|
||||
if manifest.Format < 1 || manifest.Format > backupFormat {
|
||||
return nil, fmt.Errorf("不支持的备份格式版本 %d(备份可能来自更新的 ARTEX 版本)", manifest.Format)
|
||||
}
|
||||
if len(manifest.Tables) == 0 {
|
||||
return nil, errors.New("备份中没有数据库表")
|
||||
}
|
||||
for _, table := range manifest.Tables {
|
||||
if !singleElement(table.Name) || len(table.Columns) == 0 {
|
||||
return nil, fmt.Errorf("备份表名无效: %q", table.Name)
|
||||
}
|
||||
parts := max(table.Parts, 1)
|
||||
if table.Parts < 0 || parts > backupMaxFiles {
|
||||
return nil, fmt.Errorf("备份表 %s 分片数无效: %d", table.Name, table.Parts)
|
||||
}
|
||||
for part := range parts {
|
||||
if _, err := os.Stat(filepath.Join(root, filepath.FromSlash(backupTableCopyPath(table.Name, part)))); err != nil {
|
||||
return nil, fmt.Errorf("缺少表 %s 的数据", table.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, name := range manifest.Entries {
|
||||
if !singleElement(name) || strings.HasPrefix(name, ".") {
|
||||
return nil, fmt.Errorf("备份条目名无效: %q", name)
|
||||
}
|
||||
}
|
||||
for _, name := range manifest.Skills {
|
||||
if !singleElement(name) || strings.HasPrefix(name, ".") {
|
||||
return nil, fmt.Errorf("备份 skill 名无效: %q", name)
|
||||
}
|
||||
}
|
||||
return &manifest, nil
|
||||
}
|
||||
|
||||
func singleElement(name string) bool {
|
||||
return name != "" && filepath.IsLocal(name) && !strings.ContainsAny(name, `/\`)
|
||||
}
|
||||
|
||||
// ApplyPendingRestore installs a backup staged by the restore endpoint. main
|
||||
// calls it before anything opens the data directory or the database. It returns
|
||||
// nil when nothing was pending.
|
||||
//
|
||||
// Steps are resumable: the database transaction is marked done inside the
|
||||
// pending directory, and each top-level entry is moved aside and replaced one
|
||||
// rename at a time, so a crash at any point is finished by the next boot. A
|
||||
// database failure leaves current data untouched and parks the package in
|
||||
// .restore-failed for inspection.
|
||||
func ApplyPendingRestore(dataDir, skillDir string) (*BackupRestoreResult, error) {
|
||||
if abs, err := filepath.Abs(dataDir); err == nil {
|
||||
dataDir = abs
|
||||
}
|
||||
pending := filepath.Join(dataDir, backupPendingDir)
|
||||
previous := filepath.Join(dataDir, backupPreviousDir)
|
||||
if _, err := os.Stat(pending); err != nil {
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
_ = os.RemoveAll(previous) // leftover of a restore that crashed during cleanup
|
||||
}
|
||||
cleanBackupUploads(dataDir)
|
||||
return nil, nil
|
||||
}
|
||||
cleanBackupUploads(dataDir)
|
||||
result := &BackupRestoreResult{At: time.Now()}
|
||||
if rawID, err := os.ReadFile(filepath.Join(pending, backupRestoreIDFile)); err == nil {
|
||||
result.ID = strings.TrimSpace(string(rawID))
|
||||
}
|
||||
manifest, err := readBackupManifest(pending)
|
||||
if err == nil {
|
||||
result.Source, result.Created = manifest.Version, manifest.CreatedAt
|
||||
log.Printf("[backup] 开始恢复备份(版本 %s,创建于 %s)", manifest.Version, manifest.CreatedAt.Format(time.RFC3339))
|
||||
err = restoreBackupDatabase(pending, dataDir, manifest)
|
||||
}
|
||||
if err != nil {
|
||||
result.Error = err.Error()
|
||||
log.Printf("[backup] 恢复失败,当前数据保持不变:%v", err)
|
||||
failed := filepath.Join(dataDir, backupFailedDir)
|
||||
_ = os.RemoveAll(failed)
|
||||
if renameErr := os.Rename(pending, failed); renameErr != nil {
|
||||
log.Printf("[backup] 无法移走失败的恢复包,删除之:%v", renameErr)
|
||||
_ = os.RemoveAll(pending)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
if err := installBackupFiles(pending, dataDir, skillDir, manifest); err != nil {
|
||||
// The database is already replaced; do not start the engine on half-installed
|
||||
// files. Returning an error makes main exit non-zero so the supervisor retries.
|
||||
result.Error = "数据库已恢复,但文件替换失败(下次启动会继续):" + err.Error()
|
||||
log.Printf("[backup] %s", result.Error)
|
||||
return result, err
|
||||
}
|
||||
if err := os.RemoveAll(previous); err != nil {
|
||||
log.Printf("[backup] 清理旧数据失败:%v", err)
|
||||
}
|
||||
if err := os.RemoveAll(pending); err != nil {
|
||||
result.Error = "恢复已完成,但清理暂存目录失败:" + err.Error()
|
||||
log.Printf("[backup] %s", result.Error)
|
||||
return result, err
|
||||
}
|
||||
result.Applied = true
|
||||
log.Printf("[backup] 备份恢复完成")
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func cleanBackupUploads(dataDir string) {
|
||||
entries, _ := os.ReadDir(dataDir)
|
||||
for _, entry := range entries {
|
||||
if strings.HasPrefix(entry.Name(), backupUploadPrefix) || strings.HasPrefix(entry.Name(), ".backup-") {
|
||||
_ = os.RemoveAll(filepath.Join(dataDir, entry.Name()))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func restoreBackupDatabase(pending, dataDir string, manifest *backupManifest) error {
|
||||
marker := filepath.Join(pending, backupDBDoneMarker)
|
||||
if _, err := os.Stat(marker); err == nil {
|
||||
return nil
|
||||
}
|
||||
dsn, _, err := pgdb.DSN()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
pg, err := pgdb.Open(dsn)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer pg.Close()
|
||||
// Created outside schema.sql; a fresh target database does not have them yet.
|
||||
if err := pg.EnsureLLMRecordsTable(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := pg.EnsureLLMUsageTable(); err != nil {
|
||||
return err
|
||||
}
|
||||
tables := make([]pgdb.RestoreTable, len(manifest.Tables))
|
||||
for i, table := range manifest.Tables {
|
||||
name, parts := table.Name, max(table.Parts, 1)
|
||||
tables[i] = pgdb.RestoreTable{
|
||||
BackupTable: table.BackupTable,
|
||||
Open: func() (io.ReadCloser, error) { return openBackupTableParts(pending, name, parts) },
|
||||
}
|
||||
}
|
||||
move := pgdb.PathPrefixMove{From: manifest.DataDir, FromSeparator: manifest.Separator, To: dataDir, ToSeparator: string(filepath.Separator)}
|
||||
if err := pg.RestoreTables(context.Background(), tables, move); err != nil {
|
||||
return err
|
||||
}
|
||||
// A machine-specific interpreter path from the old host would break custom
|
||||
// tools; dropping it lets startup re-detect the local one.
|
||||
if value, ok, _ := pg.GetSetting(settingPythonInterp); ok && strings.TrimSpace(value) != "" {
|
||||
if _, err := os.Stat(strings.TrimSpace(value)); err != nil {
|
||||
if _, err := pg.Exec(`DELETE FROM settings WHERE key=$1`, settingPythonInterp); err != nil {
|
||||
log.Printf("[backup] 清除失效的 python 解释器配置失败:%v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return writeArchiveJournal(marker, map[string]any{"at": time.Now()})
|
||||
}
|
||||
|
||||
func openBackupTableParts(pending, table string, parts int) (io.ReadCloser, error) {
|
||||
if parts <= 1 {
|
||||
return os.Open(filepath.Join(pending, filepath.FromSlash(backupTableCopyPath(table, 0))))
|
||||
}
|
||||
paths := make([]string, parts)
|
||||
for i := range parts {
|
||||
paths[i] = filepath.Join(pending, filepath.FromSlash(backupTableCopyPath(table, i)))
|
||||
}
|
||||
return &multiFileReader{paths: paths}, nil
|
||||
}
|
||||
|
||||
type multiFileReader struct {
|
||||
paths []string
|
||||
index int
|
||||
current *os.File
|
||||
}
|
||||
|
||||
func (r *multiFileReader) Read(p []byte) (int, error) {
|
||||
for {
|
||||
if r.current == nil {
|
||||
if r.index >= len(r.paths) {
|
||||
return 0, io.EOF
|
||||
}
|
||||
f, err := os.Open(r.paths[r.index])
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
r.current = f
|
||||
r.index++
|
||||
}
|
||||
n, err := r.current.Read(p)
|
||||
if n > 0 || (err != nil && !errors.Is(err, io.EOF)) {
|
||||
if errors.Is(err, io.EOF) {
|
||||
_ = r.current.Close()
|
||||
r.current = nil
|
||||
return n, nil
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
if closeErr := r.current.Close(); closeErr != nil {
|
||||
r.current = nil
|
||||
return 0, closeErr
|
||||
}
|
||||
r.current = nil
|
||||
}
|
||||
}
|
||||
|
||||
func (r *multiFileReader) Close() error {
|
||||
if r.current != nil {
|
||||
err := r.current.Close()
|
||||
r.current = nil
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func installBackupFiles(pending, dataDir, skillDir string, manifest *backupManifest) error {
|
||||
previous := filepath.Join(dataDir, backupPreviousDir)
|
||||
if err := os.MkdirAll(previous, archiveDirMode); err != nil {
|
||||
return err
|
||||
}
|
||||
keep := make(map[string]bool, len(manifest.Entries))
|
||||
for _, name := range manifest.Entries {
|
||||
keep[name] = true
|
||||
}
|
||||
entries, err := os.ReadDir(dataDir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if strings.HasPrefix(name, ".") || keep[name] {
|
||||
continue
|
||||
}
|
||||
if err := moveAside(filepath.Join(dataDir, name), filepath.Join(previous, name)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, name := range manifest.Entries {
|
||||
source := filepath.Join(pending, "files", name)
|
||||
if _, err := os.Lstat(source); errors.Is(err, fs.ErrNotExist) {
|
||||
continue // already installed by an interrupted earlier boot
|
||||
} else if err != nil {
|
||||
return err
|
||||
}
|
||||
destination := filepath.Join(dataDir, name)
|
||||
if _, err := os.Lstat(destination); err == nil {
|
||||
if err := moveAside(destination, filepath.Join(previous, name)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if err := os.Rename(source, destination); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if skillDir == "" {
|
||||
return nil
|
||||
}
|
||||
if err := os.MkdirAll(skillDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
// Skills already present here are kept: built-in skills ship with this
|
||||
// build and must not be downgraded by an older backup's copy.
|
||||
for _, name := range manifest.Skills {
|
||||
source := filepath.Join(pending, "skills", name)
|
||||
destination := filepath.Join(skillDir, name)
|
||||
if _, err := os.Lstat(source); err != nil {
|
||||
continue
|
||||
}
|
||||
if _, err := os.Lstat(destination); err == nil {
|
||||
continue
|
||||
}
|
||||
if err := os.Rename(source, destination); err != nil {
|
||||
if err := copyBackupTree(source, destination); err != nil {
|
||||
return fmt.Errorf("安装 skill %s: %w", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func moveAside(source, destination string) error {
|
||||
if _, err := os.Lstat(destination); err == nil {
|
||||
if err := os.RemoveAll(destination); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return os.Rename(source, destination)
|
||||
}
|
||||
|
||||
// copyBackupTree is the fallback when the skill directory sits on another
|
||||
// filesystem than the data directory (a Docker bind mount), where rename fails.
|
||||
func copyBackupTree(source, destination string) error {
|
||||
temporary := destination + ".restore-tmp"
|
||||
_ = os.RemoveAll(temporary)
|
||||
err := filepath.WalkDir(source, func(current string, entry fs.DirEntry, walkErr error) error {
|
||||
if walkErr != nil {
|
||||
return walkErr
|
||||
}
|
||||
relative, err := filepath.Rel(source, current)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
target := filepath.Join(temporary, relative)
|
||||
if entry.IsDir() {
|
||||
return os.MkdirAll(target, 0o755)
|
||||
}
|
||||
info, err := entry.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
input, err := os.Open(current)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer input.Close()
|
||||
output, err := os.OpenFile(target, os.O_CREATE|os.O_EXCL|os.O_WRONLY, info.Mode().Perm())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, copyErr := io.Copy(output, input)
|
||||
return errors.Join(copyErr, output.Close())
|
||||
})
|
||||
if err != nil {
|
||||
_ = os.RemoveAll(temporary)
|
||||
return err
|
||||
}
|
||||
return os.Rename(temporary, destination)
|
||||
}
|
||||
@@ -0,0 +1,429 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
pgdb "github.com/Autumn-27/artex/db"
|
||||
"io"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// scratchDSN points the configured server at a fresh database that is dropped
|
||||
// when the test ends, so restores never truncate the shared development DB.
|
||||
func scratchDSN(t *testing.T, base, name string) string {
|
||||
t.Helper()
|
||||
u, err := url.Parse(base)
|
||||
if err != nil {
|
||||
t.Skipf("unparseable DSN: %v", err)
|
||||
}
|
||||
u.Path = "/" + name
|
||||
admin := *u
|
||||
admin.Path = "/postgres"
|
||||
t.Cleanup(func() {
|
||||
conn, err := sql.Open("pgx", admin.String())
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
_, _ = conn.Exec(`DROP DATABASE IF EXISTS "` + name + `" WITH (FORCE)`)
|
||||
})
|
||||
return u.String()
|
||||
}
|
||||
|
||||
func writeTestFile(t *testing.T, path, body string) {
|
||||
t.Helper()
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupExportRestoreRoundTrip(t *testing.T) {
|
||||
base, _, err := pgdb.DSN()
|
||||
if err != nil {
|
||||
t.Skipf("no database config (%v)", err)
|
||||
}
|
||||
suffix := fmt.Sprintf("%d", time.Now().UnixNano())
|
||||
sourceDSN := scratchDSN(t, base, "artex_backup_src_"+suffix)
|
||||
targetDSN := scratchDSN(t, base, "artex_backup_dst_"+suffix)
|
||||
|
||||
sourceDir, sourceSkills := t.TempDir(), t.TempDir()
|
||||
t.Setenv("ARTEX_PG_DSN", sourceDSN)
|
||||
source, err := NewManager(sourceDir, "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v)", err)
|
||||
}
|
||||
task, err := source.pg.CreateTask("backup source", "migrate me", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := source.pg.SetSetting("backup.marker", "from-source"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
archivePath := filepath.Join(sourceDir, "archives", "tasks", "task-1.tar.zst")
|
||||
if _, err := source.pg.Exec(`INSERT INTO task_archives(task_id, state, archive_path) VALUES ($1, 'ready', $2)`, task.ID, archivePath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeTestFile(t, filepath.Join(sourceDir, "tasks", fmt.Sprint(task.ID), "note.txt"), "task workspace")
|
||||
writeTestFile(t, archivePath, "archive package")
|
||||
writeTestFile(t, filepath.Join(sourceDir, ".restore-failed", "junk"), "never exported")
|
||||
writeTestFile(t, filepath.Join(sourceSkills, "custom-skill", "SKILL.md"), "---\nname: custom-skill\n---\n")
|
||||
writeTestFile(t, filepath.Join(sourceSkills, "shipped", "SKILL.md"), "old copy")
|
||||
|
||||
exporter := &Server{m: source, skillDir: sourceSkills}
|
||||
|
||||
// Exporting while an active (non-terminal, non-paused) task exists must be refused.
|
||||
for _, method := range []string{"HEAD", "GET"} {
|
||||
blocked := httptest.NewRecorder()
|
||||
exporter.backupExport(blocked, httptest.NewRequest(method, "/api/system/backup/export", nil))
|
||||
if blocked.Code != 409 || blocked.Header().Get("X-Artex-Error-Code") != "backup_active_tasks" {
|
||||
t.Fatalf("%s with active task: status=%d header=%q body=%s", method, blocked.Code, blocked.Header().Get("X-Artex-Error-Code"), blocked.Body.String())
|
||||
}
|
||||
}
|
||||
if _, err := source.pg.Exec(`UPDATE tasks SET paused=true, pause_reason='paused_by_user' WHERE id=$1`, task.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Force small chunk size so multi-part COPY streaming is exercised end-to-end.
|
||||
prevChunk := backupCopyChunkBytes
|
||||
backupCopyChunkBytes = 128
|
||||
t.Cleanup(func() { backupCopyChunkBytes = prevChunk })
|
||||
|
||||
exported := httptest.NewRecorder()
|
||||
exporter.backupExport(exported, httptest.NewRequest("GET", "/api/system/backup/export", nil))
|
||||
source.Close()
|
||||
if exported.Code != 200 || exported.Body.Len() == 0 {
|
||||
t.Fatalf("export status=%d size=%d", exported.Code, exported.Body.Len())
|
||||
}
|
||||
|
||||
targetDir, targetSkills := t.TempDir(), t.TempDir()
|
||||
t.Setenv("ARTEX_PG_DSN", targetDSN)
|
||||
target, err := NewManager(targetDir, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := target.pg.SetSetting("backup.stale", "replaced"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeTestFile(t, filepath.Join(targetDir, "tasks", "999", "old.txt"), "stale workspace")
|
||||
writeTestFile(t, filepath.Join(targetSkills, "shipped", "SKILL.md"), "new copy")
|
||||
|
||||
restorer := &Server{m: target, skillDir: targetSkills}
|
||||
upload := httptest.NewRecorder()
|
||||
restorer.backupRestore(upload, httptest.NewRequest("POST", "/api/system/backup/restore", bytes.NewReader(exported.Body.Bytes())))
|
||||
if upload.Code != 202 {
|
||||
target.Close()
|
||||
t.Fatalf("restore upload status=%d body=%s", upload.Code, upload.Body.String())
|
||||
}
|
||||
var uploadResp struct {
|
||||
RestoreID string `json:"restore_id"`
|
||||
}
|
||||
if err := json.Unmarshal(upload.Body.Bytes(), &uploadResp); err != nil || uploadResp.RestoreID == "" {
|
||||
target.Close()
|
||||
t.Fatalf("missing restore_id in 202 response: %s (err=%v)", upload.Body.String(), err)
|
||||
}
|
||||
|
||||
// A failed subsequent upload must leave the already-staged pending backup intact.
|
||||
badUpload := httptest.NewRecorder()
|
||||
restorer.backupRestore(badUpload, httptest.NewRequest("POST", "/api/system/backup/restore", strings.NewReader("corrupt")))
|
||||
if badUpload.Code != 400 {
|
||||
target.Close()
|
||||
t.Fatalf("corrupt upload status=%d body=%s", badUpload.Code, badUpload.Body.String())
|
||||
}
|
||||
if rawID, err := os.ReadFile(filepath.Join(targetDir, backupPendingDir, backupRestoreIDFile)); err != nil || strings.TrimSpace(string(rawID)) != uploadResp.RestoreID {
|
||||
target.Close()
|
||||
t.Fatalf("corrupt upload clobbered pending restore_id: got=%q err=%v want=%q", string(rawID), err, uploadResp.RestoreID)
|
||||
}
|
||||
|
||||
// Uploading a second valid backup atomically replaces pending without leaving -old dirs.
|
||||
upload2 := httptest.NewRecorder()
|
||||
restorer.backupRestore(upload2, httptest.NewRequest("POST", "/api/system/backup/restore", bytes.NewReader(exported.Body.Bytes())))
|
||||
target.Close()
|
||||
if upload2.Code != 202 {
|
||||
t.Fatalf("second upload status=%d body=%s", upload2.Code, upload2.Body.String())
|
||||
}
|
||||
var uploadResp2 struct {
|
||||
RestoreID string `json:"restore_id"`
|
||||
}
|
||||
if err := json.Unmarshal(upload2.Body.Bytes(), &uploadResp2); err != nil || uploadResp2.RestoreID == "" || uploadResp2.RestoreID == uploadResp.RestoreID {
|
||||
t.Fatalf("unexpected second restore_id: %s (first=%q err=%v)", upload2.Body.String(), uploadResp.RestoreID, err)
|
||||
}
|
||||
if entries, err := os.ReadDir(targetDir); err == nil {
|
||||
for _, entry := range entries {
|
||||
if strings.HasPrefix(entry.Name(), backupUploadPrefix) {
|
||||
t.Fatalf("staging left upload/old dir behind: %s", entry.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result, err := ApplyPendingRestore(targetDir, targetSkills)
|
||||
if err != nil || result == nil || !result.Applied || result.ID != uploadResp2.RestoreID {
|
||||
t.Fatalf("restore not applied: result=%+v err=%v wantID=%q", result, err, uploadResp2.RestoreID)
|
||||
}
|
||||
if again, err := ApplyPendingRestore(targetDir, targetSkills); err != nil || again != nil {
|
||||
t.Fatalf("pending restore applied twice: result=%+v err=%v", again, err)
|
||||
}
|
||||
restored, err := pgdb.Open(targetDSN)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer restored.Close()
|
||||
if got, err := restored.GetTask(task.ID); err != nil || got == nil || got.Goal != "migrate me" {
|
||||
t.Fatalf("restored task=%+v err=%v", got, err)
|
||||
}
|
||||
if value, ok, _ := restored.GetSetting("backup.marker"); !ok || value != "from-source" {
|
||||
t.Fatalf("restored setting=%q ok=%v", value, ok)
|
||||
}
|
||||
if _, ok, _ := restored.GetSetting("backup.stale"); ok {
|
||||
t.Fatal("target-only setting survived the restore")
|
||||
}
|
||||
var restoredArchivePath string
|
||||
if err := restored.QueryRow(`SELECT archive_path FROM task_archives WHERE task_id=$1`, task.ID).Scan(&restoredArchivePath); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if want := filepath.Join(targetDir, "archives", "tasks", "task-1.tar.zst"); restoredArchivePath != want {
|
||||
t.Fatalf("archive path=%q want %q", restoredArchivePath, want)
|
||||
}
|
||||
next, err := restored.CreateTask("after restore", "fresh id", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("insert after restore (sequence not reset?): %v", err)
|
||||
}
|
||||
if next.ID <= task.ID {
|
||||
t.Fatalf("new task id %d does not follow restored id %d", next.ID, task.ID)
|
||||
}
|
||||
|
||||
for path, want := range map[string]string{
|
||||
filepath.Join(targetDir, "tasks", fmt.Sprint(task.ID), "note.txt"): "task workspace",
|
||||
filepath.Join(targetDir, "archives", "tasks", "task-1.tar.zst"): "archive package",
|
||||
filepath.Join(targetSkills, "custom-skill", "SKILL.md"): "---\nname: custom-skill\n---\n",
|
||||
filepath.Join(targetSkills, "shipped", "SKILL.md"): "new copy",
|
||||
} {
|
||||
if got, err := os.ReadFile(path); err != nil || string(got) != want {
|
||||
t.Fatalf("%s = %q, %v; want %q", path, got, err, want)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(targetDir, "traffic", "_index", "index.sqlite")); err != nil {
|
||||
t.Fatalf("traffic index snapshot not restored: %v", err)
|
||||
}
|
||||
for _, gone := range []string{
|
||||
filepath.Join(targetDir, "tasks", "999"),
|
||||
filepath.Join(targetDir, ".restore-failed"),
|
||||
filepath.Join(targetDir, backupPendingDir),
|
||||
filepath.Join(targetDir, backupPreviousDir),
|
||||
} {
|
||||
if _, err := os.Stat(gone); !os.IsNotExist(err) {
|
||||
t.Fatalf("%s should not exist after restore: %v", gone, err)
|
||||
}
|
||||
}
|
||||
|
||||
// A load that fails midway must leave the database exactly as it was.
|
||||
broken := []pgdb.RestoreTable{{
|
||||
BackupTable: pgdb.BackupTable{Name: "settings", Columns: []string{"key", "value"}, Rows: 2},
|
||||
Open: func() (io.ReadCloser, error) { return io.NopCloser(strings.NewReader("only\tone\n")), nil },
|
||||
}}
|
||||
if err := restored.RestoreTables(context.Background(), broken, pgdb.PathPrefixMove{}); err == nil {
|
||||
t.Fatal("row-count mismatch was accepted")
|
||||
}
|
||||
if got, err := restored.GetTask(task.ID); err != nil || got == nil {
|
||||
t.Fatalf("failed restore lost data: task=%+v err=%v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRestoreRejectsForeignPackage(t *testing.T) {
|
||||
if _, _, err := pgdb.DSN(); err != nil {
|
||||
t.Skipf("no database config (%v)", err)
|
||||
}
|
||||
dataDir := t.TempDir()
|
||||
m, err := NewManager(dataDir, "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v)", err)
|
||||
}
|
||||
defer m.Close()
|
||||
s := &Server{m: m}
|
||||
w := httptest.NewRecorder()
|
||||
s.backupRestore(w, httptest.NewRequest("POST", "/api/system/backup/restore", strings.NewReader("not a backup")))
|
||||
if w.Code != 400 {
|
||||
t.Fatalf("status=%d body=%s", w.Code, w.Body.String())
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dataDir, backupPendingDir)); !os.IsNotExist(err) {
|
||||
t.Fatalf("invalid upload left a pending restore: %v", err)
|
||||
}
|
||||
entries, _ := os.ReadDir(dataDir)
|
||||
for _, entry := range entries {
|
||||
if strings.HasPrefix(entry.Name(), backupUploadPrefix) {
|
||||
t.Fatalf("invalid upload left staging dir %s", entry.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBackupRestoreWindowsPathsAndPausesUnpausedTasks(t *testing.T) {
|
||||
base, _, err := pgdb.DSN()
|
||||
if err != nil {
|
||||
t.Skipf("no database config (%v)", err)
|
||||
}
|
||||
suffix := fmt.Sprintf("%d", time.Now().UnixNano())
|
||||
srcDSN := scratchDSN(t, base, "artex_backup_win_src_"+suffix)
|
||||
dstDSN := scratchDSN(t, base, "artex_backup_win_dst_"+suffix)
|
||||
|
||||
src, err := pgdb.Open(srcDSN)
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v)", err)
|
||||
}
|
||||
defer src.Close()
|
||||
activeTask, err := src.CreateTask("active", "running goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := src.Exec(`UPDATE tasks SET status='running', paused=false, queued=true, schedule_paused=true WHERE id=$1`, activeTask.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
doneTask, err := src.CreateTask("done", "finished goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := src.Exec(`UPDATE tasks SET status='done', paused=false WHERE id=$1`, doneTask.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
otherTask, err := src.CreateTask("other", "linux path goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := src.Exec(`UPDATE tasks SET status='done', paused=false WHERE id=$1`, otherTask.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, row := range []struct {
|
||||
id int64
|
||||
path string
|
||||
}{
|
||||
{activeTask.ID, `C:\artex\data\archives\tasks\win-1.tar.zst`},
|
||||
{doneTask.ID, `C:\artex\data-old\archives\tasks\prefix-collision.tar.zst`},
|
||||
{otherTask.ID, `/app/data/archives/tasks/already-linux.tar.zst`},
|
||||
} {
|
||||
if _, err := src.Exec(`INSERT INTO task_archives(task_id, state, archive_path) VALUES ($1, 'ready', $2)`, row.id, row.path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
var dumped []pgdb.RestoreTable
|
||||
err = src.DumpTables(context.Background(), func(table pgdb.BackupTable, copyTo func(io.Writer) (int64, error)) error {
|
||||
var buf bytes.Buffer
|
||||
rows, err := copyTo(&buf)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
table.Rows = rows
|
||||
raw := buf.Bytes()
|
||||
dumped = append(dumped, pgdb.RestoreTable{
|
||||
BackupTable: table,
|
||||
Open: func() (io.ReadCloser, error) { return io.NopCloser(bytes.NewReader(raw)), nil },
|
||||
})
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
dst, err := pgdb.Open(dstDSN)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer dst.Close()
|
||||
move := pgdb.PathPrefixMove{From: `C:\artex\data`, FromSeparator: `\`, To: `/app/data`, ToSeparator: `/`}
|
||||
if err := dst.RestoreTables(context.Background(), dumped, move); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var paused, queued, schedulePaused bool
|
||||
var pauseReason string
|
||||
if err := dst.QueryRow(`SELECT paused, pause_reason, queued, schedule_paused FROM tasks WHERE id=$1`, activeTask.ID).
|
||||
Scan(&paused, &pauseReason, &queued, &schedulePaused); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !paused || pauseReason != "paused_by_user" || queued || schedulePaused {
|
||||
t.Fatalf("restored active task not safely paused: paused=%v reason=%q queued=%v schedule_paused=%v", paused, pauseReason, queued, schedulePaused)
|
||||
}
|
||||
if err := dst.QueryRow(`SELECT paused FROM tasks WHERE id=$1`, doneTask.ID).Scan(&paused); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if paused {
|
||||
t.Fatal("terminal done task should not be flipped to paused")
|
||||
}
|
||||
for id, want := range map[int64]string{
|
||||
activeTask.ID: `/app/data/archives/tasks/win-1.tar.zst`,
|
||||
doneTask.ID: `C:\artex\data-old\archives\tasks\prefix-collision.tar.zst`,
|
||||
otherTask.ID: `/app/data/archives/tasks/already-linux.tar.zst`,
|
||||
} {
|
||||
var got string
|
||||
if err := dst.QueryRow(`SELECT archive_path FROM task_archives WHERE task_id=$1`, id).Scan(&got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got != want {
|
||||
t.Fatalf("task %d archive_path = %q, want %q", id, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyPendingRestoreFailsFastOnFileInstallError(t *testing.T) {
|
||||
dataDir := t.TempDir()
|
||||
pending := filepath.Join(dataDir, backupPendingDir)
|
||||
writeTestFile(t, filepath.Join(pending, backupDBDoneMarker), `{"at":"2026-11-20T00:00:00Z"}`)
|
||||
writeTestFile(t, filepath.Join(pending, backupRestoreIDFile), "rid-file-fail")
|
||||
writeTestFile(t, filepath.Join(pending, "db", "settings.copy"), "")
|
||||
writeTestFile(t, filepath.Join(pending, "files", "tasks", "1", "note.txt"), "restored note")
|
||||
writeTestFile(t, filepath.Join(pending, "skills", "custom-skill", "SKILL.md"), "skill body")
|
||||
manifest := backupManifest{
|
||||
Format: backupFormat,
|
||||
Version: "v0.3.30",
|
||||
CreatedAt: time.Now().UTC(),
|
||||
Tables: []backupTableEntry{{BackupTable: pgdb.BackupTable{Name: "settings", Columns: []string{"key", "value"}}}},
|
||||
Entries: []string{"tasks"},
|
||||
Skills: []string{"custom-skill"},
|
||||
}
|
||||
raw, err := json.Marshal(manifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writeTestFile(t, filepath.Join(pending, backupManifestName), string(raw))
|
||||
|
||||
// A regular file where skillDir should be forces installBackupFiles to fail after DB commit.
|
||||
blockingSkillPath := filepath.Join(t.TempDir(), "not-a-dir")
|
||||
writeTestFile(t, blockingSkillPath, "blocker")
|
||||
|
||||
res, err := ApplyPendingRestore(dataDir, blockingSkillPath)
|
||||
if err == nil {
|
||||
t.Fatalf("expected error on file install failure, got result=%+v", res)
|
||||
}
|
||||
if res == nil || res.Applied || res.ID != "rid-file-fail" {
|
||||
t.Fatalf("unexpected failure result: %+v", res)
|
||||
}
|
||||
if _, statErr := os.Stat(pending); statErr != nil {
|
||||
t.Fatalf("pending dir must stay for retry on next boot: %v", statErr)
|
||||
}
|
||||
|
||||
// Once the blocker is fixed, the next boot resumes and finishes cleanly.
|
||||
realSkillDir := t.TempDir()
|
||||
res, err = ApplyPendingRestore(dataDir, realSkillDir)
|
||||
if err != nil || res == nil || !res.Applied || res.ID != "rid-file-fail" {
|
||||
t.Fatalf("resume after file install failure failed: result=%+v err=%v", res, err)
|
||||
}
|
||||
if got, err := os.ReadFile(filepath.Join(dataDir, "tasks", "1", "note.txt")); err != nil || string(got) != "restored note" {
|
||||
t.Fatalf("tasks note=%q err=%v", got, err)
|
||||
}
|
||||
if got, err := os.ReadFile(filepath.Join(realSkillDir, "custom-skill", "SKILL.md")); err != nil || string(got) != "skill body" {
|
||||
t.Fatalf("skill=%q err=%v", got, err)
|
||||
}
|
||||
if _, statErr := os.Stat(pending); !os.IsNotExist(statErr) {
|
||||
t.Fatalf("pending dir should be removed after completion: %v", statErr)
|
||||
}
|
||||
}
|
||||
@@ -656,7 +656,13 @@ func extractTaskArchivePackage(path, expectedSHA, destination string) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
decoder, err := zstd.NewReader(file)
|
||||
return extractTarZst(file, destination, maxArchiveFiles)
|
||||
}
|
||||
|
||||
// extractTarZst unpacks a zstd-compressed tar stream of plain files and
|
||||
// directories into destination, refusing entries that would escape it.
|
||||
func extractTarZst(input io.Reader, destination string, maxFiles int) error {
|
||||
decoder, err := zstd.NewReader(input)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -676,17 +682,17 @@ func extractTaskArchivePackage(path, expectedSHA, destination string) error {
|
||||
return err
|
||||
}
|
||||
entries++
|
||||
if entries > maxArchiveFiles || header.Size < 0 || total+header.Size > maxArchiveBytes {
|
||||
return errors.New("task archive exceeds extraction safety limits")
|
||||
if entries > maxFiles || header.Size < 0 || total+header.Size > maxArchiveBytes {
|
||||
return errors.New("archive exceeds extraction safety limits")
|
||||
}
|
||||
total += header.Size
|
||||
clean := filepath.Clean(filepath.FromSlash(header.Name))
|
||||
if clean == "." || filepath.IsAbs(clean) || clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) {
|
||||
return fmt.Errorf("unsafe task archive path %q", header.Name)
|
||||
return fmt.Errorf("unsafe archive path %q", header.Name)
|
||||
}
|
||||
target := filepath.Join(destination, clean)
|
||||
if relative, err := filepath.Rel(destination, target); err != nil || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||
return fmt.Errorf("unsafe task archive target %q", header.Name)
|
||||
return fmt.Errorf("unsafe archive target %q", header.Name)
|
||||
}
|
||||
switch header.Typeflag {
|
||||
case tar.TypeDir:
|
||||
@@ -707,7 +713,7 @@ func extractTaskArchivePackage(path, expectedSHA, destination string) error {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("unsupported task archive entry type %d", header.Typeflag)
|
||||
return fmt.Errorf("unsupported archive entry type %d", header.Typeflag)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
harvest_static.py <BASE_URL> <OUTDIR>
|
||||
harvest_static.py <BASE_URL> <OUTDIR> [--user-agent VALUE]
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标站点调整,常见改动:
|
||||
- extract_endpoints 正则(endpoint 方言)
|
||||
@@ -16,16 +16,54 @@ Static SPA bundle harvester. Framework-agnostic; tuned for webpack + Vite.
|
||||
|
||||
Stdlib only. TLS verification is disabled (recon against self-signed/internal hosts).
|
||||
"""
|
||||
import sys, os, re, ssl, json, urllib.request, urllib.parse
|
||||
import sys, os, re, ssl, secrets, json, argparse, urllib.request, urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon)"
|
||||
USER_AGENTS = (
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
|
||||
)
|
||||
|
||||
def fetch(url, binary=False):
|
||||
|
||||
def user_agent(value=None):
|
||||
"""Explicit value > environment > random sample; explicit empty omits UA."""
|
||||
explicit = value is not None
|
||||
if value is None:
|
||||
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
|
||||
value = value.strip(" ")
|
||||
if not all(32 <= ord(c) <= 126 for c in value):
|
||||
return secrets.choice(USER_AGENTS)
|
||||
if value or explicit:
|
||||
return value
|
||||
return secrets.choice(USER_AGENTS)
|
||||
|
||||
|
||||
def fetch(url, binary=False, user_agent_value=None):
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=25) as r:
|
||||
ua = user_agent(user_agent_value)
|
||||
req = urllib.request.Request(url, headers={"User-Agent": ua} if ua else {})
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
|
||||
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
|
||||
with opener.open(req, timeout=25) as r:
|
||||
data = r.read()
|
||||
return (data if binary else data.decode("utf-8", "ignore")), r.status
|
||||
except Exception as e:
|
||||
@@ -112,14 +150,17 @@ def extract_routes(js_text):
|
||||
|
||||
# ---- main -------------------------------------------------------------------
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print("usage: harvest_static.py <BASE_URL> <OUTDIR>"); sys.exit(1)
|
||||
base, outdir = sys.argv[1], sys.argv[2]
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("base"); ap.add_argument("outdir")
|
||||
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
|
||||
args = ap.parse_args()
|
||||
base, outdir = args.base, args.outdir
|
||||
ua = user_agent(args.user_agent)
|
||||
if not base.startswith("http"): base = "https://" + base
|
||||
jsdir = os.path.join(outdir, "js"); os.makedirs(jsdir, exist_ok=True)
|
||||
|
||||
print(f"[*] entry: {base}")
|
||||
html, status = fetch(base)
|
||||
html, status = fetch(base, user_agent_value=ua)
|
||||
open(os.path.join(outdir, "index.html"), "w").write(html)
|
||||
origin = origin_of(base)
|
||||
|
||||
@@ -137,7 +178,7 @@ def main():
|
||||
if not fn.endswith(".js"): return None
|
||||
lp = os.path.join(jsdir, fn)
|
||||
if url in have: return have[url]
|
||||
data, st = fetch(url, binary=True)
|
||||
data, st = fetch(url, binary=True, user_agent_value=ua)
|
||||
if st == 200 and data and not data[:15].lstrip().startswith(b"<"):
|
||||
open(lp, "wb").write(data); have[url] = lp; return lp
|
||||
return None
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4]
|
||||
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4] [--user-agent VALUE]
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标调整 --exclude、cookie、depth/max 等同域策略。
|
||||
|
||||
@@ -14,20 +14,58 @@ the same origin and extracts:
|
||||
|
||||
Stdlib only. Same-origin, bounded, polite. Provide --cookie for an authed crawl.
|
||||
"""
|
||||
import sys, os, re, ssl, argparse, urllib.request, urllib.parse, time
|
||||
import sys, os, re, ssl, secrets, argparse, urllib.request, urllib.parse, time
|
||||
from html.parser import HTMLParser
|
||||
from collections import deque
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon spider)"
|
||||
USER_AGENTS = (
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
|
||||
)
|
||||
|
||||
|
||||
def user_agent(value=None):
|
||||
"""Explicit value > environment > random sample; explicit empty omits UA."""
|
||||
explicit = value is not None
|
||||
if value is None:
|
||||
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
|
||||
value = value.strip(" ")
|
||||
if not all(32 <= ord(c) <= 126 for c in value):
|
||||
return secrets.choice(USER_AGENTS)
|
||||
if value or explicit:
|
||||
return value
|
||||
return secrets.choice(USER_AGENTS)
|
||||
|
||||
SKIP_EXT = re.compile(r'\.(css|png|jpe?g|gif|svg|ico|woff2?|ttf|pdf|zip|mp4|webp|js)(\?|$)', re.I)
|
||||
API_HINT = re.compile(r'/(?:api|rest|service|ajax|action|do|rpc|graphql|v\d|admin|backend)\b', re.I)
|
||||
|
||||
def fetch(url, cookie):
|
||||
h = {"User-Agent": UA}
|
||||
def fetch(url, cookie, user_agent_value=None):
|
||||
ua = user_agent(user_agent_value)
|
||||
h = {"User-Agent": ua} if ua else {}
|
||||
if cookie: h["Cookie"] = cookie
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), context=CTX, timeout=20) as r:
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
|
||||
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
|
||||
with opener.open(urllib.request.Request(url, headers=h), timeout=20) as r:
|
||||
ct = r.headers.get("Content-Type", "")
|
||||
if "html" not in ct and "xml" not in ct: return "", ct
|
||||
return r.read().decode("utf-8", "ignore"), ct
|
||||
@@ -70,7 +108,9 @@ def main():
|
||||
ap.add_argument("--cookie", default=""); ap.add_argument("--max", type=int, default=200)
|
||||
ap.add_argument("--depth", type=int, default=4); ap.add_argument("--delay", type=float, default=0.1)
|
||||
ap.add_argument("--include", default=""); ap.add_argument("--exclude", default="logout|signout|delete|remove|destroy")
|
||||
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
|
||||
args = ap.parse_args()
|
||||
ua = user_agent(args.user_agent)
|
||||
base = args.base if args.base.startswith("http") else "https://" + args.base
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
origin = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(base))
|
||||
@@ -82,7 +122,7 @@ def main():
|
||||
n = 0
|
||||
while q and n < args.max:
|
||||
url, d = q.popleft(); n += 1
|
||||
html, ct = fetch(url, args.cookie)
|
||||
html, ct = fetch(url, args.cookie, user_agent_value=ua)
|
||||
if args.delay: time.sleep(args.delay)
|
||||
if not html: continue
|
||||
p = Page()
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Offline regressions: python3 -m unittest discover -s skills/api-recon/scripts."""
|
||||
import contextlib
|
||||
import http.server
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parent
|
||||
# Keep the offline fixtures independent of the runner's outbound/recording proxy.
|
||||
DIRECT_ENV = {"no_proxy": "*", "NO_PROXY": "*"}
|
||||
|
||||
|
||||
def load_script(name):
|
||||
spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def capture_server():
|
||||
seen = []
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
seen.append((self.path, self.headers.get("User-Agent"), self.headers.get("Cookie")))
|
||||
if self.path == "/start":
|
||||
self.send_response(302)
|
||||
self.send_header("Location", "/page")
|
||||
self.end_headers()
|
||||
return
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html")
|
||||
self.end_headers()
|
||||
self.wfile.write(b'<h1>Page</h1><script src="/app.js"></script><form action="/submit"></form>')
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
with http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
|
||||
thread = threading.Thread(target=server.serve_forever)
|
||||
thread.start()
|
||||
try:
|
||||
yield f"http://127.0.0.1:{server.server_port}", seen
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join()
|
||||
|
||||
|
||||
class UserAgentTests(unittest.TestCase):
|
||||
def assert_identity(self, seen, expected, samples):
|
||||
self.assertTrue(seen)
|
||||
if expected == "random":
|
||||
self.assertIn(seen[0][1], samples)
|
||||
expected = seen[0][1]
|
||||
self.assertTrue(all(ua == expected for _, ua, _ in seen), seen)
|
||||
|
||||
def test_random_selection_and_explicit_override(self):
|
||||
go_source = (SCRIPTS.parents[2] / "internal/httpidentity/useragent.go").read_text()
|
||||
go_pool = tuple(re.findall(r'^\s*"(Mozilla/5\.0 [^"]+)",$', go_source, re.MULTILINE))
|
||||
self.assertEqual(len(go_pool), 20)
|
||||
self.assertEqual(len(set(go_pool)), 20)
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
self.assertEqual(module.USER_AGENTS, go_pool)
|
||||
for sample in module.USER_AGENTS:
|
||||
with self.subTest(script=name, sample=sample):
|
||||
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": ""}), patch.object(module.secrets, "choice", return_value=sample) as choose:
|
||||
self.assertEqual(module.user_agent(), sample)
|
||||
choose.assert_called_once_with(module.USER_AGENTS)
|
||||
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": "Configured-UA"}), patch.object(module.secrets, "choice") as choose:
|
||||
self.assertEqual(module.user_agent(), "Configured-UA")
|
||||
self.assertEqual(module.user_agent("Override-UA"), "Override-UA")
|
||||
self.assertEqual(module.user_agent(""), "")
|
||||
choose.assert_not_called()
|
||||
|
||||
def test_fetch_and_redirect(self):
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
for value, expected in (("", "random"), (" Captured-UA/1.0 ", "Captured-UA/1.0"),
|
||||
("ok\r\nX-Injected: yes", "random"), ("客户端", "random")):
|
||||
with self.subTest(script=name, value=value), capture_server() as (base, seen):
|
||||
with patch.dict(os.environ, {**DIRECT_ENV, "ARTEX_HTTP_USER_AGENT": value}):
|
||||
if name == "harvest_static":
|
||||
body, status = module.fetch(base + "/start")
|
||||
self.assertEqual(status, 200)
|
||||
else:
|
||||
body, _ = module.fetch(base + "/start", "session=sample")
|
||||
self.assertTrue(all(cookie == "session=sample" for _, _, cookie in seen))
|
||||
self.assertIn("Page", body)
|
||||
self.assertEqual([path for path, _, _ in seen], ["/start", "/page"])
|
||||
self.assert_identity(seen, expected, module.USER_AGENTS)
|
||||
|
||||
def test_cli_precedence_and_outputs(self):
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
for env_ua, flags, expected in (("", [], "random"),
|
||||
("Environment-UA/1.0", [], "Environment-UA/1.0"),
|
||||
("Environment-UA/1.0", ["--user-agent", "Override-UA/1.0"], "Override-UA/1.0"),
|
||||
("Environment-UA/1.0", ["--user-agent", ""], None)):
|
||||
with self.subTest(script=name, flags=flags), capture_server() as (base, seen):
|
||||
with tempfile.TemporaryDirectory() as outdir:
|
||||
env = dict(os.environ, **DIRECT_ENV, ARTEX_HTTP_USER_AGENT=env_ua)
|
||||
command = [sys.executable, str(SCRIPTS / f"{name}.py"), base + "/start", outdir] + flags
|
||||
if name == "spider_mpa":
|
||||
command += ["--max", "1", "--delay", "0"]
|
||||
result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertGreaterEqual(len(seen), 2)
|
||||
self.assert_identity(seen, expected, module.USER_AGENTS)
|
||||
artifact = "api_static.txt" if name == "harvest_static" else "forms.txt"
|
||||
self.assertTrue((Path(outdir) / artifact).exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -378,6 +378,14 @@ func (t *Traffic) CACertPath() string {
|
||||
return filepath.Join(t.dir, "_ca", "mitmproxy-ca-cert.pem")
|
||||
}
|
||||
|
||||
// SnapshotIndex writes a transactionally consistent copy of the live SQLite
|
||||
// index to dest (which must not exist). Copying index.sqlite directly would miss
|
||||
// pages still in the WAL and could capture a half-applied write.
|
||||
func (t *Traffic) SnapshotIndex(dest string) error {
|
||||
_, err := t.db.Exec(`VACUUM INTO ?`, dest)
|
||||
return err
|
||||
}
|
||||
|
||||
// Start runs the proxy (blocking); run in a goroutine.
|
||||
func (t *Traffic) Start() error { return t.proxy.Start() }
|
||||
|
||||
@@ -412,6 +420,20 @@ type sink struct {
|
||||
t *Traffic
|
||||
}
|
||||
|
||||
// Preserve an absent User-Agent when go-mitmproxy creates its outbound Go
|
||||
// request. An explicit empty value suppresses net/http's automatic default.
|
||||
func (s *sink) Requestheaders(f *mproxy.Flow) {
|
||||
if f.Request == nil {
|
||||
return
|
||||
}
|
||||
if f.Request.Header == nil {
|
||||
f.Request.Header = make(http.Header)
|
||||
}
|
||||
if _, present := f.Request.Header["User-Agent"]; !present {
|
||||
f.Request.Header.Set("User-Agent", "")
|
||||
}
|
||||
}
|
||||
|
||||
func (s *sink) Response(f *mproxy.Flow) {
|
||||
if f.Request == nil || f.Response == nil {
|
||||
return
|
||||
@@ -683,6 +705,9 @@ func headerLines(h map[string][]string) string {
|
||||
func requestHeaderLines(req *mproxy.Request) string {
|
||||
headers := req.Header.Clone()
|
||||
headers.Del("Host")
|
||||
if headers.Get("User-Agent") == "" {
|
||||
headers.Del("User-Agent") // Empty suppression value is not sent on the wire.
|
||||
}
|
||||
host := req.URL.Host
|
||||
if raw := req.Raw(); raw != nil && strings.TrimSpace(raw.Host) != "" {
|
||||
host = raw.Host
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package traffic
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
|
||||
)
|
||||
|
||||
func TestProxyPreservesUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); got != ua {
|
||||
t.Errorf("forwarded User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
if _, present := r.Header["User-Agent"]; ua == "" && present {
|
||||
t.Error("proxy reintroduced a User-Agent")
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
u, _ := url.Parse(server.URL)
|
||||
flow := &mproxy.Flow{Request: &mproxy.Request{URL: u, Header: make(http.Header)}}
|
||||
if ua != "" {
|
||||
flow.Request.Header.Set("User-Agent", ua)
|
||||
}
|
||||
(&sink{}).Requestheaders(flow)
|
||||
// go-mitmproxy rebuilds the outbound request by adding every value.
|
||||
req, _ := http.NewRequest(http.MethodGet, server.URL, nil)
|
||||
for key, values := range flow.Request.Header {
|
||||
for _, value := range values {
|
||||
req.Header.Add(key, value)
|
||||
}
|
||||
}
|
||||
resp, err := server.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if ua == "" && strings.Contains(requestHeaderLines(flow.Request), "User-Agent:") {
|
||||
t.Error("recording contains a synthetic header that was not sent")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
"use client";
|
||||
|
||||
import * as React from "react";
|
||||
|
||||
import { ArchiveRestoreIcon, DatabaseBackupIcon, DownloadIcon } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
|
||||
import {
|
||||
AlertDialog,
|
||||
AlertDialogAction,
|
||||
AlertDialogCancel,
|
||||
AlertDialogContent,
|
||||
AlertDialogDescription,
|
||||
AlertDialogFooter,
|
||||
AlertDialogHeader,
|
||||
AlertDialogTitle,
|
||||
} from "@/components/ui/alert-dialog";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
|
||||
import { Progress } from "@/components/ui/progress";
|
||||
import { localizedError } from "@/i18n/errors";
|
||||
import { useLiveTranslation } from "@/i18n/use-live-translation";
|
||||
import { api, type BackupStatus } from "@/lib/api";
|
||||
|
||||
/** 恢复要先完成数据库整体替换和文件换装,再走一次正常启动;数据量大时比升级慢得多。 */
|
||||
const RESTORE_TIMEOUT_MS = 15 * 60_000;
|
||||
|
||||
export function BackupCard() {
|
||||
const { t, liveT } = useLiveTranslation();
|
||||
const [status, setStatus] = React.useState<BackupStatus | null>(null);
|
||||
const [file, setFile] = React.useState<File | null>(null);
|
||||
const [uploading, setUploading] = React.useState(false);
|
||||
const [restarting, setRestarting] = React.useState(false);
|
||||
const inputRef = React.useRef<HTMLInputElement>(null);
|
||||
|
||||
React.useEffect(() => {
|
||||
api
|
||||
.backupStatus()
|
||||
.then(setStatus)
|
||||
.catch(() => setStatus(null));
|
||||
}, []);
|
||||
|
||||
const download = async () => {
|
||||
try {
|
||||
await api.downloadBackup();
|
||||
toast.success(t("settings.backup.downloadStarted"));
|
||||
} catch (e) {
|
||||
toast.error(t("settings.backup.downloadFailedPrefix") + localizedError(e, t));
|
||||
}
|
||||
};
|
||||
|
||||
// 重启期间接口连不上是预期的;以"返回本次上传对应的 restore_id 结果"为准判断新进程已就绪。
|
||||
const waitForRestore = React.useCallback(
|
||||
async (restoreId: string) => {
|
||||
setRestarting(true);
|
||||
const deadline = Date.now() + RESTORE_TIMEOUT_MS;
|
||||
while (Date.now() < deadline) {
|
||||
const tick = Promise.withResolvers<void>();
|
||||
setTimeout(tick.resolve, 3000);
|
||||
await tick.promise;
|
||||
try {
|
||||
const next = await api.backupStatus();
|
||||
if (next.last_restore?.id === restoreId) {
|
||||
window.location.reload();
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// 继续轮询。
|
||||
}
|
||||
}
|
||||
setRestarting(false);
|
||||
toast.error(liveT("settings.backup.restoreTimeout"));
|
||||
},
|
||||
[liveT],
|
||||
);
|
||||
|
||||
const restore = async () => {
|
||||
if (!file) return;
|
||||
const selected = file;
|
||||
setFile(null);
|
||||
setUploading(true);
|
||||
try {
|
||||
const res = await api.restoreBackup(selected);
|
||||
setUploading(false);
|
||||
await waitForRestore(res.restore_id);
|
||||
} catch (e) {
|
||||
setUploading(false);
|
||||
toast.error(t("settings.backup.restoreFailedPrefix") + localizedError(e, t));
|
||||
}
|
||||
};
|
||||
|
||||
const busy = uploading || restarting;
|
||||
const last = status?.last_restore;
|
||||
|
||||
return (
|
||||
<Card className="mb-4 break-inside-avoid md:mb-6">
|
||||
<CardHeader>
|
||||
<CardTitle className="flex items-center gap-2 text-base">
|
||||
<DatabaseBackupIcon className="size-4" />
|
||||
{t("settings.backup.title")}
|
||||
</CardTitle>
|
||||
<CardDescription>{t("settings.backup.help")}</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="flex flex-col gap-4">
|
||||
{last?.applied ? (
|
||||
<p className="text-xs text-muted-foreground">
|
||||
{t("settings.backup.lastRestoreOk", {
|
||||
time: new Date(last.at).toLocaleString(),
|
||||
version: last.source_version || "-",
|
||||
})}
|
||||
</p>
|
||||
) : null}
|
||||
{last && !last.applied && last.error ? (
|
||||
<p role="alert" className="text-xs text-destructive">
|
||||
{t("settings.backup.lastRestoreFailed", { message: last.error })}
|
||||
</p>
|
||||
) : null}
|
||||
{status?.pending && !busy ? (
|
||||
<p className="text-xs text-muted-foreground">{t("settings.backup.pending")}</p>
|
||||
) : null}
|
||||
|
||||
{busy ? (
|
||||
<div className="space-y-1.5">
|
||||
<Progress value={100} className="animate-pulse" />
|
||||
<p className="text-xs text-muted-foreground">
|
||||
{restarting ? t("settings.backup.restarting") : t("settings.backup.uploading")}
|
||||
</p>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button size="sm" onClick={download} disabled={busy}>
|
||||
<DownloadIcon className="size-4" />
|
||||
{t("settings.backup.download")}
|
||||
</Button>
|
||||
<Button variant="outline" size="sm" onClick={() => inputRef.current?.click()} disabled={busy}>
|
||||
<ArchiveRestoreIcon className="size-4" />
|
||||
{t("settings.backup.restore")}
|
||||
</Button>
|
||||
<input
|
||||
ref={inputRef}
|
||||
type="file"
|
||||
accept=".zst,.tar.zst"
|
||||
className="hidden"
|
||||
onChange={(e) => {
|
||||
setFile(e.target.files?.[0] ?? null);
|
||||
e.target.value = "";
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<p className="text-xs text-muted-foreground">{t("settings.backup.note")}</p>
|
||||
</CardContent>
|
||||
|
||||
<AlertDialog open={file !== null} onOpenChange={(open) => (open ? null : setFile(null))}>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>{t("settings.backup.confirmTitle")}</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
{t("settings.backup.confirmBody", { name: file?.name ?? "" })}
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>{t("settings.backup.cancel")}</AlertDialogCancel>
|
||||
<AlertDialogAction variant="destructive" onClick={restore}>
|
||||
{t("settings.backup.confirm")}
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</Card>
|
||||
);
|
||||
}
|
||||
@@ -30,6 +30,7 @@ import { CHAT_SEND_MODE_OPTIONS, type ChatSendMode, setChatSendMode, useChatSend
|
||||
import type { EgressReport, LLMProfile, ProxyPoolEntry, Settings, WorkerPool } from "@/lib/types";
|
||||
|
||||
import { useEgressIp } from "../../_components/egress-ip-banner";
|
||||
import { BackupCard } from "./_components/backup-card";
|
||||
import { BasicAuthCard } from "./_components/basic-auth-card";
|
||||
import { McpServerCard } from "./_components/mcp-server-card";
|
||||
import { UpdateCard } from "./_components/update-card";
|
||||
@@ -766,6 +767,7 @@ export default function SystemSettingsPage() {
|
||||
|
||||
<div className="flex flex-col divide-y rounded-xl border bg-card/30 px-3 [&>[data-slot=card]]:mb-0 [&>[data-slot=card]]:rounded-none [&>[data-slot=card]]:bg-transparent [&>[data-slot=card]]:py-4 [&>[data-slot=card]]:ring-0">
|
||||
{isSettingVisible("system", "settings.keywords.update") ? <UpdateCard /> : null}
|
||||
{isSettingVisible("system", "settings.keywords.backup") ? <BackupCard /> : null}
|
||||
{isSettingVisible("system", "settings.keywords.basicAuth") ? <BasicAuthCard /> : null}
|
||||
{isSettingVisible("system", "settings.keywords.mcpServer") ? <McpServerCard /> : null}
|
||||
|
||||
|
||||
@@ -3,6 +3,20 @@ export const errorMessages = {
|
||||
"情报数据响应不完整,请刷新后重试",
|
||||
"The intelligence response is incomplete. Refresh to try again.",
|
||||
],
|
||||
"error.backup_busy": ["已有备份或恢复正在进行", "A backup or restore is already in progress"],
|
||||
"error.backup_active_tasks": [
|
||||
"存在未暂停的运行中或排队中任务,请先暂停或等待任务完成后再下载备份",
|
||||
"Active or queued tasks are running. Pause them or wait for them to finish before downloading a backup.",
|
||||
],
|
||||
"error.backup_update_running": [
|
||||
"更新正在进行中,无法恢复备份",
|
||||
"An update is in progress. The backup cannot be restored now.",
|
||||
],
|
||||
"error.backup_invalid": ["备份包无效:{{message}}", "Invalid backup package: {{message}}"],
|
||||
"error.demo_backup_disabled": [
|
||||
"演示模式不支持备份与恢复,请在实际部署中使用",
|
||||
"Backup and restore are not available in the demo. Use them in a deployed instance.",
|
||||
],
|
||||
"error.basic_auth_required": ["请先完成 HTTP Basic Auth 验证", "Complete HTTP Basic Auth verification first"],
|
||||
"error.auth_session_failed": ["认证会话生成失败", "Unable to create an authentication session"],
|
||||
"error.basic_auth_username_invalid": [
|
||||
|
||||
@@ -138,6 +138,7 @@ test("migrated pages reference existing keys and contain no hardcoded Chinese di
|
||||
"components/approval-execution-focus.tsx",
|
||||
"hooks/use-side-questions.ts",
|
||||
"app/(main)/dashboard/page.tsx",
|
||||
"app/(main)/system/settings/_components/backup-card.tsx",
|
||||
"app/(main)/system/settings/_components/basic-auth-card.tsx",
|
||||
"app/(main)/system/settings/_components/update-card.tsx",
|
||||
"app/(main)/_components/egress-ip-banner.tsx",
|
||||
|
||||
@@ -110,6 +110,52 @@ export const settingsMessages = {
|
||||
"settings.matchingSettings": ["匹配“{{query}}”的设置项", "Settings matching “{{query}}”"],
|
||||
"settings.clearSearch": ["清除搜索", "Clear search"],
|
||||
"settings.keywords.update": ["更新 版本 升级", "update version upgrade"],
|
||||
"settings.keywords.backup": [
|
||||
"备份 恢复 迁移 导出 导入 数据 一键备份",
|
||||
"backup restore migrate migration export import data",
|
||||
],
|
||||
"settings.backup.title": ["备份与迁移", "Backup and migration"],
|
||||
"settings.backup.help": [
|
||||
"一键导出全部数据(数据库、任务工作区、会话、流量、证据、归档包、MITM 证书和 skills),在新机器上恢复即可完成迁移。",
|
||||
"Export all data (database, task workspaces, sessions, traffic, evidence, archive packages, MITM certificate and skills) in one file, then restore it on another machine to migrate.",
|
||||
],
|
||||
"settings.backup.download": ["下载备份", "Download backup"],
|
||||
"settings.backup.downloadStarted": [
|
||||
"备份已开始生成,浏览器会直接下载,数据较多时需要一些时间",
|
||||
"The backup is being generated and will download directly. Large installations take a while.",
|
||||
],
|
||||
"settings.backup.downloadFailedPrefix": ["下载备份失败:", "Could not download backup: "],
|
||||
"settings.backup.restore": ["从备份恢复", "Restore from backup"],
|
||||
"settings.backup.uploading": ["正在上传并校验备份…", "Uploading and verifying the backup…"],
|
||||
"settings.backup.restoreFailedPrefix": ["恢复失败:", "Restore failed: "],
|
||||
"settings.backup.confirmTitle": ["用备份替换当前全部数据?", "Replace all current data with this backup?"],
|
||||
"settings.backup.confirmBody": [
|
||||
"当前实例的任务、资产、漏洞、配置、文件和流量都会被「{{name}}」中的内容替换,且无法撤销。上传校验通过后程序会重启并停止当前运行中的任务;备份中未结束的任务在恢复后会保持暂停,不会自动继续执行。建议先下载一份当前数据的备份。",
|
||||
"Tasks, assets, findings, settings, files and traffic on this instance will be replaced by the contents of “{{name}}”. This cannot be undone. After the upload is verified, ARTEX restarts and stops tasks currently running here; any unfinished tasks from the backup start paused rather than resuming automatically. Download a backup of the current data first.",
|
||||
],
|
||||
"settings.backup.cancel": ["取消", "Cancel"],
|
||||
"settings.backup.confirm": ["替换并重启", "Replace and restart"],
|
||||
"settings.backup.restarting": [
|
||||
"备份已校验,正在重启并恢复数据,请稍候(页面会自动刷新)…",
|
||||
"Backup verified. Restarting to restore data. Please wait; the page will reload automatically…",
|
||||
],
|
||||
"settings.backup.restoreTimeout": [
|
||||
"等待服务重启超时(大备份的数据库恢复可能仍在进行)。请查看后端日志、稍后刷新页面,或确认 artex 是通过 start.sh / start.bat 启动的。",
|
||||
"Timed out waiting for the service to restart (a large backup may still be restoring on the server). Check the backend logs, refresh later, or confirm ARTEX was started with start.sh / start.bat.",
|
||||
],
|
||||
"settings.backup.lastRestoreOk": [
|
||||
"已于 {{time}} 从备份恢复(备份版本 {{version}})。",
|
||||
"Restored from a backup at {{time}} (backup version {{version}}).",
|
||||
],
|
||||
"settings.backup.lastRestoreFailed": ["上次恢复失败:{{message}}", "The last restore failed: {{message}}"],
|
||||
"settings.backup.pending": [
|
||||
"有一份已上传的备份等待重启后恢复。",
|
||||
"An uploaded backup will be restored at the next restart.",
|
||||
],
|
||||
"settings.backup.note": [
|
||||
"下载备份前需先暂停或等待所有运行中、排队中的任务结束。恢复需要相同或更新版本的 ARTEX;登录密码、Basic Auth 与 LLM 配置随数据库一起迁移,恢复后请使用原实例的密码登录;新机器上已有的同名 skill 保留本机版本。",
|
||||
"Pause or finish active and queued tasks before downloading a backup. Restoring requires the same or a newer ARTEX version. The sign-in password, Basic Auth and LLM settings move with the database, so sign in with the original instance's password afterwards. Skills that already exist on the new machine keep the local copy.",
|
||||
],
|
||||
"settings.keywords.basicAuth": [
|
||||
"基础认证 登录 用户名 密码",
|
||||
"basic auth authentication sign-in login username password",
|
||||
|
||||
@@ -163,6 +163,20 @@ export async function apiFetch(input: string, init?: RequestInit): Promise<Respo
|
||||
return response;
|
||||
}
|
||||
|
||||
export interface BackupStatus {
|
||||
// 已上传、等待重启时恢复的备份。
|
||||
pending: boolean;
|
||||
// 本次启动时执行的恢复结果;未执行恢复时为 null。
|
||||
last_restore: {
|
||||
id?: string;
|
||||
applied: boolean;
|
||||
error?: string;
|
||||
at: string;
|
||||
source_version?: string;
|
||||
backup_created_at?: string;
|
||||
} | null;
|
||||
}
|
||||
|
||||
export type McpServerScope = "read" | "task:create" | "task:control";
|
||||
|
||||
export interface McpServerSettings {
|
||||
@@ -416,6 +430,33 @@ export const api = {
|
||||
version: () => http<{ version: string }>("/version", { cache: "no-store" }),
|
||||
systemMetrics: () => get<SystemMetrics>("/system/metrics"),
|
||||
|
||||
// ---- 一键备份 / 恢复 ----
|
||||
backupStatus: () => get<BackupStatus>("/system/backup/status"),
|
||||
// 备份包可能有数 GB:与任务文件下载一样先 HEAD 校验登录,再交给浏览器直接落盘(cookie 鉴权),不读进内存。
|
||||
downloadBackup: async () => {
|
||||
if (MOCK) throw new ApiError("演示模式不支持备份与恢复", "demo_backup_disabled");
|
||||
const token = getToken();
|
||||
const response = await apiFetch("/api/system/backup/export", {
|
||||
method: "HEAD",
|
||||
headers: token ? { Authorization: `Bearer ${token}` } : {},
|
||||
});
|
||||
if (!response.ok) {
|
||||
const code = response.headers.get("X-Artex-Error-Code") ?? (response.status === 409 ? "backup_busy" : undefined);
|
||||
throw new ApiError(`HEAD /system/backup/export: ${response.status}`, code);
|
||||
}
|
||||
const a = document.createElement("a");
|
||||
a.href = "/api/system/backup/export";
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
a.remove();
|
||||
},
|
||||
restoreBackup: (file: File) =>
|
||||
http<{ ok: boolean; restore_id: string; version: string; created_at: string }>("/system/backup/restore", {
|
||||
method: "POST",
|
||||
body: file,
|
||||
headers: { "Content-Type": "application/octet-stream" },
|
||||
}),
|
||||
|
||||
// ---- auth ----
|
||||
authStatus: () => get<{ initialized: boolean }>("/auth/status"),
|
||||
login: (username: string, password: string) => post<{ token: string }>("/auth/login", { username, password }),
|
||||
|
||||
@@ -2921,7 +2921,6 @@ export const trafficDetail: TrafficDetail = {
|
||||
req: `GET /v1/orders?id=1002 HTTP/1.1
|
||||
Host: api.acme.com
|
||||
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
|
||||
User-Agent: ARTEX-worker/1.0
|
||||
Accept: application/json
|
||||
|
||||
`,
|
||||
|
||||
@@ -2470,6 +2470,9 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
|
||||
throw Object.assign(new Error("演示模式不支持修改 HTTP Basic Auth,请在实际部署中配置"), {
|
||||
code: "demo_basic_auth_disabled",
|
||||
});
|
||||
if (path === "/system/backup/status") return { pending: false, last_restore: null };
|
||||
if (path === "/system/backup/restore")
|
||||
throw Object.assign(new Error("演示模式不支持备份与恢复,请在实际部署中使用"), { code: "demo_backup_disabled" });
|
||||
if (path === "/settings/web-search/test") return { ok: true, count: 5, backend: D.settings.web_search_backend };
|
||||
if (path === "/settings/proxy/test")
|
||||
return {
|
||||
|
||||
Reference in new issue
Block a user