mirror of
https://github.com/RuoJi6/ARTEX.git
synced 2026-11-21 17:01:50 +08:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7d9c5699cb | ||
|
|
6997a46619 | ||
|
|
cc1d84d573 | ||
|
|
a4ad95d716 | ||
|
|
900c176435 | ||
|
|
437a28cddc | ||
|
|
32d8619d36 | ||
|
|
c9b1e240ec | ||
|
|
9b8073a763 | ||
|
|
faf4d4fb4a | ||
|
|
0fed4ba3a3 | ||
|
|
4d66a84ac5 | ||
|
|
570dc1a14c | ||
|
|
171db405dd | ||
|
|
17c2fc5e08 | ||
|
|
40746901db | ||
|
|
e45fa6e7b3 |
No files matched your search
@@ -13,3 +13,6 @@ ARTEX_LLM_PROVIDER=
|
||||
ARTEX_LLM_MODEL=
|
||||
ARTEX_LLM_BASE_URL=
|
||||
ARTEX_LLM_PROXY=
|
||||
|
||||
# HTTP User-Agent;留空时从浏览器 UA 样本中随机抽选,填写则使用固定值
|
||||
ARTEX_HTTP_USER_AGENT=
|
||||
+41
-1
@@ -4,8 +4,48 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- 新增 MCP 服务端:在「系统设置 → MCP 服务端」开启后,外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务、补充提示。新增访问令牌(权限范围、限定企业、任务限额、有效期、吊销)与调用审计。支持来源白名单(IP/CIDR,可配置可信反向代理),在令牌校验之前生效。默认关闭;`/mcp` 只认访问令牌,不经过也无法使用 HTTP Basic Auth。详见 README「MCP 服务端」。
|
||||
### 新增与改进
|
||||
|
||||
- 支持热更新运行中任务的 Worker 并发:任务详情可设置 0(跟随全局)或 1–50;全局工作并发与 Worker 池变更会同步至跟随全局的任务。增容立即补起 Worker,缩容等待当前意图完成后退出,不中断正在执行的工作;详情展示实际存活 Worker 数与目标数。配置持久化,重启恢复时沿用。
|
||||
- `PATCH /api/tasks/{id}` 新增 `worker_count`;任务接口返回 `worker_actual_count`(存活池循环,含缩容等待中的 Worker)与 `worker_target_count`(有效目标数),与 `in_flight`(执行中意图数)区分。
|
||||
|
||||
## [0.3.30] - 2026-11-14
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增 MCP 服务端:外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务及补充提示。提供访问令牌的权限范围、企业限制、任务限额、有效期和吊销,以及调用审计、来源 IP/CIDR 白名单与可信代理配置。默认关闭;该端点使用独立访问令牌认证,不使用 HTTP Basic Auth。
|
||||
- 统一出站 HTTP 请求标识:Go 请求与两个 Python 采集模板从 20 个标准桌面浏览器 User-Agent 中随机选择,保留环境变量和显式请求头覆盖,同一请求的重定向与重试复用选定值。此变化仅涉及 HTTP 请求头。
|
||||
|
||||
### 修复
|
||||
|
||||
- 命令拦截除匹配原始工具输入 JSON 外,也匹配解码后的嵌套字符串值,修复换行、制表符和转义引号导致危险命令漏拦的问题;保留原始 JSON 规则及非 JSON 输入的兼容行为。
|
||||
- 升级 `go-mitmproxy` v1.9.2 → v1.9.5,修复上游 SOCKS5 握手失败时的空指针 panic;失败时返回 502,并正常触发代理失败回调。
|
||||
- 修正 README 中前端构建产物的复制命令,避免目录嵌套并创建目标目录。
|
||||
|
||||
## [0.3.29] - 2026-11-11
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增可选「无成果止损」:按实际运行时长、独立漏洞数量与最低等级做一次检查,未达标时自动收尾并显示「无成果停止」;暂停、排队及计划窗口外不计时,复活后重新计时,默认关闭。停止状态复用 `timeout`,已有 `on_task_timeout` 触发器仍会执行并可能消耗 token。
|
||||
- 任务详情新增只读文件浏览,支持目录导航、最近文件、文本与图片预览、单文件下载及目录 ZIP 下载;展示意图目录对应的标题和状态,并提供中英文界面。
|
||||
|
||||
### 修复
|
||||
|
||||
- 无成果检查与漏洞写入串行化,并在同一事务内重新计算独立漏洞数量,避免临界时刻已达到成果要求的任务被误停止。
|
||||
- 旁路问答摘要为空、被截断或过长时自动重试一次,并细分失败原因;失败摘要不写入缓存,重试计入原有调用次数、用量及超时限制。
|
||||
|
||||
## [0.3.28] - 2026-11-10
|
||||
|
||||
### 新增与改进
|
||||
|
||||
- 新增英文界面和语言切换,覆盖任务、审批、设置、通知及计划任务等页面,并统一常用状态、日期与错误提示的翻译。
|
||||
- 流量页面展示完整流量 ID,支持按 ID 搜索及在 HTTP 环境复制;Agent 可直接使用该 ID 读取已有流量。
|
||||
- 漏洞统计与列表使用相同筛选条件,区分独立漏洞和原始上报记录数量。
|
||||
|
||||
### 修复
|
||||
|
||||
- Basic Auth 会话失效时在当前标签页恢复认证,避免后台请求反复弹出浏览器认证框,并保留应用登录状态。
|
||||
- 修复英文界面预览、覆盖度图谱生命周期及部分页面运行时错误,并为认证、出站探测等错误补充可翻译的错误码。
|
||||
|
||||
## [0.3.18] - 2026-10-12
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# HTTP 请求标识排查
|
||||
|
||||
本次检查覆盖仓库中的 Go HTTP 请求、Python / JavaScript 脚本、安装脚本、前端示例,以及实际使用的 norma v0.4.3 的 WebFetch / WebSearch 实现。排查范围包括 User-Agent 的大小写变体、请求头构造、代理转发、重定向及代理失败后的直连重试。
|
||||
|
||||
## 已处理
|
||||
|
||||
| 通道 | 排查结果与处理 |
|
||||
|---|---|
|
||||
| 内置 WebFetch | SDK 将 `norma/0.4` 写死在首发及重试请求。worker、planner、main、chat 通过统一 session 入口使用本地适配器;保留 SDK 的工具 schema、权限、静态资源限制、代理、CA 验证及 HTML / RECON 输出。 |
|
||||
| 资产自动补全 | 移除 `artex-enrich/1.0`,HTTP 探测使用统一策略。 |
|
||||
| 出口连通性探测 | 额外发现 `artex-egress-probe/1.0`,已移除并接入统一策略。 |
|
||||
| 静态采集 / 同源爬虫模板 | 移除 `spa-api-recon` / `spa-api-recon spider`,避免 urllib 自动填入自己的默认 UA;入口、下载、重定向及重试沿用同一配置。 |
|
||||
| HTTP 自定义工具 | 原先没有 ARTEX 专属 UA,但未设置时会由 Go 填充默认值。现采用统一策略,模板显式请求头优先。 |
|
||||
| 请求重放器 | 原先没有 ARTEX 专属 UA。现采用统一策略,用户显式请求头(包括空值)优先,保持原请求的配置能力。 |
|
||||
| 记录代理 | 转发缺失 UA 的请求时,Go 会补入默认客户端标识。现保留显式省略语义,录制中不展示内部用于抑制默认值的空头。保留客户端已选择的随机或固定 UA。 |
|
||||
| GitHub 自更新 | 额外发现查询版本、下载校验和与发布包共用的两处 `artex-selfupdate`。均已移除;与其他请求统一采用随机浏览器样本或显式配置的非空 UA,不含默认 ARTEX 名称。 |
|
||||
| 前端 mock | `ARTEX-worker/1.0` 仅为示例 HTTP 文本,不会发出请求;已删除该行,避免误导。 |
|
||||
|
||||
## 配置
|
||||
|
||||
默认采用随机 UA 池方案,Go 与两个独立 Python 模板内置相同的 20 个完整浏览器 UA 样本,覆盖 Chrome / Edge / Firefox / Safari 的标准桌面格式。随机选择完整样本,允许重复,不拼接随机版本,不调用 AI 服务。
|
||||
|
||||
2026-11-11 核对并更新样本,采用主流浏览器近期稳定版本的标准格式:
|
||||
|
||||
| 浏览器 | 版本 | 平台 | 数量 |
|
||||
|---|---|---|---|
|
||||
| Chrome | 154 / 155 / 156 / 157 | Windows / macOS / Linux | 12 |
|
||||
| Edge | 156 / 157 | Windows / macOS | 4 |
|
||||
| Firefox | 159 | Windows / macOS / Linux | 3 |
|
||||
| Safari | 27.1 | macOS | 1 |
|
||||
|
||||
保留浏览器正常的 UA 缩减字段(如 Chrome 的 `0.0.0` 和 macOS 的冻结版本),不添加工具名称、随机设备型号、扩展标记或自定义后缀。“常见”指主流浏览器的标准 UA 格式,不代表已测量这 20 项在目标站点中的使用占比;浏览器类型和版本仍可从 UA 中读出。
|
||||
|
||||
格式与版本参考:
|
||||
|
||||
- [Chromium UA Reduction](https://www.chromium.org/updates/ua-reduction/)
|
||||
- [Chrome 稳定版发布记录](https://chromereleases.googleblog.com/search/label/Stable%20updates)
|
||||
- [Edge UA 格式](https://learn.microsoft.com/en-us/microsoft-edge/web-platform/user-agent-guidance)
|
||||
- [Edge 稳定版发布记录](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-relnote-stable-channel)
|
||||
- [Firefox UA 格式](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent/Firefox)
|
||||
- [Firefox 159 发布记录](https://www.firefox.com/en-US/firefox/159.0/releasenotes/)
|
||||
- [Safari 27.1 发布记录](https://developer.apple.com/documentation/safari-release-notes/safari-27_1-release-notes)
|
||||
|
||||
- Go 每次 HTTP 逻辑请求抽选一次;WebFetch 每次工具调用重新抽选,重定向、代理请求及失败后的直连重试复用同一值。
|
||||
- Python 脚本每次运行抽选一次,入口、并发资源下载及重试复用同一值;独立调用 `fetch()` 时每次调用单独选择。
|
||||
- `ARTEX_HTTP_USER_AGENT` 非空且合法时覆盖随机值;留空、仅空格或非法值回退随机池。Go 使用系统加密随机源;若随机源出错,保留池中首个非空有效样本。
|
||||
- Python 的 `--user-agent VALUE` 优先于环境变量,`--user-agent ''` 仍显式省略;自定义 HTTP 工具 / 重放器显式提供的 UA(包括空值)同样优先,保留原请求的控制能力。
|
||||
- 固定值只接受可打印 ASCII,去除首尾空格;控制字符和非 ASCII 值回退随机池,防止头注入。
|
||||
|
||||
Go 服务重启后使用新配置;Docker 在 `.env` 设置后重建容器。直接运行 Python 模板时继承同名环境变量。已同步 agent OPSEC 提示,避免旧提示与平台随机策略冲突;遇到限速等异常时仍停止,不以换 UA 继续当前方向。
|
||||
|
||||
UA 随机化不会同步改变 TLS、Cookie 或其他浏览器特征,不能保证客户端不可识别;第三方出口代理也可能再次改写请求头。
|
||||
|
||||
## 检查过的其他通道与边界
|
||||
|
||||
| 通道 | 结果 |
|
||||
|---|---|
|
||||
| SDK 联网搜索 | norma 的 DDGS 后端含固定 Firefox 128 浏览器 UA,没有 ARTEX / norma 名称;仅发往搜索服务,属于依赖的兼容策略,未改动,也不受此环境变量控制。Brave / Tavily 及 DeepSeek 搜索未发现 ARTEX 专属 UA。 |
|
||||
| 浏览器自动化 / runtime_harvest.js | 沿用浏览器请求头;运行时转发复制捕获到的请求头,没有硬编码 ARTEX UA。 |
|
||||
| LLM 请求 / 模型列表 / JEV | 未发现硬编码 ARTEX UA;仍可能有 HTTP 库、SDK、服务协议和用户自定义标识,不套用目标流量配置。 |
|
||||
| MCP HTTP / SSE | 未发现 ARTEX UA;显式自定义头保留。初始化协议体仍有 `clientInfo.name = artex`,这是对用户配置的 MCP 服务的协议自报身份,本次不改动。 |
|
||||
| Webhook / 通知 | 未发现硬编码 ARTEX UA;保留用户自定义头及客户端默认行为。 |
|
||||
| 安装脚本 | 未发现 ARTEX UA;获取 Docker 安装脚本的 curl 仍使用其原生身份。外部命令及用户自建脚本不由本次策略接管。 |
|
||||
| phpfinger | 仓库未包含其源码,无法核验默认值;需要在独立工具中通过 `--user-agent` 配置。 |
|
||||
|
||||
服务端响应中的 `X-Artex-Auth` 仍用于前端识别 Basic Auth 未认证状态,属于站点自身对访问者的响应协议;它不是向目标站点发送的请求头。本次不改变该协议,也不承诺消除服务端页面、API、TLS 或其他流量特征。
|
||||
|
||||
## 回归验证
|
||||
|
||||
- Go:检查真实 HTTP 线上头、随机样本选择 / 固定值覆盖 / 显式空值、代理及失败直连重试、重定向、CA 信任、静态资源限制、HTML / RECON 解析、agent session 注册、自定义工具 / 重放器覆盖、自更新查询及下载和代理录制。
|
||||
- Python:本地 HTTP 服务验证默认随机 / 环境变量 / CLI 覆盖 / 显式空值、非法头值、重定向、Cookie 保留及产物生成;不访问真实目标。
|
||||
|
||||
本地命令:
|
||||
|
||||
```sh
|
||||
go test ./internal/httpidentity ./internal/webfetch ./enrich ./agent ./selfupdate ./traffic
|
||||
go test ./server -run 'TestProbe|TestRepeater|TestRunHTTPTool|TestHTTPTool|TestTargetHTTPIdentity' -count=1
|
||||
go test -race ./internal/httpidentity ./internal/webfetch ./enrich ./traffic
|
||||
python3 -m unittest discover -s skills/api-recon/scripts -p 'test_*.py' -v
|
||||
```
|
||||
|
||||
以上专项回归、Go race 检查及 Python 3 项测试均通过。池扩展后的回归还检查恰好 20 个不重复样本、逐项选择以及 Go 与两个 Python 模板的样本一致性。
|
||||
|
||||
2026-11-11 使用项目提供的本地 PostgreSQL,在新建的独立测试库中执行了 `go test ./... -count=1 -p 1 -timeout 180s`,测试库用后删除。完整套件未全部通过:
|
||||
|
||||
- `agent` 的 `TestGraphOverviewExpandsAssociatedCompanyScope`、`TestCaptureApprovalLifecycle` 和 `db` 的 `TestInterceptApprovalFilters`、`TestSideTaskArchiveVersions` 失败;在原提交 `17c2fc5` 的独立副本和另一新建测试库中复现了相同失败。
|
||||
- `server` 首次运行有 3 个用例在临时目录清理阶段报 `directory not empty`,涉及 `TestInheritedActivityDetailAndRelationDeletion`、`TestTaskMetadataPatchReturnsRenameAndPin`、`TestTaskMetadataPatchGlobalAssetSharing`,不是 UA 断言失败。再次在新建隔离库中运行完整 `server` 套件时这 3 项通过,但 `TestCoreTaskLifecyclePG` 报 `expected goal nodes seeded, got 0`。原提交的完整 `server` 套件通过;修改后完整 `server` 套件仍未获得全绿结果。
|
||||
@@ -12,8 +12,13 @@ AI 自主渗透测试系统(Go 后端 + Next.js 前端)
|
||||
本仓库为独立维护的 [RuoJi6/ARTEX](https://github.com/RuoJi6/ARTEX),保留 https://github.com/Autumn-27/ARTEX 的原始提交历史与 AGPL-3.0 许可证。程序发布包从本仓库 Releases 下载,Docker 镜像 [ruoji6/artex](https://hub.docker.com/r/ruoji6/artex) 由本仓库 GitHub Actions 构建并发布。
|
||||
|
||||
---
|
||||
## 致谢
|
||||
致谢米斯特安全团队
|
||||
## 致谢
|
||||
米斯特安全团队
|
||||
|
||||
洺熙
|
||||
|
||||
wowo
|
||||
|
||||
|
||||
## 截图预览
|
||||
|
||||
@@ -157,9 +162,11 @@ cp config.example.json config.json # 填好 database 连接
|
||||
```bash
|
||||
# 1) 前端静态导出
|
||||
cd web && npm ci && npm run build:static && cd ..
|
||||
# 2) 拷进内嵌目录
|
||||
cp -r web/out server/webui/dist
|
||||
# 3) 编译(-tags embedui 才内嵌前端)
|
||||
# 2) 创建内嵌目录
|
||||
mkdir -p server/webui/dist
|
||||
# 3) 拷进内嵌目录
|
||||
cp -r web/out/. server/webui/dist
|
||||
# 4) 编译(-tags embedui 才内嵌前端)
|
||||
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
./start.sh
|
||||
```
|
||||
@@ -290,7 +297,8 @@ cp -r <解压目录>/skills ./ && cp <解压目录>/artex ./
|
||||
```bash
|
||||
git pull
|
||||
cd web && npm ci && npm run build:static && cd ..
|
||||
cp -r web/out server/webui/dist
|
||||
mkdir -p server/webui/dist
|
||||
cp -r web/out/. server/webui/dist
|
||||
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
# 重启 ./start.sh
|
||||
```
|
||||
@@ -314,6 +322,10 @@ CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
|
||||
**LLM**:`export ANTHROPIC_API_KEY=sk-...`(或 `OPENAI_API_KEY`),也可在 UI 的「LLM 配置」页填写。
|
||||
可选:`ARTEX_LLM_PROVIDER` / `ARTEX_LLM_MODEL` / `ARTEX_LLM_BASE_URL` / `ARTEX_LLM_PROXY`。
|
||||
|
||||
**HTTP User-Agent**:内置 WebFetch(worker / planner / main / chat)、资产 HTTP 自动补全、出口连通性探测、HTTP 自定义工具、请求重放器、GitHub 自更新和可选的 `harvest_static.py` / `spider_mpa.py` 默认从 20 个浏览器 UA 样本中随机抽选,不携带工具名称。Go 每次逻辑请求选择一次,重定向及 WebFetch 的代理失败直连重试复用同一个值;Python 脚本每次运行选择一次,入口、下载与重试保持一致。随机抽选允许重复,不依赖 AI 服务。通过 `ARTEX_HTTP_USER_AGENT` 可指定固定值;留空或非法值使用随机策略。直接运行时导出环境变量并重启服务,Docker 在 `.env` 配置后重建容器。Python 的 `--user-agent VALUE`、自定义工具和重放器中的显式 UA 优先,显式空值仍可省略此头。值去除首尾空格,仅接受可打印 ASCII。记录代理保留已有 UA 及显式省略语义。UA 随机化不代表 TLS、Cookie 或其他浏览器特征同步改变,也不保证无法识别客户端。此设置不影响浏览器、SDK 联网搜索、LLM / MCP / Webhook 服务通信或外部独立工具;本仓库未包含 phpfinger,其 `--user-agent` 需在该工具中单独配置。
|
||||
|
||||
完整排查范围、额外发现及未接管的服务通信见 [HTTP 请求标识排查](HTTP_USER_AGENT.md)。
|
||||
|
||||
**并发**:每个任务的 work agent 数在「系统设置」里配置(默认 3)。
|
||||
|
||||
**常用参数**:`./start.sh -addr :8787 -proxy :8788`(`-addr` 前端+API,`-proxy` 流量录制代理)。启动脚本会把参数原样透传给 `artex`。
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ import (
|
||||
// contiguous run is coalesced into a single record so the trace shows whole
|
||||
// messages, not dozens of fragments.
|
||||
func captureRun(ctx context.Context, opts agentcore.Options, input string, emit func(db.Activity)) (string, harness.TerminalReason, error) {
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close() // release the session's background-task manager (temp dir + processes)
|
||||
return captureRunSession(ctx, s, input, emit)
|
||||
}
|
||||
|
||||
+1
-1
@@ -173,7 +173,7 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
|
||||
// 实验功能:开启后由 noa 接管上下文压缩(归档集中在 <workDir>/noa/<SessionID> 下,持久)。
|
||||
enableNoa(&opts, c.noaEnabledFn, c.workDir, "chat-"+sessionID, noaWarn("chat-"+sessionID))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close()
|
||||
// reload prior conversation so the agent has context across turns (each Chat is
|
||||
// a fresh session). First turn: no file yet → Resume loads nothing and proceeds.
|
||||
|
||||
+1
-1
@@ -184,7 +184,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
|
||||
}
|
||||
enableNoa(&opts, m.noaEnabledFn, m.workDir, noaSession, noaWarn(noaSession))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close()
|
||||
// reload the prior conversation from the transcript so the agent has context
|
||||
// across turns (each Chat is a fresh session; without this it can't see earlier
|
||||
|
||||
+2
-2
@@ -21,9 +21,9 @@ const opsecPromptBlock = `【OPSEC 准则(硬约束,仅次于授权范围和
|
||||
|
||||
- 先复用已有流量、浏览器会话和已验证证据;能通过 traffic_search/traffic_get 或浏览器完成时,不要重复直接请求。
|
||||
- 所有目标交互必须经过平台配置的记录代理或浏览器工具,确保请求可追溯。
|
||||
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。命令行请求必须尽量复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;没有真实依据时不要伪造身份、设备或浏览器指纹。
|
||||
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。优先复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;平台 HTTP 工具和采集模板的默认 UA 由系统随机策略提供,同一次请求的重定向、重试必须保持一致,明确配置的 UA 优先。
|
||||
- 不得使用 -k、--ignore-https-errors、verify=False 等方式绕过证书校验;平台代理 CA 已配置时应正常验证。
|
||||
- 不得随机轮换 User-Agent、批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向。
|
||||
- 不得批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向,不得通过更换 UA 继续当前方向。
|
||||
- OPSEC 不扩大授权范围;目标、主机、端口、路径或账号不在授权范围内时不得访问。
|
||||
- 请求和响应是否留痕由平台代理和浏览器负责;不要额外复制完整请求头、Cookie 或响应正文。`
|
||||
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"github.com/Autumn-27/artex/internal/webfetch"
|
||||
"github.com/Autumn-27/norma/agentcore"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
|
||||
// newAgentSession installs the configurable WebFetch before the SDK builds the
|
||||
// registry, so every agent role gets one tool and the same proxy/TLS settings.
|
||||
func newAgentSession(opts agentcore.Options) *agentcore.Session {
|
||||
if opts.EnableWebFetch {
|
||||
opts.EnableWebFetch = false
|
||||
if opts.Tools == nil {
|
||||
opts.Tools = actool.DefaultTools()
|
||||
}
|
||||
opts.Tools = append(opts.Tools, webfetch.New(actool.WebFetchConfig{
|
||||
Proxy: opts.WebFetchProxy,
|
||||
CACert: opts.WebFetchCACert,
|
||||
InsecureTLS: opts.WebFetchInsecureTLS,
|
||||
}))
|
||||
}
|
||||
return agentcore.NewSession(opts)
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Autumn-27/norma/agentcore"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
"github.com/Autumn-27/norma/permission"
|
||||
)
|
||||
|
||||
type fetchFixtureProvider struct {
|
||||
autoTitleProvider
|
||||
t *testing.T
|
||||
url string
|
||||
calls int
|
||||
}
|
||||
|
||||
func (p *fetchFixtureProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
p.calls++
|
||||
if p.calls == 1 {
|
||||
counts := map[string]int{}
|
||||
for _, tool := range req.Tools {
|
||||
counts[tool.Name]++
|
||||
}
|
||||
if counts["WebFetch"] != 1 || counts["Read"] != 1 || counts["Bash"] != 1 {
|
||||
p.t.Errorf("session tool registration = %v", counts)
|
||||
}
|
||||
input, _ := json.Marshal(map[string]string{"url": p.url})
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{{Type: llm.BlockToolUse, ID: "fetch-fixture", Name: "WebFetch", Input: input}}}, "tool_use", llm.Usage{}, nil
|
||||
}
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("finished")}}, "end_turn", llm.Usage{}, nil
|
||||
}
|
||||
|
||||
func TestAgentSessionUsesConfiguredWebFetch(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
calls := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if ua == "" && !strings.HasPrefix(r.UserAgent(), "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", r.UserAgent())
|
||||
} else if ua != "" && r.UserAgent() != ua {
|
||||
t.Errorf("session sent User-Agent %q, want %q", r.UserAgent(), ua)
|
||||
}
|
||||
w.Write([]byte("fixture"))
|
||||
}))
|
||||
defer server.Close()
|
||||
provider := &fetchFixtureProvider{t: t, url: server.URL}
|
||||
session := newAgentSession(agentcore.Options{
|
||||
Provider: provider, NonStreaming: true, EnableWebFetch: true,
|
||||
PermissionMode: permission.ModeBypass, MaxTurns: 3,
|
||||
})
|
||||
defer session.Close()
|
||||
for _, err := range session.Prompt(t.Context(), "fetch local fixture") {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("WebFetch calls = %d, want 1", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -488,7 +488,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
|
||||
noaSession := WorkerSessionID(ts.ID(), intent.ID)
|
||||
enableNoa(&opts, w.noaEnabledFn, w.workDir, noaSession, noaWarn(noaSession))
|
||||
ctx = attachSideCapture(ctx, &opts)
|
||||
s := agentcore.NewSession(opts)
|
||||
s := newAgentSession(opts)
|
||||
defer s.Close() // release the session's background-task manager (temp dir + processes)
|
||||
|
||||
// Resume prior conversation if this intent was paused/blocked/exhausted and is
|
||||
|
||||
@@ -102,3 +102,41 @@ func TestDeleteConversationsReturnsExistingIDs(t *testing.T) {
|
||||
t.Fatalf("deleted=%v, want existing ids only", deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskPatchWorkerCountPersistsAndValidates(t *testing.T) {
|
||||
d, err := Open(testDSN(t))
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v) — skipping", err)
|
||||
}
|
||||
defer d.Close()
|
||||
task, err := d.CreateTask("worker count patch", "goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { _ = d.DeleteTask(task.ID) }()
|
||||
for _, count := range []int{1, MaxTaskWorkerCount, 0, 4} {
|
||||
updated, err := d.UpdateTask(task.ID, TaskPatch{WorkerCount: &count})
|
||||
if err != nil || updated == nil || updated.WorkerCount != count {
|
||||
t.Fatalf("patch worker_count=%d task=%+v err=%v", count, updated, err)
|
||||
}
|
||||
persisted, err := d.GetTask(task.ID)
|
||||
if err != nil || persisted == nil || persisted.WorkerCount != count {
|
||||
t.Fatalf("read worker_count=%d task=%+v err=%v", count, persisted, err)
|
||||
}
|
||||
}
|
||||
name := "renamed without worker override"
|
||||
updated, err := d.UpdateTask(task.ID, TaskPatch{Name: &name})
|
||||
if err != nil || updated == nil || updated.WorkerCount != 4 {
|
||||
t.Fatalf("omitted worker_count changed override: task=%+v err=%v", updated, err)
|
||||
}
|
||||
for _, count := range []int{-1, MaxTaskWorkerCount + 1} {
|
||||
invalidName := "must not commit"
|
||||
if updated, err := d.UpdateTask(task.ID, TaskPatch{Name: &invalidName, WorkerCount: &count}); err == nil || updated != nil {
|
||||
t.Fatalf("accepted invalid worker_count=%d: task=%+v err=%v", count, updated, err)
|
||||
}
|
||||
persisted, err := d.GetTask(task.ID)
|
||||
if err != nil || persisted == nil || persisted.WorkerCount != 4 || persisted.Name != name {
|
||||
t.Fatalf("invalid patch changed task=%+v err=%v", persisted, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
+14
-4
@@ -165,6 +165,9 @@ func normalizeWorkerCount(n int) int {
|
||||
}
|
||||
|
||||
func validateWorkerCount(n int) error {
|
||||
if n < 0 {
|
||||
return fmt.Errorf("Worker 运行数量不能为负数")
|
||||
}
|
||||
if n > MaxTaskWorkerCount {
|
||||
return fmt.Errorf("Worker 运行数量最多 %d", MaxTaskWorkerCount)
|
||||
}
|
||||
@@ -471,17 +474,23 @@ ORDER BY (pinned_at IS NOT NULL) DESC, pinned_at DESC NULLS LAST, id DESC`)
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// TaskPatch updates editable metadata and global asset visibility.
|
||||
// TaskPatch updates editable metadata, asset visibility, and worker concurrency.
|
||||
type TaskPatch struct {
|
||||
Name *string
|
||||
Remark *string
|
||||
Pinned *bool
|
||||
ShareGlobalAssets *bool
|
||||
WorkerCount *int
|
||||
}
|
||||
|
||||
// UpdateTask applies a partial metadata/visibility mutation and returns the updated row.
|
||||
// UpdateTask applies a partial metadata/runtime-policy mutation and returns the updated row.
|
||||
// Re-pinning an already pinned task preserves its original position.
|
||||
func (d *DB) UpdateTask(id int64, patch TaskPatch) (*Task, error) {
|
||||
if patch.WorkerCount != nil {
|
||||
if err := validateWorkerCount(*patch.WorkerCount); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
task, err := scanTask(d.QueryRow(`UPDATE tasks SET
|
||||
name = CASE WHEN $2::boolean THEN $3 ELSE name END,
|
||||
remark = CASE WHEN $5::boolean THEN $6 ELSE remark END,
|
||||
@@ -490,9 +499,10 @@ func (d *DB) UpdateTask(id int64, patch TaskPatch) (*Task, error) {
|
||||
WHEN $4::boolean THEN COALESCE(pinned_at, now())
|
||||
ELSE NULL
|
||||
END,
|
||||
share_global_assets = COALESCE($7::boolean, share_global_assets)
|
||||
share_global_assets = COALESCE($7::boolean, share_global_assets),
|
||||
worker_count = COALESCE($8::integer, worker_count)
|
||||
WHERE id=$1 AND deleted_at IS NULL
|
||||
RETURNING `+taskCols, id, patch.Name != nil, patch.Name, patch.Pinned, patch.Remark != nil, patch.Remark, patch.ShareGlobalAssets))
|
||||
RETURNING `+taskCols, id, patch.Name != nil, patch.Name, patch.Pinned, patch.Remark != nil, patch.Remark, patch.ShareGlobalAssets, patch.WorkerCount))
|
||||
if err == sql.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
@@ -28,6 +28,7 @@ services:
|
||||
ARTEX_LLM_MODEL: ${ARTEX_LLM_MODEL:-}
|
||||
ARTEX_LLM_BASE_URL: ${ARTEX_LLM_BASE_URL:-}
|
||||
ARTEX_LLM_PROXY: ${ARTEX_LLM_PROXY:-}
|
||||
ARTEX_HTTP_USER_AGENT: ${ARTEX_HTTP_USER_AGENT:-}
|
||||
ports:
|
||||
- "8787:8787"
|
||||
- "127.0.0.1:8788:8788"
|
||||
|
||||
+2
-1
@@ -20,6 +20,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
|
||||
"github.com/miekg/dns"
|
||||
"github.com/projectdiscovery/dnsx/libs/dnsx"
|
||||
@@ -230,7 +231,7 @@ func (e *Engine) doHTTP(id int64, rawURL string) {
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-enrich/1.0")
|
||||
httpidentity.Apply(req)
|
||||
resp, err := e.client.Do(req)
|
||||
if err != nil {
|
||||
return
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package enrich
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Stop after receiving the probe so this request regression needs no database.
|
||||
func TestProbeUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
calls := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls++
|
||||
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
http.Redirect(w, r, "/stop", http.StatusFound)
|
||||
}))
|
||||
defer server.Close()
|
||||
client := buildClient(nil)
|
||||
client.CheckRedirect = func(*http.Request, []*http.Request) error {
|
||||
return errors.New("probe captured")
|
||||
}
|
||||
e := &Engine{client: client}
|
||||
e.doHTTP(1, server.URL)
|
||||
if calls != 1 {
|
||||
t.Fatalf("probe calls = %d, want 1", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -3,19 +3,19 @@ module github.com/Autumn-27/artex
|
||||
go 1.26.10
|
||||
|
||||
require (
|
||||
mvdan.cc/sh/v3 v3.10.0
|
||||
github.com/Autumn-27/norma v0.4.3
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/jackc/pgx/v5 v5.10.0
|
||||
github.com/klauspost/compress v1.18.6
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5
|
||||
github.com/miekg/dns v1.1.72
|
||||
github.com/projectdiscovery/dnsx v1.2.3
|
||||
golang.org/x/crypto v0.55.0
|
||||
golang.org/x/net v0.58.0
|
||||
golang.org/x/text v0.41.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/net v0.61.0
|
||||
golang.org/x/text v0.43.0
|
||||
modernc.org/sqlite v1.52.0
|
||||
mvdan.cc/sh/v3 v3.10.0
|
||||
)
|
||||
|
||||
require (
|
||||
@@ -52,10 +52,10 @@ require (
|
||||
go.uber.org/atomic v1.11.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
|
||||
golang.org/x/mod v0.40.0 // indirect
|
||||
golang.org/x/sync v0.22.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/tools v0.49.0 // indirect
|
||||
golang.org/x/mod v0.41.0 // indirect
|
||||
golang.org/x/sync v0.24.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/tools v0.51.0 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
modernc.org/libc v1.72.3 // indirect
|
||||
modernc.org/mathutil v1.7.1 // indirect
|
||||
|
||||
@@ -48,8 +48,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
|
||||
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2 h1:+QTbDyUUaVRmEavPq0/eWzJ2K+K4JdL4ff91s/adnxk=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.2/go.mod h1:6H5PR+nbVsiGQJC/Y98GHJItyFcmP/Z0txAgjhG2a80=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5 h1:4ar1qWYlbXF+7HhG06AVjKkDILXX1FS0mNy+hkX4uX4=
|
||||
github.com/lqqyt2423/go-mitmproxy v1.9.5/go.mod h1:74Z1IyzuUu/pq1IpdDd555f1xXM7Gl9KtUhWn/mzU7E=
|
||||
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
|
||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
|
||||
@@ -108,21 +108,35 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
|
||||
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
|
||||
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=
|
||||
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
|
||||
golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
|
||||
golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
|
||||
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
|
||||
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
|
||||
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
|
||||
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
|
||||
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
|
||||
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
|
||||
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
|
||||
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
|
||||
golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
|
||||
golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
|
||||
+50
-9
@@ -572,8 +572,9 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
|
||||
if err != nil || len(rules) == 0 {
|
||||
return Decision{}, false
|
||||
}
|
||||
values := inputStrings(input)
|
||||
for _, r := range rules {
|
||||
if ruleMatches(r, toolName, input) {
|
||||
if ruleMatches(r, toolName, input, values) {
|
||||
msg := r.Message
|
||||
if msg == "" {
|
||||
msg = defaultMessage(r.Action, r.Name)
|
||||
@@ -593,20 +594,60 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
|
||||
return Decision{}, false
|
||||
}
|
||||
|
||||
func ruleMatches(r compiledRule, toolName string, input []byte) bool {
|
||||
var subject string
|
||||
// ruleMatches reports whether r applies to the call. tool_input rules are tried
|
||||
// against the raw JSON first, then against each decoded string value in values.
|
||||
// The raw form alone is not enough: JSON escapes newlines, tabs and quotes, and
|
||||
// Go's encoder also escapes & < >, so a rule like `\brm\b` never fires on
|
||||
// "cd /tmp\nrm -rf /var/www" (the raw bytes read `\nrm`, with no word boundary).
|
||||
func ruleMatches(r compiledRule, toolName string, input []byte, values []string) bool {
|
||||
switch r.MatchTarget {
|
||||
case "tool_name":
|
||||
subject = toolName
|
||||
return r.matchString(toolName)
|
||||
case "tool_input":
|
||||
subject = string(input)
|
||||
default:
|
||||
return false
|
||||
if r.matchString(string(input)) {
|
||||
return true
|
||||
}
|
||||
for _, v := range values {
|
||||
if r.matchString(v) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (r compiledRule) matchString(s string) bool {
|
||||
if r.MatchType == "regex" {
|
||||
return r.re != nil && r.re.MatchString(subject)
|
||||
return r.re != nil && r.re.MatchString(s)
|
||||
}
|
||||
return strings.Contains(subject, r.Pattern)
|
||||
return strings.Contains(s, r.Pattern)
|
||||
}
|
||||
|
||||
// inputStrings returns every string value in a JSON tool input, at any depth.
|
||||
// Object keys are skipped. It returns nil if the input is not valid JSON.
|
||||
func inputStrings(input []byte) []string {
|
||||
var doc any
|
||||
if json.Unmarshal(input, &doc) != nil {
|
||||
return nil
|
||||
}
|
||||
var out []string
|
||||
var walk func(any)
|
||||
walk = func(v any) {
|
||||
switch v := v.(type) {
|
||||
case string:
|
||||
out = append(out, v)
|
||||
case []any:
|
||||
for _, e := range v {
|
||||
walk(e)
|
||||
}
|
||||
case map[string]any:
|
||||
for _, e := range v {
|
||||
walk(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(doc)
|
||||
return out
|
||||
}
|
||||
|
||||
func defaultMessage(action, name string) string {
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
package intercept
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"regexp"
|
||||
"testing"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
)
|
||||
|
||||
// Patterns copied from the built-in rules seeded in db/db.go.
|
||||
const (
|
||||
builtinRmRf = `(?i)\brm\b.{0,80}(?:-[a-z]*r[a-z]*f[a-z]*|-[a-z]*f[a-z]*r[a-z]*|--recursive|--no-preserve-root)`
|
||||
builtinMkfs = `\bmkfs\b`
|
||||
builtinDrop = `(?i)\bDROP\s+(?:DATABASE|TABLE|SCHEMA|INDEX|VIEW|TABLESPACE|USER|ROLE)\b`
|
||||
builtinFlush = `(?i)\b(?:FLUSHALL|FLUSHDB)\b`
|
||||
builtinV2 = `(?i)\b(rm\s+-rf\s+/|mkfs|dd\s+if=|:\(\)\s*\{|shutdown|reboot|>\s*/dev/sd)`
|
||||
)
|
||||
|
||||
func testRule(target, matchType, pattern string) compiledRule {
|
||||
r := compiledRule{InterceptRule: db.InterceptRule{
|
||||
ID: 1, Name: "test", Enabled: true, Action: "deny",
|
||||
MatchTarget: target, MatchType: matchType, Pattern: pattern,
|
||||
}}
|
||||
if matchType == "regex" {
|
||||
r.re = regexp.MustCompile(pattern)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// interceptorWith returns an Interceptor whose rule cache is pre-filled, so
|
||||
// Match runs without a database.
|
||||
func interceptorWith(rules ...compiledRule) *Interceptor {
|
||||
return &Interceptor{cached: rules}
|
||||
}
|
||||
|
||||
func toolInput(t *testing.T, v any) []byte {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func TestMatchToolInputSeesDecodedValues(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
pattern string
|
||||
tool string
|
||||
input any
|
||||
}{
|
||||
{"rm -rf on its own line", builtinRmRf, "Bash", map[string]string{"command": "cd /tmp\nrm -rf /var/www"}},
|
||||
{"rm -rf after a tab", builtinRmRf, "Bash", map[string]string{"command": "echo done\trm -rf /var/www"}},
|
||||
{"mkfs on its own line", builtinMkfs, "Bash", map[string]string{"command": "lsblk\nmkfs.ext4 /dev/sdb"}},
|
||||
{"DROP inside a heredoc", builtinDrop, "Bash", map[string]string{"command": "psql <<EOF\nDROP TABLE users;\nEOF"}},
|
||||
{"FLUSHALL inside a heredoc", builtinFlush, "Bash", map[string]string{"command": "redis-cli <<EOF\nFLUSHALL\nEOF"}},
|
||||
{"shutdown on its own line", builtinV2, "Bash", map[string]string{"command": "sync\nshutdown -h now"}},
|
||||
// encoding/json writes > as >, so this only matches the decoded value.
|
||||
{"redirect to a disk", `>\s*/dev/sd`, "Bash", map[string]string{"command": "cat img > /dev/sda"}},
|
||||
{"interactive shell text", builtinRmRf, "shell_send", map[string]string{"text": "ls\nrm -rf /srv/data\n"}},
|
||||
{"nested edit value", builtinDrop, "MultiEdit", map[string]any{
|
||||
"file_path": "/tmp/m.sql",
|
||||
"edits": []map[string]string{{"old_string": "-- x", "new_string": "-- x\nDROP TABLE users;"}},
|
||||
}},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", tc.pattern))
|
||||
in := toolInput(t, tc.input)
|
||||
if _, ok := ic.Match(tc.tool, in); !ok {
|
||||
t.Fatalf("rule %q did not match input %s", tc.pattern, in)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputStringRuleSeesDecodedValues(t *testing.T) {
|
||||
// The raw JSON holds rm -rf \"/var/www\", so the quotes only line up once decoded.
|
||||
ic := interceptorWith(testRule("tool_input", "string", `rm -rf "/var/www"`))
|
||||
in := toolInput(t, map[string]string{"command": `rm -rf "/var/www"`})
|
||||
if _, ok := ic.Match("Bash", in); !ok {
|
||||
t.Fatalf("string rule did not match input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
// Rules written against the raw JSON form must keep working.
|
||||
func TestMatchToolInputStillMatchesRawJSON(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", `"command":"rm `))
|
||||
in := toolInput(t, map[string]string{"command": "rm notes.txt"})
|
||||
if _, ok := ic.Match("Bash", in); !ok {
|
||||
t.Fatalf("raw JSON rule did not match input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputInvalidJSONFallsBackToRaw(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", builtinMkfs))
|
||||
if _, ok := ic.Match("Bash", []byte(`not json: mkfs /dev/sdb`)); !ok {
|
||||
t.Fatal("rule did not match non-JSON input")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolInputNoFalsePositives(t *testing.T) {
|
||||
ic := interceptorWith(
|
||||
testRule("tool_input", "regex", builtinRmRf),
|
||||
testRule("tool_input", "regex", builtinMkfs),
|
||||
testRule("tool_input", "regex", builtinDrop),
|
||||
)
|
||||
for _, cmd := range []string{
|
||||
"cd /tmp\nls -la",
|
||||
"nmap -sV 10.0.0.5\ncurl -s http://10.0.0.5/",
|
||||
"echo firm\nrmdir build", // "firm" and "rmdir" must not look like rm
|
||||
} {
|
||||
in := toolInput(t, map[string]string{"command": cmd})
|
||||
if dec, ok := ic.Match("Bash", in); ok {
|
||||
t.Fatalf("unexpected match %q for input %s", dec.RuleName, in)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The match key ("command") must not be matched as a value.
|
||||
func TestMatchToolInputIgnoresObjectKeys(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_input", "regex", `^mkfs$`))
|
||||
in := toolInput(t, map[string]string{"mkfs": "ls"})
|
||||
if _, ok := ic.Match("Bash", in); ok {
|
||||
t.Fatalf("object key matched as a value for input %s", in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMatchToolNameUnchanged(t *testing.T) {
|
||||
ic := interceptorWith(testRule("tool_name", "string", "WebFetch"))
|
||||
if _, ok := ic.Match("WebFetch", []byte(`{"url":"http://x"}`)); !ok {
|
||||
t.Fatal("tool_name rule did not match")
|
||||
}
|
||||
if _, ok := ic.Match("Bash", []byte(`{"command":"WebFetch"}`)); ok {
|
||||
t.Fatal("tool_name rule matched tool input")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
// Package httpidentity defines the explicit identity for target-facing HTTP requests.
|
||||
package httpidentity
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"io"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Keep these 20 standard desktop browser UA samples in sync with the standalone
|
||||
// Python templates. Reviewed 2026-11-11; see HTTP_USER_AGENT.md for versions and
|
||||
// vendor format references. Keep normal reduced platform/version fields and no
|
||||
// tool identifiers, device-specific details or custom suffixes.
|
||||
var userAgents = [...]string{
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0",
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15",
|
||||
}
|
||||
|
||||
// UserAgent returns the configured printable ASCII value, or a random browser
|
||||
// sample when unset, empty or malformed. Call once per logical request and reuse
|
||||
// the selected value for redirects and retries.
|
||||
func UserAgent() string {
|
||||
value := strings.Trim(os.Getenv("ARTEX_HTTP_USER_AGENT"), " ")
|
||||
for _, c := range value {
|
||||
if c < 32 || c > 126 {
|
||||
return randomUserAgent(rand.Reader)
|
||||
}
|
||||
}
|
||||
if value != "" {
|
||||
return value
|
||||
}
|
||||
return randomUserAgent(rand.Reader)
|
||||
}
|
||||
|
||||
func randomUserAgent(source io.Reader) string {
|
||||
index, err := rand.Int(source, big.NewInt(int64(len(userAgents))))
|
||||
if err != nil {
|
||||
return userAgents[0] // Keep a valid nonempty UA if system entropy fails.
|
||||
}
|
||||
return userAgents[index.Int64()]
|
||||
}
|
||||
|
||||
// Apply selects an identity for this request; net/http preserves it on redirects.
|
||||
func Apply(req *http.Request) {
|
||||
req.Header.Set("User-Agent", UserAgent())
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
package httpidentity
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestApplyOnWire(t *testing.T) {
|
||||
for _, tc := range []struct{ name, value, want string }{
|
||||
{"empty", "", ""},
|
||||
{"configured", " Captured-UA/1.0 ", "Captured-UA/1.0"},
|
||||
{"whitespace", " ", ""},
|
||||
{"injection", "ok\r\nX-Injected: yes", ""},
|
||||
{"non-ascii", "客户端", ""},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.value)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got := r.UserAgent()
|
||||
if tc.want == "" && !slices.Contains(userAgents[:], got) {
|
||||
t.Errorf("default User-Agent is not a browser sample: %q", got)
|
||||
} else if tc.want != "" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
if r.Header.Get("X-Injected") != "" {
|
||||
t.Error("header injection")
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
req, err := http.NewRequest(http.MethodGet, server.URL, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
Apply(req)
|
||||
resp, err := server.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRandomUserAgentSelection(t *testing.T) {
|
||||
if len(userAgents) != 20 {
|
||||
t.Fatalf("pool size = %d, want 20", len(userAgents))
|
||||
}
|
||||
seen := make(map[string]bool, len(userAgents))
|
||||
// Exercise every choice deterministically without a statistical/flaky test.
|
||||
for index, want := range userAgents {
|
||||
if seen[want] {
|
||||
t.Errorf("duplicate sample: %q", want)
|
||||
}
|
||||
seen[want] = true
|
||||
if got := randomUserAgent(bytes.NewReader([]byte{byte(index)})); got != want {
|
||||
t.Errorf("choice %d = %q, want %q", index, got, want)
|
||||
}
|
||||
}
|
||||
if got := randomUserAgent(bytes.NewReader(nil)); got != userAgents[0] {
|
||||
t.Errorf("entropy failure fallback = %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,467 @@
|
||||
// Adapted from github.com/Autumn-27/norma/tool/webfetch.go at v0.4.3.
|
||||
// Keep the SDK tool metadata and fetch/render behavior; use ARTEX's explicit
|
||||
// HTTP identity for both the proxy attempt and its direct fallback. This local
|
||||
// adapter is needed until the SDK exposes request headers in actool.WebFetchConfig.
|
||||
package webfetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"crypto/x509"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
"golang.org/x/net/html"
|
||||
)
|
||||
|
||||
// New keeps the SDK's schema, permission checks and tool metadata.
|
||||
func New(cfg actool.WebFetchConfig) actool.CoreTool {
|
||||
base := actool.NewWebFetch(cfg)
|
||||
return actool.Build(actool.Spec{
|
||||
Name: base.Name(), Description: base.Description(), Prompt: base.Prompt(),
|
||||
Schema: base.InputSchema(), Permissions: base.CheckPermissions,
|
||||
ReadOnly: base.IsReadOnly, Concurrent: base.IsConcurrencySafe,
|
||||
Run: webFetchRunner(cfg),
|
||||
})
|
||||
}
|
||||
|
||||
const maxFetchBytes = 2 << 20 // 2 MiB
|
||||
|
||||
// newFetchClient returns an HTTP client honoring the proxy / TLS config.
|
||||
func newFetchClient(cfg actool.WebFetchConfig) *http.Client {
|
||||
tr := &http.Transport{}
|
||||
if cfg.Proxy != "" {
|
||||
if pu, err := url.Parse(cfg.Proxy); err == nil {
|
||||
tr.Proxy = http.ProxyURL(pu)
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case cfg.InsecureTLS:
|
||||
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
|
||||
case cfg.CACert != "":
|
||||
// Trust the proxy's CA in addition to the system roots, so certs it
|
||||
// re-signs verify while every other connection stays fully checked.
|
||||
if pem, err := os.ReadFile(cfg.CACert); err == nil {
|
||||
pool, _ := x509.SystemCertPool()
|
||||
if pool == nil {
|
||||
pool = x509.NewCertPool()
|
||||
}
|
||||
if pool.AppendCertsFromPEM(pem) {
|
||||
tr.TLSClientConfig = &tls.Config{RootCAs: pool}
|
||||
}
|
||||
}
|
||||
}
|
||||
return &http.Client{Transport: tr}
|
||||
}
|
||||
|
||||
// proxyLikelyErr reports whether a fetch error is the kind a recording/MITM proxy
|
||||
// introduces — a closed tunnel (EOF / connection reset), a failed CONNECT, or a
|
||||
// TLS/HTTP2 mismatch — so a direct (no-proxy) retry may succeed. Name-resolution
|
||||
// failures and caller cancellation/timeout are target- or caller-side problems a
|
||||
// direct retry won't fix, so those return false (no retry).
|
||||
func proxyLikelyErr(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
s := strings.ToLower(err.Error())
|
||||
for _, n := range []string{"no such host", "context canceled", "context deadline", "timeout"} {
|
||||
if strings.Contains(s, n) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
for _, p := range []string{
|
||||
"eof", "connection reset", "connection refused", "proxyconnect",
|
||||
"broken pipe", "forcibly closed", "malformed", "http2", "http/2",
|
||||
"tls:", "protocol error",
|
||||
} {
|
||||
if strings.Contains(s, p) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// staticAssetExts are file extensions WebFetch refuses: linked resources (scripts,
|
||||
// styles, source maps, images, fonts, media, wasm) that are not page content.
|
||||
var staticAssetExts = map[string]bool{
|
||||
".js": true, ".mjs": true, ".cjs": true, ".css": true, ".map": true,
|
||||
".png": true, ".jpg": true, ".jpeg": true, ".gif": true, ".svg": true,
|
||||
".webp": true, ".avif": true, ".ico": true, ".bmp": true,
|
||||
".woff": true, ".woff2": true, ".ttf": true, ".otf": true, ".eot": true,
|
||||
".wasm": true, ".mp4": true, ".webm": true, ".mp3": true, ".wav": true, ".ogg": true,
|
||||
}
|
||||
|
||||
// staticAssetExt returns the offending extension (with dot) if rawURL points at a
|
||||
// static asset, or "" otherwise. The query string and fragment are ignored, so
|
||||
// "/app.js?v=3" is still caught.
|
||||
func staticAssetExt(rawURL string) string {
|
||||
path := rawURL
|
||||
if u, err := url.Parse(rawURL); err == nil {
|
||||
path = u.Path
|
||||
}
|
||||
if i := strings.LastIndexByte(path, '/'); i >= 0 {
|
||||
path = path[i+1:] // last segment only
|
||||
}
|
||||
dot := strings.LastIndexByte(path, '.')
|
||||
if dot < 0 {
|
||||
return ""
|
||||
}
|
||||
if ext := strings.ToLower(path[dot:]); staticAssetExts[ext] {
|
||||
return ext
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func webFetchRunner(cfg actool.WebFetchConfig) func(context.Context, json.RawMessage, *actool.ToolContext) (actool.Result, error) {
|
||||
client := newFetchClient(cfg)
|
||||
// direct is the no-proxy fallback used when a proxied fetch fails for a
|
||||
// proxy-caused reason (closed tunnel / failed CONNECT / TLS or HTTP2 mismatch):
|
||||
// it bypasses the configured proxy so a recording/MITM hiccup doesn't fail an
|
||||
// otherwise-reachable URL. nil when no proxy is configured (nothing to bypass).
|
||||
var direct *http.Client
|
||||
if cfg.Proxy != "" {
|
||||
direct = newFetchClient(actool.WebFetchConfig{InsecureTLS: cfg.InsecureTLS})
|
||||
}
|
||||
return func(ctx context.Context, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
|
||||
var in struct {
|
||||
URL string `json:"url"`
|
||||
Extract bool `json:"extract"`
|
||||
}
|
||||
if err := json.Unmarshal(input, &in); err != nil {
|
||||
return actool.Result{}, err
|
||||
}
|
||||
if !strings.HasPrefix(in.URL, "http://") && !strings.HasPrefix(in.URL, "https://") {
|
||||
return actool.Errorf("Error: url must start with http:// or https://"), nil
|
||||
}
|
||||
if ext := staticAssetExt(in.URL); ext != "" {
|
||||
return actool.Errorf("Error: WebFetch does not fetch static assets (" + ext + " file). It reads HTML pages, documentation, and API responses — not linked resources like scripts, styles, images, or fonts. Fetch the page that references this asset instead."), nil
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
req, err := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil)
|
||||
if err != nil {
|
||||
return actool.Errorf("Error: " + err.Error()), nil
|
||||
}
|
||||
userAgent := httpidentity.UserAgent()
|
||||
req.Header.Set("User-Agent", userAgent)
|
||||
resp, err := client.Do(req)
|
||||
// One direct retry when the proxied attempt failed for a proxy-caused reason.
|
||||
// A non-proxy error (bad host, caller timeout/cancel) is returned as-is — a
|
||||
// direct retry wouldn't fix it. The direct response is NOT captured by the proxy.
|
||||
if err != nil && direct != nil && proxyLikelyErr(err) {
|
||||
if r2, e2 := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil); e2 == nil {
|
||||
r2.Header.Set("User-Agent", userAgent)
|
||||
if resp2, err2 := direct.Do(r2); err2 == nil {
|
||||
resp, err = resp2, nil
|
||||
} else {
|
||||
err = err2 // surface the direct failure (more representative of the target)
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return actool.Errorf("Error fetching URL: " + err.Error()), nil
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
body, _ := io.ReadAll(io.LimitReader(resp.Body, maxFetchBytes))
|
||||
|
||||
header := fmt.Sprintf("[%d %s] %s\n", resp.StatusCode, http.StatusText(resp.StatusCode), in.URL)
|
||||
// Cheap fingerprint signal that HTML→Markdown would otherwise hide.
|
||||
if sv := resp.Header.Get("Server"); sv != "" {
|
||||
header += "Server: " + sv + "\n"
|
||||
}
|
||||
if xp := resp.Header.Get("X-Powered-By"); xp != "" {
|
||||
header += "X-Powered-By: " + xp + "\n"
|
||||
}
|
||||
|
||||
var out string
|
||||
if strings.Contains(strings.ToLower(resp.Header.Get("content-type")), "html") {
|
||||
doc, perr := html.Parse(strings.NewReader(string(body)))
|
||||
if perr != nil { // html.Parse is lenient; this is effectively unreachable
|
||||
out = string(body)
|
||||
} else {
|
||||
out = htmlToMarkdown(doc)
|
||||
if in.Extract {
|
||||
if recon := extractRecon(doc); recon != "" {
|
||||
out += "\n\n" + recon
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
out = string(body)
|
||||
}
|
||||
return actool.Text(truncate(header+strings.TrimSpace(out), 50000)), nil
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
reWS = regexp.MustCompile(`[ \t]+`)
|
||||
reLineTrim = regexp.MustCompile(` *\n *`) // strip spaces hugging a newline
|
||||
reBlankLines = regexp.MustCompile(`\n{2,}`) // collapse blank lines → one newline
|
||||
// reEndpoint matches quoted absolute URLs and root-relative paths in inline JS.
|
||||
reEndpoint = regexp.MustCompile("[\"'`](https?://[^\"'`\\s]+|/[A-Za-z0-9._~$&'()*+,;=:@%/?#\\[\\]-]{2,})[\"'`]")
|
||||
)
|
||||
|
||||
// skipMarkdown are elements whose subtree carries no readable prose.
|
||||
var skipMarkdown = map[string]bool{"script": true, "style": true, "noscript": true, "svg": true, "template": true}
|
||||
|
||||
// htmlToMarkdown renders the visible content of a parsed document as Markdown.
|
||||
// Text nodes are already entity-decoded by the parser, so no unescaping is needed.
|
||||
func htmlToMarkdown(doc *html.Node) string {
|
||||
var b strings.Builder
|
||||
renderNode(&b, doc)
|
||||
s := reWS.ReplaceAllString(b.String(), " ")
|
||||
s = reLineTrim.ReplaceAllString(s, "\n") // drop indentation / trailing spaces per line
|
||||
s = reBlankLines.ReplaceAllString(s, "\n") // no blank lines — compact, token-lean output
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
func renderNode(b *strings.Builder, n *html.Node) {
|
||||
switch n.Type {
|
||||
case html.TextNode:
|
||||
b.WriteString(strings.ReplaceAll(n.Data, "\n", " "))
|
||||
return
|
||||
case html.ElementNode:
|
||||
if skipMarkdown[n.Data] {
|
||||
return
|
||||
}
|
||||
default:
|
||||
if n.Type != html.DocumentNode {
|
||||
return
|
||||
}
|
||||
}
|
||||
tag := ""
|
||||
if n.Type == html.ElementNode {
|
||||
tag = n.Data
|
||||
}
|
||||
switch tag {
|
||||
case "h1", "h2", "h3", "h4", "h5", "h6":
|
||||
b.WriteString("\n\n" + strings.Repeat("#", int(tag[1]-'0')) + " ")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "title":
|
||||
b.WriteString("\n\n# ")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "p", "div", "section", "article", "header", "footer", "main", "ul", "ol", "table", "tr", "blockquote":
|
||||
b.WriteString("\n\n")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("\n\n")
|
||||
case "br":
|
||||
b.WriteString("\n")
|
||||
case "hr":
|
||||
b.WriteString("\n\n---\n\n")
|
||||
case "li":
|
||||
b.WriteString("\n- ")
|
||||
renderChildren(b, n)
|
||||
case "a":
|
||||
var inner strings.Builder
|
||||
renderChildren(&inner, n)
|
||||
text := strings.TrimSpace(inner.String())
|
||||
if href := attr(n, "href"); href != "" && text != "" {
|
||||
fmt.Fprintf(b, "[%s](%s)", text, href)
|
||||
} else {
|
||||
b.WriteString(text)
|
||||
}
|
||||
case "img":
|
||||
if src := attr(n, "src"); src != "" {
|
||||
fmt.Fprintf(b, "", attr(n, "alt"), src)
|
||||
}
|
||||
case "strong", "b":
|
||||
b.WriteString("**")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("**")
|
||||
case "em", "i":
|
||||
b.WriteString("*")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("*")
|
||||
case "code":
|
||||
b.WriteString("`")
|
||||
renderChildren(b, n)
|
||||
b.WriteString("`")
|
||||
case "pre":
|
||||
var inner strings.Builder
|
||||
textOnly(&inner, n)
|
||||
b.WriteString("\n\n```\n" + strings.TrimSpace(inner.String()) + "\n```\n\n")
|
||||
default:
|
||||
renderChildren(b, n)
|
||||
}
|
||||
}
|
||||
|
||||
func renderChildren(b *strings.Builder, n *html.Node) {
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
renderNode(b, c)
|
||||
}
|
||||
}
|
||||
|
||||
// extractRecon walks the document for security-relevant signal that Markdown
|
||||
// rendering drops: comments, script sources, inline endpoints, forms, meta, links.
|
||||
func extractRecon(doc *html.Node) string {
|
||||
var comments, scriptSrcs, inlineEndpoints, forms, metas, links []string
|
||||
seenLink := map[string]bool{}
|
||||
|
||||
var walk func(n *html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
switch n.Type {
|
||||
case html.CommentNode:
|
||||
if c := oneLine(n.Data, 300); c != "" {
|
||||
comments = append(comments, c)
|
||||
}
|
||||
case html.ElementNode:
|
||||
switch n.Data {
|
||||
case "script":
|
||||
if src := attr(n, "src"); src != "" {
|
||||
scriptSrcs = append(scriptSrcs, src)
|
||||
} else {
|
||||
var body strings.Builder
|
||||
textOnly(&body, n)
|
||||
for _, m := range reEndpoint.FindAllStringSubmatch(body.String(), 40) {
|
||||
inlineEndpoints = append(inlineEndpoints, m[1])
|
||||
}
|
||||
}
|
||||
case "form":
|
||||
forms = append(forms, describeForm(n))
|
||||
case "meta":
|
||||
name := attr(n, "name")
|
||||
if name == "" {
|
||||
name = attr(n, "property")
|
||||
}
|
||||
if content := attr(n, "content"); name != "" && content != "" {
|
||||
metas = append(metas, name+"="+oneLine(content, 120))
|
||||
}
|
||||
case "a", "link":
|
||||
if h := attr(n, "href"); h != "" && !seenLink[h] {
|
||||
seenLink[h] = true
|
||||
links = append(links, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(doc)
|
||||
|
||||
var b strings.Builder
|
||||
b.WriteString("=== RECON ===")
|
||||
reconSection(&b, "COMMENTS", comments)
|
||||
reconSection(&b, "SCRIPTS (src)", dedup(scriptSrcs))
|
||||
reconSection(&b, "INLINE ENDPOINTS", dedup(inlineEndpoints))
|
||||
reconSection(&b, "FORMS", forms)
|
||||
reconSection(&b, "META", metas)
|
||||
reconSection(&b, "LINKS", capSlice(links, 100))
|
||||
if b.Len() == len("=== RECON ===") {
|
||||
return "" // nothing worth reporting
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// describeForm summarizes a <form> as "METHOD action → name(type[=value]) …".
|
||||
// Hidden inputs keep their value (CSRF tokens, parameter defaults matter in recon).
|
||||
func describeForm(form *html.Node) string {
|
||||
method := strings.ToUpper(attr(form, "method"))
|
||||
if method == "" {
|
||||
method = "GET"
|
||||
}
|
||||
var inputs []string
|
||||
var walk func(n *html.Node)
|
||||
walk = func(n *html.Node) {
|
||||
if n.Type == html.ElementNode {
|
||||
switch n.Data {
|
||||
case "input", "textarea", "select":
|
||||
typ := attr(n, "type")
|
||||
if typ == "" {
|
||||
typ = n.Data
|
||||
}
|
||||
desc := attr(n, "name") + "(" + typ
|
||||
if typ == "hidden" {
|
||||
if v := attr(n, "value"); v != "" {
|
||||
desc += "=" + oneLine(v, 60)
|
||||
}
|
||||
}
|
||||
inputs = append(inputs, desc+")")
|
||||
}
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
walk(c)
|
||||
}
|
||||
}
|
||||
walk(form)
|
||||
return fmt.Sprintf("%s %s → %s", method, attr(form, "action"), strings.Join(inputs, " "))
|
||||
}
|
||||
|
||||
// --- small helpers ---
|
||||
|
||||
func attr(n *html.Node, key string) string {
|
||||
for _, a := range n.Attr {
|
||||
if a.Key == key {
|
||||
return a.Val
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func textOnly(b *strings.Builder, n *html.Node) {
|
||||
if n.Type == html.TextNode {
|
||||
b.WriteString(n.Data)
|
||||
return
|
||||
}
|
||||
for c := n.FirstChild; c != nil; c = c.NextSibling {
|
||||
textOnly(b, c)
|
||||
}
|
||||
}
|
||||
|
||||
func oneLine(s string, max int) string {
|
||||
s = strings.Join(strings.Fields(s), " ")
|
||||
if len(s) > max {
|
||||
return s[:max] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func reconSection(b *strings.Builder, title string, items []string) {
|
||||
if len(items) == 0 {
|
||||
return
|
||||
}
|
||||
b.WriteString("\n\n## " + title + "\n")
|
||||
for _, it := range items {
|
||||
b.WriteString("- " + it + "\n")
|
||||
}
|
||||
}
|
||||
|
||||
func dedup(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := in[:0:0]
|
||||
for _, s := range in {
|
||||
if s == "" || seen[s] {
|
||||
continue
|
||||
}
|
||||
seen[s] = true
|
||||
out = append(out, s)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func capSlice(in []string, n int) []string {
|
||||
if len(in) <= n {
|
||||
return in
|
||||
}
|
||||
return append(in[:n:n], fmt.Sprintf("… (%d more)", len(in)-n))
|
||||
}
|
||||
|
||||
// truncate retains the SDK's head/tail output limit.
|
||||
func truncate(s string, max int) string {
|
||||
if max <= 0 || len(s) <= max {
|
||||
return s
|
||||
}
|
||||
half := max / 2
|
||||
return s[:half] + fmt.Sprintf("\n\n... [%d characters truncated] ...\n\n", len(s)-max) + s[len(s)-half:]
|
||||
}
|
||||
@@ -0,0 +1,175 @@
|
||||
package webfetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"encoding/pem"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
|
||||
func callFetch(t *testing.T, tool actool.CoreTool, target string, extract bool) actool.Result {
|
||||
t.Helper()
|
||||
input, err := json.Marshal(map[string]any{"url": target, "extract": extract})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := tool.Call(context.Background(), input, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func TestIdentityAcrossRoutes(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
for _, route := range []string{"direct", "proxy", "fallback"} {
|
||||
t.Run(route+"/"+ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
var targetCalls, proxyCalls atomic.Int32
|
||||
var mu sync.Mutex
|
||||
var selected string
|
||||
checkUA := func(r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
got := r.UserAgent()
|
||||
if ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
if selected != "" && selected != got {
|
||||
t.Errorf("redirect/retry UA changed from %q to %q", selected, got)
|
||||
}
|
||||
selected = got
|
||||
}
|
||||
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
targetCalls.Add(1)
|
||||
checkUA(r)
|
||||
if r.URL.Path == "/start" {
|
||||
http.Redirect(w, r, "/page", http.StatusFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
fmt.Fprint(w, `<h1>Page</h1><!--keep--><script src="/app.js"></script><form action="/submit"><input name="csrf" type="hidden" value="token"></form>`)
|
||||
}))
|
||||
defer target.Close()
|
||||
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
proxyCalls.Add(1)
|
||||
checkUA(r)
|
||||
if route == "fallback" {
|
||||
conn, _, err := w.(http.Hijacker).Hijack()
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
conn.Close() // Deterministic EOF, causing the SDK-compatible retry.
|
||||
return
|
||||
}
|
||||
req := r.Clone(r.Context())
|
||||
req.RequestURI = ""
|
||||
// A forwarding proxy must preserve absence, like ARTEX's sink.
|
||||
if _, present := req.Header["User-Agent"]; !present {
|
||||
req.Header.Set("User-Agent", "")
|
||||
}
|
||||
resp, err := target.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
for key, values := range resp.Header {
|
||||
w.Header()[key] = values
|
||||
}
|
||||
w.WriteHeader(resp.StatusCode)
|
||||
io.Copy(w, resp.Body)
|
||||
}))
|
||||
defer proxy.Close()
|
||||
cfg := actool.WebFetchConfig{}
|
||||
if route != "direct" {
|
||||
cfg.Proxy = proxy.URL
|
||||
}
|
||||
result := callFetch(t, New(cfg), target.URL+"/start", true)
|
||||
for _, want := range []string{"[200 OK]", "# Page", "=== RECON ===", "keep", "/app.js", "csrf(hidden=token)"} {
|
||||
if result.IsError || !strings.Contains(result.Flatten(), want) {
|
||||
t.Fatalf("missing %q in result: %+v", want, result)
|
||||
}
|
||||
}
|
||||
if targetCalls.Load() != 2 {
|
||||
t.Errorf("target calls = %d, want 2", targetCalls.Load())
|
||||
}
|
||||
if route != "direct" && proxyCalls.Load() == 0 {
|
||||
t.Error("configured proxy was bypassed")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTLSAndStaticAssetGuard(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", "Captured-UA/1.0")
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
if r.UserAgent() != "Captured-UA/1.0" {
|
||||
t.Errorf("User-Agent = %q", r.UserAgent())
|
||||
}
|
||||
fmt.Fprint(w, "verified")
|
||||
}))
|
||||
defer server.Close()
|
||||
certPath := filepath.Join(t.TempDir(), "ca.pem")
|
||||
cert := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})
|
||||
if err := os.WriteFile(certPath, cert, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tool := New(actool.WebFetchConfig{CACert: certPath})
|
||||
if result := callFetch(t, tool, server.URL, false); result.IsError || !strings.Contains(result.Flatten(), "verified") {
|
||||
t.Fatalf("CA trust lost: %+v", result)
|
||||
}
|
||||
if result := callFetch(t, tool, server.URL+"/app.js?v=1", false); !result.IsError {
|
||||
t.Fatalf("static asset guard lost: %+v", result)
|
||||
}
|
||||
if calls.Load() != 1 {
|
||||
t.Errorf("static asset triggered network request: calls = %d", calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentitySelectedForEachCall(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", "Before-Tool-Creation")
|
||||
tool := New(actool.WebFetchConfig{})
|
||||
seen := make(chan string, 2)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
seen <- r.UserAgent()
|
||||
fmt.Fprint(w, "ok")
|
||||
}))
|
||||
defer server.Close()
|
||||
for _, want := range []string{"First-Call", "Second-Call"} {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", want)
|
||||
if result := callFetch(t, tool, server.URL, false); result.IsError {
|
||||
t.Fatal(result.Flatten())
|
||||
}
|
||||
if got := <-seen; got != want {
|
||||
t.Fatalf("identity cached across calls: got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSDKMetadataPreserved(t *testing.T) {
|
||||
base := actool.NewWebFetch(actool.WebFetchConfig{})
|
||||
local := New(actool.WebFetchConfig{})
|
||||
if local.Name() != base.Name() || local.Description() != base.Description() || !reflect.DeepEqual(local.InputSchema(), base.InputSchema()) {
|
||||
t.Fatal("SDK tool contract changed")
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
// Repo 是发布源。写死而不是做成配置项:更新源可配等于给任何能改配置的人一条
|
||||
@@ -125,7 +127,7 @@ func FetchLatest(ctx context.Context, c *http.Client) (*Release, error) {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Accept", "application/vnd.github+json")
|
||||
req.Header.Set("User-Agent", "artex-selfupdate")
|
||||
httpidentity.Apply(req)
|
||||
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
|
||||
@@ -56,3 +56,38 @@ func TestFetchLatestMissingRelease(t *testing.T) {
|
||||
t.Fatalf("unexpected missing-release error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpdateUserAgent(t *testing.T) {
|
||||
for _, tc := range []struct{ env, want string }{
|
||||
{"", ""},
|
||||
{"Captured-UA/1.0", "Captured-UA/1.0"},
|
||||
} {
|
||||
t.Run(tc.env, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
|
||||
calls := 0
|
||||
client := &http.Client{Transport: releaseTransport(func(req *http.Request) (*http.Response, error) {
|
||||
calls++
|
||||
if got := req.UserAgent(); tc.want == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if tc.want != "" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
body := `{"tag_name":"v0.3.17"}`
|
||||
return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(body))}, nil
|
||||
})}
|
||||
if _, err := FetchLatest(context.Background(), client); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, name := range []string{"SHA256SUMS", "artex.zip"} {
|
||||
body, err := get(context.Background(), client, "https://github.com/RuoJi6/ARTEX/releases/download/v0.3.17/"+name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body.Close()
|
||||
}
|
||||
if calls != 3 {
|
||||
t.Fatalf("calls = %d, want 3", calls)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -13,6 +13,8 @@ import (
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
// sumsAsset 是 release.yml 生成的校验和清单,覆盖 Release 里全部 zip。
|
||||
@@ -229,7 +231,7 @@ func get(ctx context.Context, c *http.Client, rawURL string) (io.ReadCloser, err
|
||||
if err := checkURL(req.URL); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-selfupdate")
|
||||
httpidentity.Apply(req)
|
||||
resp, err := c.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -18,6 +18,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
"github.com/Autumn-27/norma/permission"
|
||||
actool "github.com/Autumn-27/norma/tool"
|
||||
)
|
||||
@@ -438,6 +439,7 @@ func (s *Server) runHTTPTool(ctx context.Context, execRaw json.RawMessage, param
|
||||
if err != nil {
|
||||
return actool.Errorf(maskSecrets(err.Error())), nil
|
||||
}
|
||||
httpidentity.Apply(req)
|
||||
for k, v := range spec.Headers {
|
||||
req.Header.Set(k, renderTemplate(v, params, identity))
|
||||
}
|
||||
|
||||
+4
-2
@@ -70,7 +70,9 @@ type TaskDTO struct {
|
||||
CompanyIDs []int64 `json:"company_ids"`
|
||||
CoverageEnabled bool `json:"coverage_enabled"` // 资产覆盖度功能开关(创建时定)
|
||||
ShareGlobalAssets bool `json:"share_global_assets"`
|
||||
WorkerCount int `json:"worker_count"` // 本任务 Worker 数;0=用全局默认
|
||||
WorkerCount int `json:"worker_count"` // 本任务 Worker 数;0=用全局默认
|
||||
WorkerActualCount int `json:"worker_actual_count"` // live pool loops, including draining workers
|
||||
WorkerTargetCount int `json:"worker_target_count"` // desired worker slots
|
||||
|
||||
NoResultSeconds int `json:"no_result_seconds"`
|
||||
NoResultMinFindings int `json:"no_result_min_findings"`
|
||||
@@ -170,7 +172,7 @@ func taskDTO(t *Task, status string) TaskDTO {
|
||||
CompanyIDs: lifecycle.CompanyIDs,
|
||||
CoverageEnabled: t.CoverageEnabled,
|
||||
ShareGlobalAssets: lifecycle.ShareGlobalAssets,
|
||||
WorkerCount: t.WorkerCount,
|
||||
WorkerCount: lifecycle.WorkerCount,
|
||||
|
||||
NoResultSeconds: t.NoResultSeconds,
|
||||
NoResultMinFindings: t.NoResultMinFindings,
|
||||
|
||||
@@ -23,6 +23,7 @@ import (
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
"github.com/Autumn-27/artex/traffic"
|
||||
)
|
||||
|
||||
@@ -305,7 +306,7 @@ func probeOne(ctx context.Context, client *http.Client, target string) (string,
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("User-Agent", "artex-egress-probe/1.0")
|
||||
httpidentity.Apply(req)
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
|
||||
@@ -27,6 +27,27 @@ func fakeIPServer(ip string) *httptest.Server {
|
||||
}))
|
||||
}
|
||||
|
||||
func TestProbeOneUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if ua != "" && got != ua {
|
||||
t.Errorf("User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
fmt.Fprint(w, `{"ip":"203.0.113.7"}`)
|
||||
}))
|
||||
defer server.Close()
|
||||
ip, err := probeOne(context.Background(), server.Client(), server.URL)
|
||||
if err != nil || ip != "203.0.113.7" {
|
||||
t.Fatalf("ip = %q, err = %v", ip, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestProbeEgress 覆盖 probeEgress 的输入/响应形态:成功、非法协议、死代理、坏响应、回落。
|
||||
// 每条用例自带服务端与断言,失败时 t.Run 的名字直接指出是哪一条。
|
||||
func TestProbeEgress(t *testing.T) {
|
||||
|
||||
+42
-47
@@ -111,8 +111,9 @@ type Engine struct {
|
||||
// Every long-lived task goroutine (planner, workers and deadline coordinator)
|
||||
// runs under one task-scoped context. Successful deletion cancels that context,
|
||||
// waits for all goroutines, then releases every task-level Engine reference.
|
||||
runtimeMu sync.Mutex
|
||||
runtimes map[string]*taskRuntime
|
||||
runtimeMu sync.Mutex
|
||||
runtimes map[string]*taskRuntime
|
||||
workerPools sync.Map // taskID -> *taskWorkerPool; each pool serializes claims with resizing
|
||||
|
||||
// per-task execution context: each planner.Plan / worker.Execute runs under it,
|
||||
// so pausing can CANCEL an in-flight run (not just skip the next one). Recreated
|
||||
@@ -150,7 +151,7 @@ type Engine struct {
|
||||
resolveAuthoritative bool
|
||||
workerResolve func(t *Task, slot WorkerSlot) (*agent.Worker, error)
|
||||
slotState sync.Map // task/slot -> slotAvailability
|
||||
slotPlans sync.Map // task -> map[slot name]profile ID; immutable for a run
|
||||
slotPlans sync.Map // task -> immutable snapshot of slot name/profile routing
|
||||
// readiness reports whether a global LLM provider is configured — the signal behind
|
||||
// Ready()/the llm_configured indicator. Wired once at startup; nil → not ready.
|
||||
readiness func() bool
|
||||
@@ -297,6 +298,7 @@ func (e *Engine) StopTask(taskID string) {
|
||||
e.inflight.Delete(taskID)
|
||||
e.coordStarted.Delete(taskID)
|
||||
e.slotPlans.Delete(taskID)
|
||||
e.workerPools.Delete(taskID)
|
||||
e.slotState.Range(func(key, _ any) bool {
|
||||
keyText, ok := key.(string)
|
||||
if ok && strings.HasPrefix(keyText, taskID+"/") {
|
||||
@@ -406,10 +408,14 @@ func (e *Engine) ActiveLLMCalls(taskID string) int64 {
|
||||
func (e *Engine) touch(taskID string) { e.lastAct.Store(taskID, time.Now().Unix()) }
|
||||
|
||||
func NewEngine(m *Manager) *Engine {
|
||||
return &Engine{m: m, debounce: 800 * time.Millisecond, bc: NewBroadcaster(),
|
||||
e := &Engine{m: m, debounce: 800 * time.Millisecond, bc: NewBroadcaster(),
|
||||
execCancel: map[string]context.CancelCauseFunc{}, execCtx: map[string]context.Context{},
|
||||
work: map[int64]*workExecution{}, steerBox: map[int64][]string{},
|
||||
runtimes: map[string]*taskRuntime{}}
|
||||
if m != nil {
|
||||
m.SetWorkerConfigChangedCallback(e.ReconcileDefaultWorkers)
|
||||
}
|
||||
return e
|
||||
}
|
||||
|
||||
// registerWork records the cancel for the work currently running intentID.
|
||||
@@ -796,11 +802,6 @@ func (e *Engine) ReadyFor(t *Task) bool {
|
||||
// but no-op until an LLM is configured (so a task created while idle picks up
|
||||
// automatically once LLM is set from the UI).
|
||||
func (e *Engine) Run(ctx context.Context, t *Task) {
|
||||
slots, err := workerSlotsForTask(t, e.m.WorkerPools(), e.m.Workers())
|
||||
if err != nil {
|
||||
log.Printf("[engine] task %s worker pool invalid, using legacy workers: %v", t.ID, err)
|
||||
slots, _ = ExpandWorkerSlots(nil, workerCountForTask(t, e.m.Workers()))
|
||||
}
|
||||
e.deleteMu.RLock()
|
||||
if e.IsDeleting(t.ID) {
|
||||
e.deleteMu.RUnlock()
|
||||
@@ -811,19 +812,13 @@ func (e *Engine) Run(ctx context.Context, t *Task) {
|
||||
t.Notify() // already running — just nudge a planning round
|
||||
return
|
||||
}
|
||||
rt := e.registerTaskRoutines(ctx, t.ID, 1+len(slots))
|
||||
plan := make(map[string]int64, len(slots))
|
||||
for _, slot := range slots {
|
||||
plan[slot.Name] = slot.ProfileID
|
||||
}
|
||||
e.slotPlans.Store(t.ID, plan)
|
||||
rt := e.registerTaskRoutines(ctx, t.ID, 1)
|
||||
pool := &taskWorkerPool{runtime: rt, task: t, loops: make(map[string]*workerSlotRuntime)}
|
||||
e.workerPools.Store(t.ID, pool)
|
||||
e.deleteMu.RUnlock()
|
||||
e.touch(t.ID)
|
||||
runTaskRoutine(rt, func(loopCtx context.Context) { e.plannerLoop(loopCtx, t) })
|
||||
for _, slot := range slots {
|
||||
slot := slot
|
||||
runTaskRoutine(rt, func(loopCtx context.Context) { e.workerLoop(loopCtx, t, slot) })
|
||||
}
|
||||
e.ReconcileWorkers(t)
|
||||
e.startDeadlineCoordinator(ctx, t) // 任务级超时定时器(仅 timeout>0;去重)
|
||||
// 仅在「完全没有活动意图(open+running)」时才 kick 首轮规划。带种子意图的任务:种子已
|
||||
// 是 open,或已被上面刚起的 worker 抢先 claim 成 running——两种都算「有活干」,一律跳过
|
||||
@@ -838,20 +833,20 @@ func (e *Engine) Run(ctx context.Context, t *Task) {
|
||||
// workerSlotsForTask preserves an explicit task concurrency override. Tasks
|
||||
// using the global default inherit its model pools and their pinned profiles.
|
||||
func workerSlotsForTask(t *Task, pools []WorkerPool, global int) ([]WorkerSlot, error) {
|
||||
if t != nil && t.WorkerCount > 0 {
|
||||
if t != nil && t.lifecycleSnapshot().WorkerCount > 0 {
|
||||
return ExpandWorkerSlots(nil, workerCountForTask(t, global))
|
||||
}
|
||||
return ExpandWorkerSlots(pools, global)
|
||||
}
|
||||
|
||||
func workerCountForTask(t *Task, global int) int {
|
||||
if t != nil && t.WorkerCount > 0 {
|
||||
// Direct database edits can bypass validation; cap those values so the
|
||||
// engine never starts more than the supported per-task worker count.
|
||||
if t.WorkerCount > db.MaxTaskWorkerCount {
|
||||
return db.MaxTaskWorkerCount
|
||||
if t != nil {
|
||||
if count := t.lifecycleSnapshot().WorkerCount; count > 0 {
|
||||
if count > db.MaxTaskWorkerCount {
|
||||
return db.MaxTaskWorkerCount
|
||||
}
|
||||
return count
|
||||
}
|
||||
return t.WorkerCount
|
||||
}
|
||||
return global
|
||||
}
|
||||
@@ -1004,11 +999,12 @@ func (e *Engine) plannerLoop(ctx context.Context, t *Task) {
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
|
||||
func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot, control *workerSlotRuntime) {
|
||||
name := slot.Name
|
||||
idleCtx := control.idleCtx
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-idleCtx.Done():
|
||||
return
|
||||
default:
|
||||
}
|
||||
@@ -1031,13 +1027,13 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
|
||||
_, worker = e.snapshotFor(t)
|
||||
}
|
||||
if worker == nil {
|
||||
if sleepCtx(ctx, 1500*time.Millisecond) {
|
||||
if sleepCtx(idleCtx, 1500*time.Millisecond) {
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
if e.IsPaused(t.ID) {
|
||||
if sleepCtx(ctx, 1000*time.Millisecond) {
|
||||
if sleepCtx(idleCtx, 1000*time.Millisecond) {
|
||||
return
|
||||
}
|
||||
continue // user-paused: don't claim/execute intents
|
||||
@@ -1046,13 +1042,13 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
|
||||
return
|
||||
}
|
||||
if e.isSettling(t.ID) {
|
||||
if sleepCtx(ctx, 1000*time.Millisecond) {
|
||||
if sleepCtx(idleCtx, 1000*time.Millisecond) {
|
||||
return
|
||||
}
|
||||
continue // 任务超时收尾中:不再领新意图(在跑的自行收尾,协调器等其 drain)
|
||||
}
|
||||
if isTerminalStatus(e.m.TaskStatus(t.ID)) {
|
||||
if sleepCtx(ctx, 1000*time.Millisecond) {
|
||||
if sleepCtx(idleCtx, 1000*time.Millisecond) {
|
||||
return
|
||||
}
|
||||
continue // 任务已终态(done/failed/timeout):停止领取遗留意图,别在完成后空跑 frontier
|
||||
@@ -1060,28 +1056,27 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
|
||||
if !e.beginTaskOperation(t.ID) {
|
||||
return
|
||||
}
|
||||
claimed := e.runWorkerStep(ctx, t, name, worker)
|
||||
// A resize and the next claim share the pool lock. Once retirement is
|
||||
// requested this loop cannot claim again; execution keeps the task context.
|
||||
control.pool.mu.Lock()
|
||||
var intent *db.Node
|
||||
if idleCtx.Err() == nil {
|
||||
intent = e.claimNext(t, name)
|
||||
}
|
||||
control.pool.mu.Unlock()
|
||||
if intent != nil {
|
||||
log.Printf("[worker %s] task %s 领取意图 #%d", name, t.ID, intent.ID)
|
||||
e.runIntent(ctx, t, name, worker, intent, "", "")
|
||||
}
|
||||
e.decInflight(t.ID)
|
||||
if !claimed && sleepCtx(ctx, 800*time.Millisecond) {
|
||||
if intent == nil && sleepCtx(idleCtx, 800*time.Millisecond) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// runWorkerStep claims one intent from the frontier and fully settles it via
|
||||
// runIntent. Returns false when nothing was claimable. The pool worker loop is its
|
||||
// only caller.
|
||||
func (e *Engine) runWorkerStep(ctx context.Context, t *Task, name string, worker *agent.Worker) bool {
|
||||
intent := e.claimNext(t, name)
|
||||
if intent == nil {
|
||||
return false
|
||||
}
|
||||
log.Printf("[worker %s] task %s 领取意图 #%d", name, t.ID, intent.ID)
|
||||
return e.runIntent(ctx, t, name, worker, intent, "", "")
|
||||
}
|
||||
|
||||
// runIntent executes and fully settles one already-claimed (state=running) intent.
|
||||
// Both the pool worker loop (via runWorkerStep) and the human-message handler (via
|
||||
// Both the pool worker loop and the human-message handler (via
|
||||
// runDetachedIntent, a dedicated goroutine outside the worker pool) call it, so the
|
||||
// execute/retry/state-write logic lives in exactly one place. A non-empty message
|
||||
// is injected as this turn's input through ExecuteWithMessage; requestID keys the
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// taskWorkerPool owns the desired plan and all live loops, including retirees.
|
||||
// A slot name is never reused until its previous loop has fully settled its work.
|
||||
type taskWorkerPool struct {
|
||||
mu sync.Mutex
|
||||
runtime *taskRuntime
|
||||
task *Task
|
||||
loops map[string]*workerSlotRuntime
|
||||
desired map[string]WorkerSlot
|
||||
}
|
||||
|
||||
type workerSlotRuntime struct {
|
||||
pool *taskWorkerPool
|
||||
slot WorkerSlot
|
||||
idleCtx context.Context
|
||||
retire context.CancelFunc // stops claiming/sleeping, never cancels Execute
|
||||
}
|
||||
|
||||
func (e *Engine) desiredWorkerSlots(t *Task) []WorkerSlot {
|
||||
slots, err := workerSlotsForTask(t, e.m.WorkerPools(), e.m.Workers())
|
||||
if err != nil {
|
||||
log.Printf("[engine] task %s worker pool invalid, using legacy workers: %v", t.ID, err)
|
||||
slots, _ = ExpandWorkerSlots(nil, workerCountForTask(t, e.m.Workers()))
|
||||
}
|
||||
return slots
|
||||
}
|
||||
|
||||
// ReconcileWorkers applies the latest persisted settings without restarting a task.
|
||||
// Claims and retirement share the pool lock, so no retired slot takes new work.
|
||||
func (e *Engine) ReconcileWorkers(t *Task) {
|
||||
e.deleteMu.RLock()
|
||||
defer e.deleteMu.RUnlock()
|
||||
if e.IsDeleting(t.ID) {
|
||||
return
|
||||
}
|
||||
value, ok := e.workerPools.Load(t.ID)
|
||||
if !ok {
|
||||
return // queued/not started: Run will read the latest configuration
|
||||
}
|
||||
pool := value.(*taskWorkerPool)
|
||||
pool.mu.Lock()
|
||||
defer pool.mu.Unlock()
|
||||
if pool.runtime.ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
slots := e.desiredWorkerSlots(t)
|
||||
pool.desired = make(map[string]WorkerSlot, len(slots))
|
||||
for _, slot := range slots {
|
||||
pool.desired[slot.Name] = slot
|
||||
}
|
||||
for name, loop := range pool.loops {
|
||||
if slot, keep := pool.desired[name]; !keep || slot != loop.slot {
|
||||
loop.retire()
|
||||
}
|
||||
}
|
||||
for _, slot := range slots {
|
||||
if pool.loops[slot.Name] == nil {
|
||||
e.startWorkerSlotLocked(pool, slot)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ReconcileDefaultWorkers is called after a global workers/pool save. Explicit
|
||||
// task overrides keep both their concurrency and legacy task-router profiles.
|
||||
func (e *Engine) ReconcileDefaultWorkers() {
|
||||
e.workerPools.Range(func(_, value any) bool {
|
||||
t := value.(*taskWorkerPool).task
|
||||
if t.lifecycleSnapshot().WorkerCount == 0 {
|
||||
e.ReconcileWorkers(t)
|
||||
}
|
||||
return true
|
||||
})
|
||||
}
|
||||
|
||||
// WorkerCounts distinguishes live pool loops (including draining workers) from
|
||||
// the effective configured target. Neither count is the number of busy intents.
|
||||
func (e *Engine) WorkerCounts(t *Task) (actual, target int) {
|
||||
if value, ok := e.workerPools.Load(t.ID); ok {
|
||||
pool := value.(*taskWorkerPool)
|
||||
pool.mu.Lock()
|
||||
defer pool.mu.Unlock()
|
||||
return len(pool.loops), len(pool.desired)
|
||||
}
|
||||
return 0, len(e.desiredWorkerSlots(t))
|
||||
}
|
||||
|
||||
// Caller holds deleteMu.RLock and pool.mu. The delete barrier protects wg.Add
|
||||
// from StopTask.Wait, including replacements started as a retiree exits.
|
||||
func (e *Engine) startWorkerSlotLocked(pool *taskWorkerPool, slot WorkerSlot) {
|
||||
idleCtx, retire := context.WithCancel(pool.runtime.ctx)
|
||||
loop := &workerSlotRuntime{pool: pool, slot: slot, idleCtx: idleCtx, retire: retire}
|
||||
pool.loops[slot.Name] = loop
|
||||
// Publish a fresh immutable routing snapshot. Retired slot routes remain
|
||||
// available for human continuation of paused intents until the name is reused.
|
||||
plan := make(map[string]int64)
|
||||
if old, ok := e.slotPlans.Load(pool.task.ID); ok {
|
||||
for name, profile := range old.(map[string]int64) {
|
||||
plan[name] = profile
|
||||
}
|
||||
}
|
||||
plan[slot.Name] = slot.ProfileID
|
||||
e.slotPlans.Store(pool.task.ID, plan)
|
||||
pool.runtime.wg.Add(1)
|
||||
runTaskRoutine(pool.runtime, func(ctx context.Context) {
|
||||
defer e.finishWorkerSlot(loop)
|
||||
e.workerLoop(ctx, pool.task, slot, loop)
|
||||
})
|
||||
}
|
||||
|
||||
func (e *Engine) finishWorkerSlot(loop *workerSlotRuntime) {
|
||||
loop.retire()
|
||||
e.deleteMu.RLock()
|
||||
defer e.deleteMu.RUnlock()
|
||||
pool := loop.pool
|
||||
pool.mu.Lock()
|
||||
defer pool.mu.Unlock()
|
||||
delete(pool.loops, loop.slot.Name)
|
||||
e.slotState.Delete(pool.task.ID + "/" + loop.slot.Name)
|
||||
if !e.IsDeleting(pool.task.ID) && pool.runtime.ctx.Err() == nil {
|
||||
if slot, needed := pool.desired[loop.slot.Name]; needed {
|
||||
e.startWorkerSlotLocked(pool, slot)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/agent"
|
||||
"github.com/Autumn-27/artex/db"
|
||||
"github.com/Autumn-27/norma/llm"
|
||||
)
|
||||
|
||||
type resizingCall struct {
|
||||
ctx context.Context
|
||||
release chan struct{}
|
||||
}
|
||||
|
||||
type resizingProvider struct{ calls chan resizingCall }
|
||||
|
||||
func (p *resizingProvider) Complete(ctx context.Context, _ llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
|
||||
call := resizingCall{ctx: ctx, release: make(chan struct{})}
|
||||
select {
|
||||
case p.calls <- call:
|
||||
case <-ctx.Done():
|
||||
return llm.Message{}, "", llm.Usage{}, ctx.Err()
|
||||
}
|
||||
select {
|
||||
case <-call.release:
|
||||
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("fixture completed")}}, "end_turn", llm.Usage{}, nil
|
||||
case <-ctx.Done():
|
||||
return llm.Message{}, "", llm.Usage{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (*resizingProvider) Stream(context.Context, llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
|
||||
panic("resizing fixture uses non-streaming calls")
|
||||
}
|
||||
|
||||
func waitWorkerCondition(t *testing.T, condition func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if condition() {
|
||||
return
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("timed out waiting for worker state")
|
||||
}
|
||||
|
||||
// Exercises the real worker claim/Execute/settlement path. Both slots remain
|
||||
// blocked in provider calls while shrinking; only the retained slot may claim
|
||||
// another intent, and re-expansion must reuse a name without overlapping loops.
|
||||
func TestWorkerHotResizeDrainsBeforeReusingSlot(t *testing.T) {
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable: %v", err)
|
||||
}
|
||||
defer m.Close()
|
||||
task, err := m.CreateTask("resize workers", "fixture", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.DeleteTask(task.ID, DeleteTaskOptions{})
|
||||
count := 2
|
||||
if _, err := m.UpdateTaskMetadata(task.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for range 2 {
|
||||
if _, err := task.Store.AddIntent(map[string]any{"text": "fixture"}, 0, nil, "test"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
e := NewEngine(m)
|
||||
defer e.StopTask(task.ID)
|
||||
var mu sync.Mutex
|
||||
providers := make(map[string]*resizingProvider)
|
||||
e.SetAuthoritativeAgentResolver(func(*Task) (*agent.Planner, *agent.Worker) { return nil, nil })
|
||||
// Use a pinned pool so each slot's blocked call can be released independently.
|
||||
profileID, err := m.pg.SaveProfile(&db.LLMProfile{Name: "resize-fixture", Format: "openai", Model: "fixture"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.pg.DeleteProfile(profileID)
|
||||
oldPools, _, _ := m.pg.GetSetting(settingWorkerPools)
|
||||
oldWorkers, _, _ := m.pg.GetSetting(settingWorkers)
|
||||
defer m.pg.SetSetting(settingWorkerPools, oldPools)
|
||||
defer m.pg.SetSetting(settingWorkers, oldWorkers)
|
||||
count = 0
|
||||
if _, err := m.UpdateTaskMetadata(task.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
workDir := t.TempDir()
|
||||
e.SetWorkerResolver(func(_ *Task, slot WorkerSlot) (*agent.Worker, error) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
p := providers[slot.Name]
|
||||
if p == nil {
|
||||
p = &resizingProvider{calls: make(chan resizingCall, 4)}
|
||||
providers[slot.Name] = p
|
||||
}
|
||||
w := agent.NewWorker(p, "fixture", workDir, nil, 0, 0)
|
||||
w.SetNonStreaming(func() bool { return true })
|
||||
return w, nil
|
||||
})
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.Run(t.Context(), task)
|
||||
getCall := func(name string) resizingCall {
|
||||
t.Helper()
|
||||
var p *resizingProvider
|
||||
waitWorkerCondition(t, func() bool {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
p = providers[name]
|
||||
return p != nil
|
||||
})
|
||||
select {
|
||||
case call := <-p.calls:
|
||||
return call
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatalf("%s did not execute an intent", name)
|
||||
return resizingCall{}
|
||||
}
|
||||
}
|
||||
first, second := getCall("work#1"), getCall("work#2")
|
||||
thirdID, err := task.Store.AddIntent(map[string]any{"text": "next"}, 0, nil, "test")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 1}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if actual, target := e.WorkerCounts(task); actual != 2 || target != 1 {
|
||||
t.Fatalf("draining counts=%d/%d, want 2/1", actual, target)
|
||||
}
|
||||
if second.ctx.Err() != nil || first.ctx.Err() != nil {
|
||||
t.Fatal("resize cancelled an executing intent")
|
||||
}
|
||||
close(second.release)
|
||||
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(task); return actual == 1 && target == 1 })
|
||||
third, err := task.Store.GetNode(thirdID)
|
||||
if err != nil || third.State != "open" {
|
||||
t.Fatalf("retired slot claimed queued intent: node=%+v err=%v", third, err)
|
||||
}
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
thirdCall := getCall("work#2")
|
||||
third, err = task.Store.GetNode(thirdID)
|
||||
if err != nil || third.State != "running" || third.Owner != "work#2" {
|
||||
t.Fatalf("expanded slot did not claim queued intent: node=%+v err=%v", third, err)
|
||||
}
|
||||
// A rapid shrink/grow must not start a duplicate work#2 while it drains.
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 1}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if actual, target := e.WorkerCounts(task); actual != 2 || target != 2 || thirdCall.ctx.Err() != nil {
|
||||
t.Fatalf("rapid resize interrupted work or duplicated slot: %d/%d err=%v", actual, target, thirdCall.ctx.Err())
|
||||
}
|
||||
close(thirdCall.release)
|
||||
close(first.release)
|
||||
waitWorkerCondition(t, func() bool {
|
||||
node, _ := task.Store.GetNode(thirdID)
|
||||
return node != nil && node.State != "running" && e.inflightCount(task.ID) == 0
|
||||
})
|
||||
}
|
||||
|
||||
func TestWorkerHotResizeTaskOverrideAndGlobalDefault(t *testing.T) {
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable: %v", err)
|
||||
}
|
||||
defer m.Close()
|
||||
oldPools, _, _ := m.pg.GetSetting(settingWorkerPools)
|
||||
oldWorkers, _, _ := m.pg.GetSetting(settingWorkers)
|
||||
defer m.pg.SetSetting(settingWorkerPools, oldPools)
|
||||
defer m.pg.SetSetting(settingWorkers, oldWorkers)
|
||||
if err := m.SetWorkerPools(nil, 2); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e := NewEngine(m)
|
||||
global, err := m.CreateTask("global resize fixture", "fixture", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.DeleteTask(global.ID, DeleteTaskOptions{})
|
||||
defer e.StopTask(global.ID)
|
||||
override, err := m.CreateTask("explicit resize fixture", "fixture", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.DeleteTask(override.ID, DeleteTaskOptions{})
|
||||
defer e.StopTask(override.ID)
|
||||
count := 3
|
||||
if _, err := m.UpdateTaskMetadata(override.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.Run(t.Context(), global)
|
||||
e.Run(t.Context(), override)
|
||||
if err := m.SetWorkers(4); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if actual, target := e.WorkerCounts(global); actual != 4 || target != 4 {
|
||||
t.Fatalf("global default did not expand: %d/%d", actual, target)
|
||||
}
|
||||
if actual, target := e.WorkerCounts(override); actual != 3 || target != 3 {
|
||||
t.Fatalf("global setting changed explicit override: %d/%d", actual, target)
|
||||
}
|
||||
if err := m.SetWorkers(1); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(global); return actual == 1 && target == 1 })
|
||||
if actual, target := e.WorkerCounts(override); actual != 3 || target != 3 {
|
||||
t.Fatalf("global shrink changed explicit override: %d/%d", actual, target)
|
||||
}
|
||||
// Exercise the live PATCH path, then restore global inheritance through PATCH.
|
||||
s := &Server{m: m, engine: e, ctx: t.Context()}
|
||||
patch := func(count int) {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(http.MethodPatch, "/api/tasks/"+override.ID, strings.NewReader(fmt.Sprintf(`{"worker_count":%d}`, count)))
|
||||
req.SetPathValue("id", override.ID)
|
||||
rec := httptest.NewRecorder()
|
||||
s.updateTaskMetadata(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("live patch=%d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
patch(5)
|
||||
if actual, target := e.WorkerCounts(override); actual != 5 || target != 5 {
|
||||
t.Fatalf("task PATCH did not expand: %d/%d", actual, target)
|
||||
}
|
||||
patch(0)
|
||||
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(override); return actual == 1 && target == 1 })
|
||||
persisted, err := m.pg.GetTask(taskIDInt(t, override.ID))
|
||||
if err != nil || persisted.WorkerCount != 0 {
|
||||
t.Fatalf("inheritance not persisted: task=%+v err=%v", persisted, err)
|
||||
}
|
||||
// Reconstructing a task after restart must use the persisted override policy.
|
||||
restored := taskFromPG(persisted, override.Store, m.interceptor)
|
||||
if slots, err := workerSlotsForTask(restored, m.WorkerPools(), m.Workers()); err != nil || len(slots) != 1 {
|
||||
t.Fatalf("restored task lost concurrency policy: slots=%v err=%v", slots, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTargetHTTPIdentityOverrides(t *testing.T) {
|
||||
t.Setenv(repeaterAllowLocalEnv, "1")
|
||||
for _, route := range []string{"custom", "repeater"} {
|
||||
for _, tc := range []struct {
|
||||
name, env, want string
|
||||
headers map[string]string
|
||||
}{
|
||||
{name: "default"},
|
||||
{name: "environment", env: "Captured-UA/1.0", want: "Captured-UA/1.0"},
|
||||
{name: "override", env: "Environment-UA/1.0", want: "Explicit-UA/1.0", headers: map[string]string{"user-agent": "Explicit-UA/1.0"}},
|
||||
{name: "explicit-empty", env: "Environment-UA/1.0", headers: map[string]string{"User-Agent": ""}},
|
||||
} {
|
||||
t.Run(route+"/"+tc.name, func(t *testing.T) {
|
||||
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); tc.name == "default" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
|
||||
t.Errorf("random UA missing: %q", got)
|
||||
} else if tc.name != "default" && got != tc.want {
|
||||
t.Errorf("User-Agent = %q, want %q", got, tc.want)
|
||||
}
|
||||
if _, present := r.Header["User-Agent"]; tc.name == "explicit-empty" && present {
|
||||
t.Error("empty User-Agent present on wire")
|
||||
}
|
||||
w.Write([]byte("ok"))
|
||||
}))
|
||||
defer server.Close()
|
||||
payload := map[string]any{"url": server.URL, "headers": tc.headers, "use_recording_proxy": false}
|
||||
if route == "custom" {
|
||||
raw, _ := json.Marshal(payload)
|
||||
result, err := (&Server{}).runHTTPTool(context.Background(), raw, nil, nil)
|
||||
if err != nil || result.IsError {
|
||||
t.Fatalf("custom tool: %v %+v", err, result)
|
||||
}
|
||||
} else {
|
||||
status, result, body := repeaterTestSend(t, &Server{}, payload)
|
||||
if status != http.StatusOK || result.Status != http.StatusOK {
|
||||
t.Fatalf("repeater: %d %s", status, body)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
+55
-19
@@ -99,6 +99,7 @@ type Task struct {
|
||||
// directly after the task has been published by Manager.
|
||||
type taskLifecycleState struct {
|
||||
ShareGlobalAssets bool
|
||||
WorkerCount int
|
||||
Name string
|
||||
Remark string
|
||||
PinnedAt int64
|
||||
@@ -140,6 +141,7 @@ func (t *Task) lifecycleSnapshot() taskLifecycleState {
|
||||
func (t *Task) lifecycleSnapshotLocked() taskLifecycleState {
|
||||
return taskLifecycleState{
|
||||
ShareGlobalAssets: t.ShareGlobalAssets,
|
||||
WorkerCount: t.WorkerCount,
|
||||
Name: t.Name,
|
||||
Remark: t.Remark,
|
||||
PinnedAt: t.PinnedAt,
|
||||
@@ -171,6 +173,7 @@ func (t *Task) updateLifecycle(update func(*taskLifecycleState)) {
|
||||
state := t.lifecycleSnapshotLocked()
|
||||
update(&state)
|
||||
t.ShareGlobalAssets = state.ShareGlobalAssets
|
||||
t.WorkerCount = state.WorkerCount
|
||||
t.Name = state.Name
|
||||
t.Remark = state.Remark
|
||||
t.PinnedAt = state.PinnedAt
|
||||
@@ -282,13 +285,14 @@ type Manager struct {
|
||||
// taskStateMu preserves commit order between PostgreSQL lifecycle writes and
|
||||
// their in-memory mirrors. lifecycleMu makes snapshots race-free, but without
|
||||
// this outer write lock an older request could commit first and publish last.
|
||||
taskStateMu sync.Mutex
|
||||
controlMu *sync.Mutex // Server's admission/task-control mutex, when attached.
|
||||
mu sync.RWMutex
|
||||
tasks map[string]*Task
|
||||
active string
|
||||
trafficOn bool // 流量捕获开关(默认关;settings.traffic_capture)
|
||||
llmRecOn bool // LLM 录制开关(默认关;settings.llm_record)
|
||||
taskStateMu sync.Mutex
|
||||
controlMu *sync.Mutex // Server's admission/task-control mutex, when attached.
|
||||
mu sync.RWMutex
|
||||
tasks map[string]*Task
|
||||
workerConfigChanged func()
|
||||
active string
|
||||
trafficOn bool // 流量捕获开关(默认关;settings.traffic_capture)
|
||||
llmRecOn bool // LLM 录制开关(默认关;settings.llm_record)
|
||||
// 联网搜索开关与来源(默认关;settings.web_search_*)。brave-free 需要 braveKey;tavily 需要 tavilyKey。
|
||||
// webSearchProxy 是独立出口代理(http/https/socks5),与记录流量的 MITM 代理无关。
|
||||
webSearchOn bool
|
||||
@@ -310,6 +314,23 @@ type Manager struct {
|
||||
// participate in the Server's admission/task-control critical section.
|
||||
func (m *Manager) SetTaskControlMutex(mu *sync.Mutex) { m.controlMu = mu }
|
||||
|
||||
// SetWorkerConfigChangedCallback installs the engine's live worker reconciler.
|
||||
// Successful worker-setting writes invoke it after releasing Manager locks.
|
||||
func (m *Manager) SetWorkerConfigChangedCallback(callback func()) {
|
||||
m.mu.Lock()
|
||||
m.workerConfigChanged = callback
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
func (m *Manager) notifyWorkerConfigChanged() {
|
||||
m.mu.RLock()
|
||||
callback := m.workerConfigChanged
|
||||
m.mu.RUnlock()
|
||||
if callback != nil {
|
||||
callback()
|
||||
}
|
||||
}
|
||||
|
||||
// Settings keys the UI toggles at runtime.
|
||||
const (
|
||||
settingTrafficCapture = "traffic_capture"
|
||||
@@ -433,6 +454,9 @@ func ExpandWorkerSlots(pools []WorkerPool, legacy int) ([]WorkerSlot, error) {
|
||||
if legacy <= 0 {
|
||||
legacy = defaultWorkers
|
||||
}
|
||||
if legacy > pgdb.MaxTaskWorkerCount {
|
||||
legacy = pgdb.MaxTaskWorkerCount
|
||||
}
|
||||
slots := make([]WorkerSlot, legacy)
|
||||
for i := range slots {
|
||||
slots[i] = WorkerSlot{Name: fmt.Sprintf("work#%d", i+1)}
|
||||
@@ -478,8 +502,8 @@ func (m *Manager) SetConcurrency(enabled bool, limit int) error {
|
||||
return m.pg.SetSetting(settingConcurrencyOn, strconv.FormatBool(enabled))
|
||||
}
|
||||
|
||||
// Workers returns the configured concurrent work-agent count (default 3). Read
|
||||
// per-task at engine.Run, so a change applies to tasks started afterwards.
|
||||
// Workers returns the configured concurrent work-agent count (default 3),
|
||||
// capped at the supported maximum even for settings saved by older versions.
|
||||
func (m *Manager) Workers() int {
|
||||
v, ok, err := m.pg.GetSetting(settingWorkers)
|
||||
if err != nil || !ok {
|
||||
@@ -489,6 +513,9 @@ func (m *Manager) Workers() int {
|
||||
if err != nil || n <= 0 {
|
||||
return defaultWorkers
|
||||
}
|
||||
if n > pgdb.MaxTaskWorkerCount {
|
||||
return pgdb.MaxTaskWorkerCount
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
@@ -542,23 +569,31 @@ func (m *Manager) SetWorkerPools(pools []WorkerPool, legacyWorkers ...int) error
|
||||
workerCount += pool.Workers
|
||||
}
|
||||
} else if len(legacyWorkers) > 0 {
|
||||
if legacyWorkers[0] <= 0 {
|
||||
return fmt.Errorf("workers 必须 >0")
|
||||
if legacyWorkers[0] <= 0 || legacyWorkers[0] > pgdb.MaxTaskWorkerCount {
|
||||
return fmt.Errorf("workers 必须在 1 到 %d 之间", pgdb.MaxTaskWorkerCount)
|
||||
}
|
||||
workerCount = legacyWorkers[0]
|
||||
}
|
||||
return m.pg.SetSettingsAtomic(context.Background(), map[string]string{
|
||||
if err := m.pg.SetSettingsAtomic(context.Background(), map[string]string{
|
||||
settingWorkerPools: string(b),
|
||||
settingWorkers: strconv.Itoa(workerCount),
|
||||
})
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
m.notifyWorkerConfigChanged()
|
||||
return nil
|
||||
}
|
||||
|
||||
// SetWorkers persists the concurrent work-agent count. Values <=0 are rejected.
|
||||
// SetWorkers persists the concurrent work-agent count and reconciles live tasks.
|
||||
func (m *Manager) SetWorkers(n int) error {
|
||||
if n <= 0 {
|
||||
return fmt.Errorf("workers 必须 >0")
|
||||
if n <= 0 || n > pgdb.MaxTaskWorkerCount {
|
||||
return fmt.Errorf("workers 必须在 1 到 %d 之间", pgdb.MaxTaskWorkerCount)
|
||||
}
|
||||
return m.pg.SetSetting(settingWorkers, strconv.Itoa(n))
|
||||
if err := m.pg.SetSetting(settingWorkers, strconv.Itoa(n)); err != nil {
|
||||
return err
|
||||
}
|
||||
m.notifyWorkerConfigChanged()
|
||||
return nil
|
||||
}
|
||||
|
||||
// Enrich returns the asset auto-completion engine (may be nil if init failed).
|
||||
@@ -1179,8 +1214,8 @@ func taskFromPG(pt *pgdb.Task, store *pgdb.ExplorationStore, ic *intercept.Inter
|
||||
}
|
||||
}
|
||||
|
||||
// UpdateTaskMetadata changes task metadata and asset visibility without interrupting any
|
||||
// planner, main-agent, or worker call.
|
||||
// UpdateTaskMetadata changes task metadata, asset visibility, and worker policy
|
||||
// without interrupting any planner, main-agent, or worker call.
|
||||
func (m *Manager) UpdateTaskMetadata(taskID string, patch pgdb.TaskPatch) (*Task, error) {
|
||||
m.taskStateMu.Lock()
|
||||
defer m.taskStateMu.Unlock()
|
||||
@@ -1200,6 +1235,7 @@ func (m *Manager) UpdateTaskMetadata(taskID string, patch pgdb.TaskPatch) (*Task
|
||||
}
|
||||
task.updateLifecycle(func(state *taskLifecycleState) {
|
||||
state.ShareGlobalAssets = updated.ShareGlobalAssets
|
||||
state.WorkerCount = updated.WorkerCount
|
||||
state.Name = updated.Name
|
||||
state.Remark = updated.Remark
|
||||
state.PinnedAt = unixOrZero(updated.PinnedAt)
|
||||
|
||||
@@ -55,6 +55,7 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
|
||||
state.CompletedAt = generation
|
||||
state.FirstRunAt = generation
|
||||
state.DeadlineAt = generation
|
||||
state.WorkerCount = int(generation % 51)
|
||||
state.SourceTaskIDs = []int64{generation, generation + 1}
|
||||
state.CompanyIDs = []int64{generation, generation + 1}
|
||||
})
|
||||
@@ -93,7 +94,7 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
|
||||
}
|
||||
generation := state.QueuedAt
|
||||
if state.CompletedAt != generation || state.FirstRunAt != generation || state.DeadlineAt != generation ||
|
||||
len(state.SourceTaskIDs) != 2 || len(state.CompanyIDs) != 2 ||
|
||||
state.WorkerCount != int(generation%51) || len(state.SourceTaskIDs) != 2 || len(state.CompanyIDs) != 2 ||
|
||||
state.SourceTaskIDs[0] != generation || state.CompanyIDs[0] != generation {
|
||||
errCh <- fmt.Errorf("mixed lifecycle snapshot: %+v", state)
|
||||
return
|
||||
@@ -107,3 +108,19 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskLifecycleWorkerCountSnapshot(t *testing.T) {
|
||||
task := &Task{WorkerCount: 2}
|
||||
if got := taskDTO(task, "created").WorkerCount; got != 2 {
|
||||
t.Fatalf("DTO worker_count=%d, want 2", got)
|
||||
}
|
||||
task.updateLifecycle(func(state *taskLifecycleState) {
|
||||
state.WorkerCount = 7
|
||||
})
|
||||
if got := task.lifecycleSnapshot().WorkerCount; got != 7 {
|
||||
t.Fatalf("snapshot worker_count=%d, want 7", got)
|
||||
}
|
||||
if got := taskDTO(task, "created").WorkerCount; got != 7 {
|
||||
t.Fatalf("DTO worker_count=%d, want 7", got)
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,8 @@ import (
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/Autumn-27/artex/internal/httpidentity"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -141,6 +143,7 @@ func (s *Server) sendRepeater(w http.ResponseWriter, r *http.Request) {
|
||||
writeErr(w, http.StatusBadRequest, "构造请求失败")
|
||||
return
|
||||
}
|
||||
httpidentity.Apply(req)
|
||||
for key, value := range in.Headers {
|
||||
name := strings.TrimSpace(key)
|
||||
if name == "" {
|
||||
|
||||
+4
-6
@@ -1285,6 +1285,7 @@ func (s *Server) listTasks(w http.ResponseWriter, r *http.Request) {
|
||||
dtos := make([]TaskDTO, 0, len(list))
|
||||
for _, t := range list {
|
||||
dto := taskDTO(t, s.resolvedTaskStatus(t))
|
||||
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(t)
|
||||
applyTaskArchiveBlocker(&dto, archiveBlockers)
|
||||
metric := metrics[t.ExpID]
|
||||
dto.Tokens = tokenTotalDTO(metric.Tokens)
|
||||
@@ -2068,6 +2069,7 @@ func (s *Server) getTask(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
dto := taskDTO(t, s.resolvedTaskStatus(t))
|
||||
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(t)
|
||||
archiveBlockers, _ := s.m.PG().TaskArchiveBlockers()
|
||||
applyTaskArchiveBlocker(&dto, archiveBlockers)
|
||||
writeJSON(w, 200, dto)
|
||||
@@ -3740,7 +3742,7 @@ func (s *Server) settingsPayload() map[string]any {
|
||||
"show_egress_ip": s.m.pg.GetBool(settingShowEgressIP, false),
|
||||
"egress_probe_url": s.m.EgressProbeURL(),
|
||||
"python_interpreter": strings.TrimSpace(pyStored), // 用户/自动设的值(空=用运行时检测)
|
||||
"workers": s.m.Workers(), // 并发工作 agent 数(默认3);对之后启动的任务生效
|
||||
"workers": s.m.Workers(), // 并发工作 agent 数(默认3);实时更新使用全局默认的任务
|
||||
"worker_pools": s.m.WorkerPools(), // 非空时按池顺序绑定 worker 会话
|
||||
"task_concurrency_enabled": concOn, // 任务并发上限开关(默认关)
|
||||
"task_concurrency_limit": concLimit, // 同时运行任务上限(开启后默认5)
|
||||
@@ -3833,7 +3835,7 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request) {
|
||||
ShowEgressIP *bool `json:"show_egress_ip"` // 在顶栏显示直连/全局代理出口 IP;默认关
|
||||
EgressProbeURL *string `json:"egress_probe_url"` // null=不改;空串=默认公共探测服务
|
||||
PythonInterp *string `json:"python_interpreter"` // 自定义脚本工具的 python 解释器路径
|
||||
Workers *int `json:"workers"` // 并发工作 agent 数(>0);对之后启动的任务生效
|
||||
Workers *int `json:"workers"` // 并发工作 agent 数(1..50);实时更新使用全局默认的任务
|
||||
WorkerPools *[]WorkerPool `json:"worker_pools"`
|
||||
// 任务并发上限:同时「运行中」的任务数上限。关闭=不限;开启后新建任务超限则排队,有空位自动启动。
|
||||
ConcurrencyEnabled *bool `json:"task_concurrency_enabled"`
|
||||
@@ -3923,10 +3925,6 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request) {
|
||||
// synchronizes workers to its total. An empty list clears pools; when workers
|
||||
// is also present it sets the legacy count for the newly restored legacy mode.
|
||||
if req.WorkerPools != nil {
|
||||
if len(*req.WorkerPools) == 0 && req.Workers != nil && *req.Workers <= 0 {
|
||||
writeErr(w, 400, "workers 必须 >0")
|
||||
return
|
||||
}
|
||||
legacyWorkers := []int(nil)
|
||||
if req.Workers != nil {
|
||||
legacyWorkers = []int{*req.Workers}
|
||||
|
||||
+15
-5
@@ -15,7 +15,7 @@ const (
|
||||
maxTaskMetadataRequestSize = 16 << 10
|
||||
)
|
||||
|
||||
// updateTaskMetadata updates editable metadata and the live asset-sharing policy.
|
||||
// updateTaskMetadata updates editable metadata and live task policies.
|
||||
func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
taskID := r.PathValue("id")
|
||||
if _, ok := s.m.Task(taskID); !ok {
|
||||
@@ -28,6 +28,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
Remark *string `json:"remark"`
|
||||
Pinned *bool `json:"pinned"`
|
||||
ShareGlobalAssets *bool `json:"share_global_assets"`
|
||||
WorkerCount *int `json:"worker_count"`
|
||||
}
|
||||
if err := decode(r, &request); err != nil {
|
||||
var tooLarge *http.MaxBytesError
|
||||
@@ -38,8 +39,12 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
return
|
||||
}
|
||||
if request.Name == nil && request.Pinned == nil && request.Remark == nil && request.ShareGlobalAssets == nil {
|
||||
writeErr(w, http.StatusBadRequest, "至少需要提供 name、pinned、remark 或 share_global_assets")
|
||||
if request.Name == nil && request.Pinned == nil && request.Remark == nil && request.ShareGlobalAssets == nil && request.WorkerCount == nil {
|
||||
writeErr(w, http.StatusBadRequest, "至少需要提供 name、pinned、remark、share_global_assets 或 worker_count")
|
||||
return
|
||||
}
|
||||
if request.WorkerCount != nil && (*request.WorkerCount < 0 || *request.WorkerCount > db.MaxTaskWorkerCount) {
|
||||
writeErr(w, http.StatusBadRequest, fmt.Sprintf("worker_count 必须在 0 到 %d 之间", db.MaxTaskWorkerCount))
|
||||
return
|
||||
}
|
||||
if request.Name != nil {
|
||||
@@ -63,7 +68,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
request.Remark = &remark
|
||||
}
|
||||
task, err := s.m.UpdateTaskMetadata(taskID, db.TaskPatch{Name: request.Name, Pinned: request.Pinned, Remark: request.Remark, ShareGlobalAssets: request.ShareGlobalAssets})
|
||||
task, err := s.m.UpdateTaskMetadata(taskID, db.TaskPatch{Name: request.Name, Pinned: request.Pinned, Remark: request.Remark, ShareGlobalAssets: request.ShareGlobalAssets, WorkerCount: request.WorkerCount})
|
||||
if err != nil {
|
||||
writeErr(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
@@ -72,5 +77,10 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
|
||||
writeErr(w, http.StatusNotFound, "task not found")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, taskDTO(task, s.resolvedTaskStatus(task)))
|
||||
if request.WorkerCount != nil {
|
||||
s.engine.ReconcileWorkers(task)
|
||||
}
|
||||
dto := taskDTO(task, s.resolvedTaskStatus(task))
|
||||
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(task)
|
||||
writeJSON(w, http.StatusOK, dto)
|
||||
}
|
||||
@@ -159,3 +159,111 @@ func TestConversationBatchDeleteReportsMissing(t *testing.T) {
|
||||
t.Fatalf("missing conversation %s must include an error", fmt.Sprint(missing))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskMetadataPatchWorkerCount(t *testing.T) {
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable (%v) — skipping", err)
|
||||
}
|
||||
defer m.Close()
|
||||
task, err := m.CreateTask("worker count patch", "goal", nil, 0, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
taskID, _ := strconv.ParseInt(task.ID, 10, 64)
|
||||
defer func() { _ = m.pg.DeleteTask(taskID) }()
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
s := &Server{m: m, engine: NewEngine(m), ctx: ctx}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("PATCH /api/tasks/{id}", s.updateTaskMetadata)
|
||||
mux.HandleFunc("GET /api/tasks/{id}", s.getTask)
|
||||
mux.HandleFunc("GET /api/tasks", s.listTasks)
|
||||
request := func(method, path, body string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
assertCounts := func(body []byte, want int) {
|
||||
t.Helper()
|
||||
var dto struct {
|
||||
WorkerCount int `json:"worker_count"`
|
||||
WorkerActualCount *int `json:"worker_actual_count"`
|
||||
WorkerTargetCount *int `json:"worker_target_count"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &dto); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
actual, target := s.engine.WorkerCounts(task)
|
||||
if dto.WorkerCount != want || dto.WorkerActualCount == nil || dto.WorkerTargetCount == nil {
|
||||
t.Fatalf("missing or incorrect worker counts: %s", body)
|
||||
}
|
||||
if *dto.WorkerActualCount != actual || *dto.WorkerTargetCount != target {
|
||||
t.Fatalf("worker counts=%d/%d, engine=%d/%d", *dto.WorkerActualCount, *dto.WorkerTargetCount, actual, target)
|
||||
}
|
||||
if actual != 0 || (want > 0 && target != want) {
|
||||
t.Fatalf("inactive task worker counts=%d/%d, want actual=0 and configured target=%d", actual, target, want)
|
||||
}
|
||||
}
|
||||
for _, count := range []int{1, 50, 0} {
|
||||
rec := request(http.MethodPatch, "/api/tasks/"+task.ID, fmt.Sprintf(`{"worker_count":%d}`, count))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("patch worker_count=%d status=%d body=%s", count, rec.Code, rec.Body.String())
|
||||
}
|
||||
assertCounts(rec.Body.Bytes(), count)
|
||||
if got := task.lifecycleSnapshot().WorkerCount; got != count {
|
||||
t.Fatalf("in-memory worker count=%d, want %d", got, count)
|
||||
}
|
||||
persisted, err := m.pg.GetTask(taskID)
|
||||
if err != nil || persisted == nil || persisted.WorkerCount != count {
|
||||
t.Fatalf("persisted task=%+v err=%v, want worker_count=%d", persisted, err, count)
|
||||
}
|
||||
rec = request(http.MethodGet, "/api/tasks/"+task.ID, "")
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("get task status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
assertCounts(rec.Body.Bytes(), count)
|
||||
}
|
||||
for _, body := range []string{
|
||||
`{"worker_count":-1,"name":"must not commit"}`,
|
||||
`{"worker_count":51,"name":"must not commit"}`,
|
||||
`{"worker_count":1.5}`,
|
||||
`{"worker_count":"2"}`,
|
||||
`{"worker_count":null}`,
|
||||
} {
|
||||
rec := request(http.MethodPatch, "/api/tasks/"+task.ID, body)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("invalid patch %s status=%d body=%s", body, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
state := task.lifecycleSnapshot()
|
||||
persisted, err := m.pg.GetTask(taskID)
|
||||
if err != nil || persisted == nil || persisted.WorkerCount != 0 || persisted.Name != state.Name || state.Name == "must not commit" || state.WorkerCount != 0 {
|
||||
t.Fatalf("invalid patch changed state: memory=%+v persisted=%+v err=%v", state, persisted, err)
|
||||
}
|
||||
rec := request(http.MethodGet, "/api/tasks", "")
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("list tasks status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var list struct {
|
||||
Tasks []json.RawMessage `json:"tasks"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &list); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, raw := range list.Tasks {
|
||||
var dto struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &dto); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dto.ID == task.ID {
|
||||
assertCounts(raw, 0)
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("updated task missing from list")
|
||||
}
|
||||
@@ -26,6 +26,7 @@ func TestExpandWorkerSlotsLegacy(t *testing.T) {
|
||||
{"configured", 2, 2},
|
||||
{"zero defaults", 0, 3},
|
||||
{"negative defaults", -1, 3},
|
||||
{"oversized caps", db.MaxTaskWorkerCount + 1, db.MaxTaskWorkerCount},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
slots, err := ExpandWorkerSlots(nil, tc.count)
|
||||
@@ -170,6 +171,13 @@ func TestWorkerPoolSettingsRoundTrip(t *testing.T) {
|
||||
if clearResult.Code != http.StatusOK || m.Workers() != 4 || len(m.WorkerPools()) != 0 {
|
||||
t.Fatalf("PUT legacy clear status=%d body=%s workers=%d pools=%+v", clearResult.Code, clearResult.Body.String(), m.Workers(), m.WorkerPools())
|
||||
}
|
||||
for _, body := range []string{`{"workers":51}`, `{"workers":51,"worker_pools":[]}`, `{"workers":0}`} {
|
||||
invalid := httptest.NewRecorder()
|
||||
s.putSettings(invalid, httptest.NewRequest(http.MethodPut, "/api/settings", strings.NewReader(body)))
|
||||
if invalid.Code != http.StatusBadRequest || m.Workers() != 4 || len(m.WorkerPools()) != 0 {
|
||||
t.Fatalf("invalid legacy workers %s status=%d body=%s workers=%d", body, invalid.Code, invalid.Body.String(), m.Workers())
|
||||
}
|
||||
}
|
||||
res = httptest.NewRecorder()
|
||||
s.getSettings(res, httptest.NewRequest(http.MethodGet, "/api/settings", nil))
|
||||
var cleared map[string]any
|
||||
@@ -280,7 +288,7 @@ func TestPinnedWorkerLoopDoesNotConsultTaskRouterAndReportsFailureOnce(t *testin
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
e.workerLoop(ctx, task, WorkerSlot{Name: "work#4", ProfileID: 12})
|
||||
e.workerLoop(ctx, task, WorkerSlot{Name: "work#4", ProfileID: 12}, &workerSlotRuntime{idleCtx: ctx})
|
||||
close(done)
|
||||
}()
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
@@ -391,3 +399,92 @@ func TestDetachedIntentPinnedOwnerUsesPinnedResolver(t *testing.T) {
|
||||
t.Fatalf("intent changed after unavailable resolver: node=%+v err=%v", node, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWorkerSettingsDirectSettersNotifyAfterPersistence(t *testing.T) {
|
||||
m, err := NewManager(t.TempDir(), "")
|
||||
if err != nil {
|
||||
t.Skipf("postgres unavailable: %v", err)
|
||||
}
|
||||
defer m.Close()
|
||||
oldPools, poolsSet, _ := m.pg.GetSetting(settingWorkerPools)
|
||||
oldWorkers, workersSet, _ := m.pg.GetSetting(settingWorkers)
|
||||
defer func() {
|
||||
for key, saved := range map[string]struct {
|
||||
value string
|
||||
set bool
|
||||
}{
|
||||
settingWorkerPools: {oldPools, poolsSet},
|
||||
settingWorkers: {oldWorkers, workersSet},
|
||||
} {
|
||||
if saved.set {
|
||||
_ = m.pg.SetSetting(key, saved.value)
|
||||
} else {
|
||||
_, _ = m.pg.Exec("DELETE FROM settings WHERE key=$1", key)
|
||||
}
|
||||
}
|
||||
}()
|
||||
profileID, err := m.pg.SaveProfile(&db.LLMProfile{Name: "worker-notify-test", Format: "openai", Model: "worker-notify-test", APIKey: "worker-notify-test-key"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer m.pg.DeleteProfile(profileID)
|
||||
var notifications int
|
||||
wantWorkers, wantPools := 0, 0
|
||||
callback := func() {
|
||||
notifications++
|
||||
// Re-enter Manager APIs: callbacks must run after the write and outside mu.
|
||||
if got := m.Workers(); got != wantWorkers {
|
||||
t.Errorf("callback observed workers=%d, want %d", got, wantWorkers)
|
||||
}
|
||||
if got := len(m.WorkerPools()); got != wantPools {
|
||||
t.Errorf("callback observed %d pools, want %d", got, wantPools)
|
||||
}
|
||||
m.SetWorkerConfigChangedCallback(nil)
|
||||
}
|
||||
m.SetWorkerConfigChangedCallback(callback)
|
||||
check := func(want int) {
|
||||
t.Helper()
|
||||
if notifications != want {
|
||||
t.Fatalf("notifications=%d, want %d", notifications, want)
|
||||
}
|
||||
}
|
||||
wantWorkers, wantPools = 50, len(m.WorkerPools())
|
||||
if err := m.SetWorkers(50); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(1)
|
||||
m.SetWorkerConfigChangedCallback(callback)
|
||||
for _, count := range []int{-1, 0, 51} {
|
||||
if err := m.SetWorkers(count); err == nil {
|
||||
t.Fatalf("SetWorkers(%d) accepted", count)
|
||||
}
|
||||
if err := m.SetWorkerPools(nil, count); err == nil {
|
||||
t.Fatalf("SetWorkerPools(nil, %d) accepted", count)
|
||||
}
|
||||
check(1)
|
||||
}
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: 9223372036854775807, Workers: 1}}); err == nil {
|
||||
t.Fatal("missing profile accepted")
|
||||
}
|
||||
check(1)
|
||||
if got := m.Workers(); got != 50 {
|
||||
t.Fatalf("rejected worker settings changed persisted workers=%d, want 50", got)
|
||||
}
|
||||
wantWorkers, wantPools = 2, 1
|
||||
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(2)
|
||||
m.SetWorkerConfigChangedCallback(callback)
|
||||
wantWorkers, wantPools = 3, 0
|
||||
if err := m.SetWorkerPools(nil, 3); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
check(3)
|
||||
if err := m.pg.SetSetting(settingWorkers, "1000"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := m.Workers(); got != db.MaxTaskWorkerCount {
|
||||
t.Fatalf("old oversized workers=%d, want %d", got, db.MaxTaskWorkerCount)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
harvest_static.py <BASE_URL> <OUTDIR>
|
||||
harvest_static.py <BASE_URL> <OUTDIR> [--user-agent VALUE]
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标站点调整,常见改动:
|
||||
- extract_endpoints 正则(endpoint 方言)
|
||||
@@ -16,16 +16,54 @@ Static SPA bundle harvester. Framework-agnostic; tuned for webpack + Vite.
|
||||
|
||||
Stdlib only. TLS verification is disabled (recon against self-signed/internal hosts).
|
||||
"""
|
||||
import sys, os, re, ssl, json, urllib.request, urllib.parse
|
||||
import sys, os, re, ssl, secrets, json, argparse, urllib.request, urllib.parse
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon)"
|
||||
USER_AGENTS = (
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
|
||||
)
|
||||
|
||||
def fetch(url, binary=False):
|
||||
|
||||
def user_agent(value=None):
|
||||
"""Explicit value > environment > random sample; explicit empty omits UA."""
|
||||
explicit = value is not None
|
||||
if value is None:
|
||||
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
|
||||
value = value.strip(" ")
|
||||
if not all(32 <= ord(c) <= 126 for c in value):
|
||||
return secrets.choice(USER_AGENTS)
|
||||
if value or explicit:
|
||||
return value
|
||||
return secrets.choice(USER_AGENTS)
|
||||
|
||||
|
||||
def fetch(url, binary=False, user_agent_value=None):
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"User-Agent": UA})
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=25) as r:
|
||||
ua = user_agent(user_agent_value)
|
||||
req = urllib.request.Request(url, headers={"User-Agent": ua} if ua else {})
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
|
||||
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
|
||||
with opener.open(req, timeout=25) as r:
|
||||
data = r.read()
|
||||
return (data if binary else data.decode("utf-8", "ignore")), r.status
|
||||
except Exception as e:
|
||||
@@ -112,14 +150,17 @@ def extract_routes(js_text):
|
||||
|
||||
# ---- main -------------------------------------------------------------------
|
||||
def main():
|
||||
if len(sys.argv) < 3:
|
||||
print("usage: harvest_static.py <BASE_URL> <OUTDIR>"); sys.exit(1)
|
||||
base, outdir = sys.argv[1], sys.argv[2]
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("base"); ap.add_argument("outdir")
|
||||
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
|
||||
args = ap.parse_args()
|
||||
base, outdir = args.base, args.outdir
|
||||
ua = user_agent(args.user_agent)
|
||||
if not base.startswith("http"): base = "https://" + base
|
||||
jsdir = os.path.join(outdir, "js"); os.makedirs(jsdir, exist_ok=True)
|
||||
|
||||
print(f"[*] entry: {base}")
|
||||
html, status = fetch(base)
|
||||
html, status = fetch(base, user_agent_value=ua)
|
||||
open(os.path.join(outdir, "index.html"), "w").write(html)
|
||||
origin = origin_of(base)
|
||||
|
||||
@@ -137,7 +178,7 @@ def main():
|
||||
if not fn.endswith(".js"): return None
|
||||
lp = os.path.join(jsdir, fn)
|
||||
if url in have: return have[url]
|
||||
data, st = fetch(url, binary=True)
|
||||
data, st = fetch(url, binary=True, user_agent_value=ua)
|
||||
if st == 200 and data and not data[:15].lstrip().startswith(b"<"):
|
||||
open(lp, "wb").write(data); have[url] = lp; return lp
|
||||
return None
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4]
|
||||
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4] [--user-agent VALUE]
|
||||
|
||||
参考模板 — 非通用成品。执行前须按目标调整 --exclude、cookie、depth/max 等同域策略。
|
||||
|
||||
@@ -14,20 +14,58 @@ the same origin and extracts:
|
||||
|
||||
Stdlib only. Same-origin, bounded, polite. Provide --cookie for an authed crawl.
|
||||
"""
|
||||
import sys, os, re, ssl, argparse, urllib.request, urllib.parse, time
|
||||
import sys, os, re, ssl, secrets, argparse, urllib.request, urllib.parse, time
|
||||
from html.parser import HTMLParser
|
||||
from collections import deque
|
||||
|
||||
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
|
||||
UA = "Mozilla/5.0 (spa-api-recon spider)"
|
||||
USER_AGENTS = (
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
|
||||
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
|
||||
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
|
||||
)
|
||||
|
||||
|
||||
def user_agent(value=None):
|
||||
"""Explicit value > environment > random sample; explicit empty omits UA."""
|
||||
explicit = value is not None
|
||||
if value is None:
|
||||
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
|
||||
value = value.strip(" ")
|
||||
if not all(32 <= ord(c) <= 126 for c in value):
|
||||
return secrets.choice(USER_AGENTS)
|
||||
if value or explicit:
|
||||
return value
|
||||
return secrets.choice(USER_AGENTS)
|
||||
|
||||
SKIP_EXT = re.compile(r'\.(css|png|jpe?g|gif|svg|ico|woff2?|ttf|pdf|zip|mp4|webp|js)(\?|$)', re.I)
|
||||
API_HINT = re.compile(r'/(?:api|rest|service|ajax|action|do|rpc|graphql|v\d|admin|backend)\b', re.I)
|
||||
|
||||
def fetch(url, cookie):
|
||||
h = {"User-Agent": UA}
|
||||
def fetch(url, cookie, user_agent_value=None):
|
||||
ua = user_agent(user_agent_value)
|
||||
h = {"User-Agent": ua} if ua else {}
|
||||
if cookie: h["Cookie"] = cookie
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(url, headers=h), context=CTX, timeout=20) as r:
|
||||
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
|
||||
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
|
||||
with opener.open(urllib.request.Request(url, headers=h), timeout=20) as r:
|
||||
ct = r.headers.get("Content-Type", "")
|
||||
if "html" not in ct and "xml" not in ct: return "", ct
|
||||
return r.read().decode("utf-8", "ignore"), ct
|
||||
@@ -70,7 +108,9 @@ def main():
|
||||
ap.add_argument("--cookie", default=""); ap.add_argument("--max", type=int, default=200)
|
||||
ap.add_argument("--depth", type=int, default=4); ap.add_argument("--delay", type=float, default=0.1)
|
||||
ap.add_argument("--include", default=""); ap.add_argument("--exclude", default="logout|signout|delete|remove|destroy")
|
||||
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
|
||||
args = ap.parse_args()
|
||||
ua = user_agent(args.user_agent)
|
||||
base = args.base if args.base.startswith("http") else "https://" + args.base
|
||||
os.makedirs(args.outdir, exist_ok=True)
|
||||
origin = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(base))
|
||||
@@ -82,7 +122,7 @@ def main():
|
||||
n = 0
|
||||
while q and n < args.max:
|
||||
url, d = q.popleft(); n += 1
|
||||
html, ct = fetch(url, args.cookie)
|
||||
html, ct = fetch(url, args.cookie, user_agent_value=ua)
|
||||
if args.delay: time.sleep(args.delay)
|
||||
if not html: continue
|
||||
p = Page()
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Offline regressions: python3 -m unittest discover -s skills/api-recon/scripts."""
|
||||
import contextlib
|
||||
import http.server
|
||||
import importlib.util
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
SCRIPTS = Path(__file__).resolve().parent
|
||||
# Keep the offline fixtures independent of the runner's outbound/recording proxy.
|
||||
DIRECT_ENV = {"no_proxy": "*", "NO_PROXY": "*"}
|
||||
|
||||
|
||||
def load_script(name):
|
||||
spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def capture_server():
|
||||
seen = []
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
seen.append((self.path, self.headers.get("User-Agent"), self.headers.get("Cookie")))
|
||||
if self.path == "/start":
|
||||
self.send_response(302)
|
||||
self.send_header("Location", "/page")
|
||||
self.end_headers()
|
||||
return
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html")
|
||||
self.end_headers()
|
||||
self.wfile.write(b'<h1>Page</h1><script src="/app.js"></script><form action="/submit"></form>')
|
||||
|
||||
def log_message(self, *args):
|
||||
pass
|
||||
|
||||
with http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
|
||||
thread = threading.Thread(target=server.serve_forever)
|
||||
thread.start()
|
||||
try:
|
||||
yield f"http://127.0.0.1:{server.server_port}", seen
|
||||
finally:
|
||||
server.shutdown()
|
||||
thread.join()
|
||||
|
||||
|
||||
class UserAgentTests(unittest.TestCase):
|
||||
def assert_identity(self, seen, expected, samples):
|
||||
self.assertTrue(seen)
|
||||
if expected == "random":
|
||||
self.assertIn(seen[0][1], samples)
|
||||
expected = seen[0][1]
|
||||
self.assertTrue(all(ua == expected for _, ua, _ in seen), seen)
|
||||
|
||||
def test_random_selection_and_explicit_override(self):
|
||||
go_source = (SCRIPTS.parents[2] / "internal/httpidentity/useragent.go").read_text()
|
||||
go_pool = tuple(re.findall(r'^\s*"(Mozilla/5\.0 [^"]+)",$', go_source, re.MULTILINE))
|
||||
self.assertEqual(len(go_pool), 20)
|
||||
self.assertEqual(len(set(go_pool)), 20)
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
self.assertEqual(module.USER_AGENTS, go_pool)
|
||||
for sample in module.USER_AGENTS:
|
||||
with self.subTest(script=name, sample=sample):
|
||||
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": ""}), patch.object(module.secrets, "choice", return_value=sample) as choose:
|
||||
self.assertEqual(module.user_agent(), sample)
|
||||
choose.assert_called_once_with(module.USER_AGENTS)
|
||||
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": "Configured-UA"}), patch.object(module.secrets, "choice") as choose:
|
||||
self.assertEqual(module.user_agent(), "Configured-UA")
|
||||
self.assertEqual(module.user_agent("Override-UA"), "Override-UA")
|
||||
self.assertEqual(module.user_agent(""), "")
|
||||
choose.assert_not_called()
|
||||
|
||||
def test_fetch_and_redirect(self):
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
for value, expected in (("", "random"), (" Captured-UA/1.0 ", "Captured-UA/1.0"),
|
||||
("ok\r\nX-Injected: yes", "random"), ("客户端", "random")):
|
||||
with self.subTest(script=name, value=value), capture_server() as (base, seen):
|
||||
with patch.dict(os.environ, {**DIRECT_ENV, "ARTEX_HTTP_USER_AGENT": value}):
|
||||
if name == "harvest_static":
|
||||
body, status = module.fetch(base + "/start")
|
||||
self.assertEqual(status, 200)
|
||||
else:
|
||||
body, _ = module.fetch(base + "/start", "session=sample")
|
||||
self.assertTrue(all(cookie == "session=sample" for _, _, cookie in seen))
|
||||
self.assertIn("Page", body)
|
||||
self.assertEqual([path for path, _, _ in seen], ["/start", "/page"])
|
||||
self.assert_identity(seen, expected, module.USER_AGENTS)
|
||||
|
||||
def test_cli_precedence_and_outputs(self):
|
||||
for name in ("harvest_static", "spider_mpa"):
|
||||
module = load_script(name)
|
||||
for env_ua, flags, expected in (("", [], "random"),
|
||||
("Environment-UA/1.0", [], "Environment-UA/1.0"),
|
||||
("Environment-UA/1.0", ["--user-agent", "Override-UA/1.0"], "Override-UA/1.0"),
|
||||
("Environment-UA/1.0", ["--user-agent", ""], None)):
|
||||
with self.subTest(script=name, flags=flags), capture_server() as (base, seen):
|
||||
with tempfile.TemporaryDirectory() as outdir:
|
||||
env = dict(os.environ, **DIRECT_ENV, ARTEX_HTTP_USER_AGENT=env_ua)
|
||||
command = [sys.executable, str(SCRIPTS / f"{name}.py"), base + "/start", outdir] + flags
|
||||
if name == "spider_mpa":
|
||||
command += ["--max", "1", "--delay", "0"]
|
||||
result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 0, result.stderr)
|
||||
self.assertGreaterEqual(len(seen), 2)
|
||||
self.assert_identity(seen, expected, module.USER_AGENTS)
|
||||
artifact = "api_static.txt" if name == "harvest_static" else "forms.txt"
|
||||
self.assertTrue((Path(outdir) / artifact).exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -412,6 +412,20 @@ type sink struct {
|
||||
t *Traffic
|
||||
}
|
||||
|
||||
// Preserve an absent User-Agent when go-mitmproxy creates its outbound Go
|
||||
// request. An explicit empty value suppresses net/http's automatic default.
|
||||
func (s *sink) Requestheaders(f *mproxy.Flow) {
|
||||
if f.Request == nil {
|
||||
return
|
||||
}
|
||||
if f.Request.Header == nil {
|
||||
f.Request.Header = make(http.Header)
|
||||
}
|
||||
if _, present := f.Request.Header["User-Agent"]; !present {
|
||||
f.Request.Header.Set("User-Agent", "")
|
||||
}
|
||||
}
|
||||
|
||||
func (s *sink) Response(f *mproxy.Flow) {
|
||||
if f.Request == nil || f.Response == nil {
|
||||
return
|
||||
@@ -683,6 +697,9 @@ func headerLines(h map[string][]string) string {
|
||||
func requestHeaderLines(req *mproxy.Request) string {
|
||||
headers := req.Header.Clone()
|
||||
headers.Del("Host")
|
||||
if headers.Get("User-Agent") == "" {
|
||||
headers.Del("User-Agent") // Empty suppression value is not sent on the wire.
|
||||
}
|
||||
host := req.URL.Host
|
||||
if raw := req.Raw(); raw != nil && strings.TrimSpace(raw.Host) != "" {
|
||||
host = raw.Host
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
package traffic
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
|
||||
)
|
||||
|
||||
func TestProxyPreservesUserAgent(t *testing.T) {
|
||||
for _, ua := range []string{"", "Captured-UA/1.0"} {
|
||||
t.Run(ua, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if got := r.UserAgent(); got != ua {
|
||||
t.Errorf("forwarded User-Agent = %q, want %q", got, ua)
|
||||
}
|
||||
if _, present := r.Header["User-Agent"]; ua == "" && present {
|
||||
t.Error("proxy reintroduced a User-Agent")
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
u, _ := url.Parse(server.URL)
|
||||
flow := &mproxy.Flow{Request: &mproxy.Request{URL: u, Header: make(http.Header)}}
|
||||
if ua != "" {
|
||||
flow.Request.Header.Set("User-Agent", ua)
|
||||
}
|
||||
(&sink{}).Requestheaders(flow)
|
||||
// go-mitmproxy rebuilds the outbound request by adding every value.
|
||||
req, _ := http.NewRequest(http.MethodGet, server.URL, nil)
|
||||
for key, values := range flow.Request.Header {
|
||||
for _, value := range values {
|
||||
req.Header.Add(key, value)
|
||||
}
|
||||
}
|
||||
resp, err := server.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if ua == "" && strings.Contains(requestHeaderLines(flow.Request), "User-Agent:") {
|
||||
t.Error("recording contains a synthetic header that was not sent")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,8 @@ import {
|
||||
AlertDialogTrigger,
|
||||
} from "@/components/ui/alert-dialog";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Label } from "@/components/ui/label";
|
||||
import {
|
||||
Popover,
|
||||
PopoverContent,
|
||||
@@ -40,6 +42,7 @@ import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip
|
||||
import { type UITranslate, useUITranslation } from "@/i18n/use-ui-translation";
|
||||
import { api } from "@/lib/api";
|
||||
import type { LLMProfile, Task } from "@/lib/types";
|
||||
import { MAX_TASK_WORKER_COUNT, parseWorkerCountInput } from "@/lib/types";
|
||||
|
||||
import { AssetsTab } from "./_tabs/assets-tab";
|
||||
import { BroadcastTab } from "./_tabs/broadcast-tab";
|
||||
@@ -212,6 +215,87 @@ function TaskLLMControl({ task, profiles, onUpdated }: { task: Task; profiles: L
|
||||
);
|
||||
}
|
||||
|
||||
function TaskWorkerControl({ task, onUpdated }: { task: Task; onUpdated: (task: Task) => void }) {
|
||||
const { ui } = useUITranslation();
|
||||
const [open, setOpen] = React.useState(false);
|
||||
const [draft, setDraft] = React.useState(String(task.worker_count ?? 0));
|
||||
const [saving, setSaving] = React.useState(false);
|
||||
const actual = task.worker_actual_count;
|
||||
const target = task.worker_target_count;
|
||||
const draining = actual !== undefined && target !== undefined && actual > target;
|
||||
|
||||
const save = async (event: React.FormEvent<HTMLFormElement>) => {
|
||||
event.preventDefault();
|
||||
const count = parseWorkerCountInput(draft);
|
||||
if (count === null) {
|
||||
toast.error(ui("taskUI.invalidWorkers", { max: MAX_TASK_WORKER_COUNT }));
|
||||
return;
|
||||
}
|
||||
setSaving(true);
|
||||
try {
|
||||
const updated = await api.updateTask(task.id, { worker_count: count });
|
||||
onUpdated(updated);
|
||||
setOpen(false);
|
||||
toast.success(ui("taskWorkers.saved"));
|
||||
} catch (error) {
|
||||
toast.error(ui("taskWorkers.saveFailed", { message: (error as Error).message }));
|
||||
} finally {
|
||||
setSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
return (
|
||||
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted-foreground">
|
||||
<span className="tabular-nums">{ui("taskWorkers.counts", { actual: actual ?? "—", target: target ?? "—" })}</span>
|
||||
<Popover
|
||||
open={open}
|
||||
onOpenChange={(next) => {
|
||||
setOpen(next);
|
||||
if (next) setDraft(String(task.worker_count ?? 0));
|
||||
}}
|
||||
>
|
||||
<PopoverTrigger asChild>
|
||||
<Button size="sm" variant="outline" className="h-7 text-xs">
|
||||
{ui("taskWorkers.edit")}
|
||||
</Button>
|
||||
</PopoverTrigger>
|
||||
<PopoverContent align="start" className="w-[min(25rem,calc(100vw-2rem))] gap-4 p-4">
|
||||
<PopoverHeader>
|
||||
<PopoverTitle>{ui("taskWorkers.edit")}</PopoverTitle>
|
||||
<PopoverDescription>{ui("taskWorkers.help")}</PopoverDescription>
|
||||
</PopoverHeader>
|
||||
<form onSubmit={save} className="space-y-3">
|
||||
<div className="space-y-2">
|
||||
<Label htmlFor="task-worker-count">{ui("taskWorkers.label")}</Label>
|
||||
<Input
|
||||
id="task-worker-count"
|
||||
type="number"
|
||||
min={0}
|
||||
max={MAX_TASK_WORKER_COUNT}
|
||||
step={1}
|
||||
value={draft}
|
||||
disabled={saving}
|
||||
onChange={(event) => setDraft(event.target.value)}
|
||||
/>
|
||||
</div>
|
||||
<p className="text-xs text-muted-foreground">{ui("taskWorkers.transitionHelp")}</p>
|
||||
<div className="flex justify-end gap-2">
|
||||
<Button type="button" variant="outline" size="sm" onClick={() => setOpen(false)} disabled={saving}>
|
||||
{ui("screen.3fd47edce45b36")}
|
||||
</Button>
|
||||
<Button type="submit" size="sm" disabled={saving}>
|
||||
{saving && <Spinner data-icon="inline-start" />}
|
||||
{ui(saving ? "screen.a1a1720d3eb58a" : "screen.a3030bf8f16dc6")}
|
||||
</Button>
|
||||
</div>
|
||||
</form>
|
||||
</PopoverContent>
|
||||
</Popover>
|
||||
{draining && <span className="text-amber-600">{ui("taskWorkers.draining")}</span>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function TaskDetailInner() {
|
||||
const { ui, liveUI } = useUITranslation();
|
||||
|
||||
@@ -437,6 +521,14 @@ function TaskDetailInner() {
|
||||
</Button>
|
||||
</div>
|
||||
<p className="truncate text-xs text-muted-foreground">{task.goal}</p>
|
||||
<TaskWorkerControl
|
||||
key={task.id}
|
||||
task={task}
|
||||
onUpdated={(updated) => {
|
||||
setTask(updated);
|
||||
load();
|
||||
}}
|
||||
/>
|
||||
{task.paused && task.pause_reason && (
|
||||
<div className="flex flex-wrap items-center gap-2 text-xs text-amber-600">
|
||||
<span>
|
||||
|
||||
@@ -1731,8 +1731,8 @@ export default function SystemSettingsPage() {
|
||||
</CardTitle>
|
||||
<CardDescription>
|
||||
{t("settings.workersHelpStart")}
|
||||
<b>{t("settings.futureTasksOnly")}</b>
|
||||
{t("settings.runningTasksUnchanged")}
|
||||
<b>{t("settings.workersLiveUpdate")}</b>
|
||||
{t("settings.workersDrainHelp")}
|
||||
</CardDescription>
|
||||
</CardHeader>
|
||||
<CardContent className="flex flex-col gap-3">
|
||||
|
||||
@@ -12,14 +12,14 @@ export const settingsMessages = {
|
||||
"settings.workerLimit": ["Worker 总数不能超过 50", "The total number of Workers cannot exceed 50"],
|
||||
"settings.duplicatePoolProfile": ["不能重复选择同一个 LLM 配置", "Each LLM profile can only be selected once"],
|
||||
"settings.workerPoolsSaved": [
|
||||
"已保存多模型 Worker 池(对之后启动的任务生效)",
|
||||
"Model pools saved; applies to tasks started afterward",
|
||||
"已保存多模型 Worker 池,跟随全局配置的任务已热更新",
|
||||
"Model pools saved; tasks following the global configuration have been updated live",
|
||||
],
|
||||
"settings.saveFailed": ["保存失败:{{message}}", "Could not save: {{message}}"],
|
||||
"settings.positiveConcurrency": ["并发数必须是大于 0 的整数", "Concurrency must be a positive whole number"],
|
||||
"settings.workersSaved": [
|
||||
"已保存并发工作 agent 数(对之后启动的任务生效)",
|
||||
"Worker count saved; applies to tasks started afterward",
|
||||
"已保存工作并发,跟随全局配置的任务已热更新",
|
||||
"Worker count saved; tasks following the global configuration have been updated live",
|
||||
],
|
||||
"settings.saveFailedPrefix": ["保存失败:", "Could not save: "],
|
||||
"settings.pythonSaved": ["已保存 Python 解释器配置", "Python interpreter settings saved"],
|
||||
@@ -427,8 +427,14 @@ export const settingsMessages = {
|
||||
"每个任务并发运行的工作 agent 数量(默认 3)。数值越大并发探测越多、消耗也越高。修改后",
|
||||
"The number of Workers running concurrently per task (default: 3). More Workers mean more parallel probes and higher usage. Changes ",
|
||||
],
|
||||
"settings.futureTasksOnly": ["对之后启动的任务生效", "apply to tasks started afterward"],
|
||||
"settings.runningTasksUnchanged": [",正在运行的任务不受影响。", "; running tasks are unaffected."],
|
||||
"settings.workersLiveUpdate": [
|
||||
"即时更新跟随全局配置(worker_count = 0)的任务",
|
||||
"apply immediately to tasks following the global configuration (worker_count = 0)",
|
||||
],
|
||||
"settings.workersDrainHelp": [
|
||||
";单任务覆盖值不受影响。增容启动新 Worker loops;缩容等待在跑意图完成后退出,不会强杀,实际数量可能暂时高于目标。",
|
||||
"; per-task overrides are unchanged. Scaling up starts new Worker loops; scaling down lets running intents finish before loops exit, without force-cancelling them. Actual counts may temporarily exceed target.",
|
||||
],
|
||||
"settings.workerPools": ["多模型 Worker 池", "Model pools"],
|
||||
"settings.addPool": ["添加模型池", "Add model pool"],
|
||||
"settings.unavailableProfile": ["配置 #{{id}} 已删除/不可用", "Profile #{{id}} deleted or unavailable"],
|
||||
|
||||
@@ -1,4 +1,21 @@
|
||||
export const taskMessages = {
|
||||
"taskWorkers.edit": ["调整 Worker 并发", "Adjust Worker concurrency"],
|
||||
"taskWorkers.label": ["Worker 数量(0–50)", "Worker count (0–50)"],
|
||||
"taskWorkers.counts": ["Worker 实际 / 目标:{{actual}} / {{target}}", "Workers actual / target: {{actual}} / {{target}}"],
|
||||
"taskWorkers.help": [
|
||||
"0 表示跟随全局 Worker 配置;1–50 为本任务覆盖值,保存后热更新,无需重启任务。",
|
||||
"Use 0 to follow the global Worker configuration, or 1–50 to override it for this task. Saving applies changes live without restarting the task.",
|
||||
],
|
||||
"taskWorkers.transitionHelp": [
|
||||
"实际数量是存活的 Worker loops(含缩容退出中的 loops),不是正在执行的意图数。增容会启动新 loops;缩容等待正在执行的意图完成后退出,不会强杀意图,因此实际数量可能暂时高于目标。",
|
||||
"Actual counts live Worker loops, including draining loops, not in-flight intents. Scaling up starts new loops. Scaling down lets running intents finish before loops exit, without force-cancelling them, so actual may temporarily exceed target.",
|
||||
],
|
||||
"taskWorkers.draining": [
|
||||
"缩容过渡中:等待在跑意图完成,不会强杀。",
|
||||
"Scaling down: waiting for running intents to finish; they will not be force-cancelled.",
|
||||
],
|
||||
"taskWorkers.saved": ["Worker 并发已更新", "Worker concurrency updated"],
|
||||
"taskWorkers.saveFailed": ["Worker 并发更新失败:{{message}}", "Could not update Worker concurrency: {{message}}"],
|
||||
"taskFiles.tab": ["文件", "Files"],
|
||||
"taskFiles.tree": ["目录树", "Directory tree"],
|
||||
"taskFiles.recent": ["最近文件", "Recent files"],
|
||||
|
||||
+1
-1
@@ -483,7 +483,7 @@ export const api = {
|
||||
taskCategories: () => get<{ categories: TaskCategory[] }>("/task-categories").then((r) => arr(r.categories)),
|
||||
updateTask: (
|
||||
id: string,
|
||||
input: { name?: string; pinned?: boolean; remark?: string; share_global_assets?: boolean },
|
||||
input: { name?: string; pinned?: boolean; remark?: string; share_global_assets?: boolean; worker_count?: number },
|
||||
) => patch<Task>(`/tasks/${id}`, input),
|
||||
renameTask: (id: string, name: string) => patch<Task>(`/tasks/${id}`, { name }),
|
||||
pinTask: (id: string, pinned: boolean) => patch<Task>(`/tasks/${id}`, { pinned }),
|
||||
|
||||
@@ -58,6 +58,9 @@ export const tasks: Task[] = [
|
||||
description: "Acme 官网与后台外部渗透(acme.com)",
|
||||
goal: "拿到 acme.com 后台管理权限,确认可读取用户敏感数据。",
|
||||
status: "running",
|
||||
worker_count: 0,
|
||||
worker_actual_count: 3,
|
||||
worker_target_count: 3,
|
||||
created_at: T("2026-07-24T09:12:00Z"),
|
||||
created_unix: 1785489120,
|
||||
active_run_seconds: 28800,
|
||||
@@ -87,6 +90,9 @@ export const tasks: Task[] = [
|
||||
description: "api.acme.com 越权与注入测试",
|
||||
goal: "评估 api.acme.com 订单/用户接口的越权(IDOR)与注入风险。",
|
||||
status: "running",
|
||||
worker_count: 0,
|
||||
worker_actual_count: 3,
|
||||
worker_target_count: 3,
|
||||
created_at: T("2026-07-25T14:05:00Z"),
|
||||
created_unix: 1785592500,
|
||||
active_run_seconds: 14400,
|
||||
@@ -116,6 +122,9 @@ export const tasks: Task[] = [
|
||||
description: "shop.acme.com 支付与订单链路",
|
||||
goal: "评估支付与订单接口的越权、金额篡改与竞态风险。",
|
||||
status: "paused",
|
||||
worker_count: 0,
|
||||
worker_actual_count: 3,
|
||||
worker_target_count: 3,
|
||||
created_at: T("2026-07-22T08:30:00Z"),
|
||||
created_unix: 1785313800,
|
||||
active_run_seconds: 21600,
|
||||
@@ -144,6 +153,9 @@ export const tasks: Task[] = [
|
||||
description: "对外暴露面侦察(VPN / 边界服务)",
|
||||
goal: "识别 acme.com 对外暴露的可利用边界服务。",
|
||||
status: "done",
|
||||
worker_count: 0,
|
||||
worker_actual_count: 0,
|
||||
worker_target_count: 3,
|
||||
created_at: T("2026-07-18T08:00:00Z"),
|
||||
created_unix: 1784966400,
|
||||
active_run_seconds: 86400,
|
||||
@@ -2921,7 +2933,6 @@ export const trafficDetail: TrafficDetail = {
|
||||
req: `GET /v1/orders?id=1002 HTTP/1.1
|
||||
Host: api.acme.com
|
||||
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
|
||||
User-Agent: ARTEX-worker/1.0
|
||||
Accept: application/json
|
||||
|
||||
`,
|
||||
|
||||
@@ -48,6 +48,7 @@ const mockInterceptHistory = structuredClone(D.interceptHistory);
|
||||
const mockInterceptPending = structuredClone(D.interceptPending);
|
||||
const mockInterceptDetails = structuredClone(D.interceptDetails);
|
||||
const mockTasks = structuredClone(D.tasks);
|
||||
const mockWorkerDrainUntil = new Map<string, number>();
|
||||
const mockFindings = structuredClone(D.findings);
|
||||
const mockLLMRecords = structuredClone(D.llmRecords);
|
||||
const mockTaskTemplates = structuredClone(D.taskTemplates);
|
||||
@@ -256,7 +257,33 @@ function mockTaskArchiveBlocker(taskID: string): string | undefined {
|
||||
)?.id;
|
||||
}
|
||||
|
||||
function reconcileMockWorkers(task: Task): void {
|
||||
const target = task.worker_count || D.settings.workers || 3;
|
||||
const previousTarget = task.worker_target_count;
|
||||
task.worker_target_count = target;
|
||||
if (task.status !== "running" && task.status !== "paused") {
|
||||
task.worker_actual_count = 0;
|
||||
mockWorkerDrainUntil.delete(task.id);
|
||||
return;
|
||||
}
|
||||
const actual = task.worker_actual_count ?? target;
|
||||
if (actual <= target) {
|
||||
task.worker_actual_count = target;
|
||||
mockWorkerDrainUntil.delete(task.id);
|
||||
} else {
|
||||
// Demo loops drain independently of in_flight; retain them across polls briefly.
|
||||
if (!mockWorkerDrainUntil.has(task.id) || previousTarget !== target) {
|
||||
mockWorkerDrainUntil.set(task.id, Date.now() + 10_000);
|
||||
}
|
||||
if (Date.now() >= (mockWorkerDrainUntil.get(task.id) ?? 0)) {
|
||||
task.worker_actual_count = target;
|
||||
mockWorkerDrainUntil.delete(task.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function publicMockTask(task: Task): Task {
|
||||
reconcileMockWorkers(task);
|
||||
const item = structuredClone(task);
|
||||
item.numeric_id = mockTaskAssetIDs.get(task.id);
|
||||
item.share_global_assets = task.share_global_assets ?? false;
|
||||
@@ -1323,6 +1350,7 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
|
||||
company_ids: companyIDs,
|
||||
share_global_assets: b.share_global_assets === true,
|
||||
};
|
||||
reconcileMockWorkers(created);
|
||||
// Validate the attached preview schedule before mutating task state.
|
||||
const numericTaskID = nextMockTaskAssetID;
|
||||
if (b.schedule && typeof b.schedule === "object") {
|
||||
@@ -1527,8 +1555,10 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
|
||||
const namePresent = has("name") && b.name !== null;
|
||||
const remarkPresent = has("remark") && b.remark !== null;
|
||||
const pinnedPresent = has("pinned") && b.pinned !== null;
|
||||
if (!namePresent && !remarkPresent && !pinnedPresent && !sharingPresent)
|
||||
throw new Error("至少需要提供 name、pinned、remark 或 share_global_assets");
|
||||
const workerCountPresent = has("worker_count") && b.worker_count !== null;
|
||||
const nextWorkerCount = workerCountPresent ? mockWorkerCount(b.worker_count) : task.worker_count;
|
||||
if (!namePresent && !remarkPresent && !pinnedPresent && !sharingPresent && !workerCountPresent)
|
||||
throw new Error("至少需要提供 name、pinned、remark、share_global_assets 或 worker_count");
|
||||
let nextName = task.name;
|
||||
let nextRemark = task.remark;
|
||||
let nextPinned = task.pinned;
|
||||
@@ -1550,8 +1580,9 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
|
||||
task.remark = nextRemark;
|
||||
task.pinned = nextPinned;
|
||||
task.pinned_at = nextPinnedAt;
|
||||
task.worker_count = nextWorkerCount;
|
||||
sortMockTasks();
|
||||
return structuredClone(task);
|
||||
return publicMockTask(task);
|
||||
}
|
||||
if (seg[0] === "tasks" && seg.length === 2 && m === "DELETE") {
|
||||
const id = seg[1];
|
||||
@@ -2463,6 +2494,7 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
|
||||
next.workers = next.worker_pools.reduce((sum, pool) => sum + Number(pool.workers), 0);
|
||||
}
|
||||
Object.assign(D.settings, next);
|
||||
for (const task of mockTasks) reconcileMockWorkers(task);
|
||||
return next;
|
||||
}
|
||||
if (path === "/settings/basic-auth" && m === "GET") return { enabled: false, username: "", password_set: false };
|
||||
|
||||
@@ -26,6 +26,8 @@ export interface Task {
|
||||
id: string;
|
||||
remark?: string;
|
||||
worker_count?: number; // 本任务 Worker 数;0/缺省=用全局默认
|
||||
worker_actual_count?: number; // 当前存活的 Worker pool loops,含缩容退出中的 loops
|
||||
worker_target_count?: number; // 当前期望的 Worker slots 数量
|
||||
name?: string; // 可选任务名称;空/缺省=未命名,展示时回退到描述
|
||||
category_id?: number;
|
||||
category_name?: string;
|
||||
@@ -1028,7 +1030,7 @@ export interface Settings {
|
||||
show_egress_ip?: boolean; // 在顶栏显示直连 / 全局代理出口 IP;默认关闭
|
||||
egress_probe_url?: string; // 自定义 IP 探测服务;空=默认公共回显服务
|
||||
python_interpreter?: string; // 自定义脚本工具的 python 解释器路径(空=运行时检测)
|
||||
workers?: number; // 并发工作 agent 数(默认3);对之后启动的任务生效
|
||||
workers?: number; // 并发工作 agent 数(默认3);热更新跟随全局配置的任务
|
||||
worker_pools: WorkerPool[];
|
||||
// 任务并发上限:同时「运行中」的任务数上限。关闭=不限;开启后新建任务超限则排队,有空位自动启动。
|
||||
task_concurrency_enabled?: boolean; // 默认 false
|
||||
|
||||
Reference in new issue
Block a user