Compare commits

..
Author SHA1 Message Date
ji ruo 7d9c5699cb feat(workers): hot-reload task worker concurrency
Support live task overrides and global default/pool reconciliation.
Drain retiring workers without cancelling active intents; expose actual/target loop counts in the API and task UI.
Persist worker policy and cover resizing, API validation, settings notification, and restoration with regression tests.

Fixes #171
2026-11-19 17:26:51 +08:00
ji ruo 6997a46619 chore(release): prepare v0.3.30 release notes 2026-11-14 21:19:39 +08:00
ji ruo cc1d84d573 Merge pull request #155 from RuoJi6/mcp-server-implementation 2026-11-14 20:51:24 +08:00
ji ruo a4ad95d716 Merge pull request #154 from RuoJi6/codex/fix-socks5-nil-panic 2026-11-14 20:51:24 +08:00
ji ruo 900c176435 Merge pull request #152 from RuoJi6/fix/intercept-match-decoded-input 2026-11-14 20:51:24 +08:00
ji ruo 437a28cddc Merge pull request #138 from RuoJi6/agent/agent/github-10 2026-11-14 20:51:24 +08:00
ji ruo 32d8619d36 Merge pull request #122 from RuoJi6/docs/fix-readme-mistakes 2026-11-14 20:51:23 +08:00
RuoJi6 c9b1e240ec Update README.md 2026-11-14 14:22:43 +08:00
RuoJi6 9b8073a763 Update README.md 2026-11-14 14:22:19 +08:00
RuoJi6 faf4d4fb4a Update README.md 2026-11-14 14:22:01 +08:00
darkkid 0fed4ba3a3 fix(traffic): 升级 go-mitmproxy 修复 socks5 上游握手失败时的 nil panic
全局出口代理配置为 socks5:// 时,SOCKS5 握手失败会刷出
"http: panic serving ...: nil pointer dereference"。

根因在依赖库 go-mitmproxy v1.9.2 的 internal/helper/proxy.go:40:
x/net 的 SOCKS5 DialContext 失败时返回 (nil, err),库仍调用
conn.Close(),必然空指针。net/http 每连接 recover,进程不退出,
但每条 CONNECT 被直接掐断(客户端见到 EOF),且 panic 发生在
HTTPConnectError 回调之前,导致 reportUpstreamFailure 不触发,
代理池自愈与任务重连完全收不到失败信号。

上游已在 v1.9.4 修复(issue #110,commit 388f47a),本次升级
v1.9.2 -> v1.9.5。实测同一份 traffic 包:v1.9.2 下 CONNECT 返回
unexpected EOF 且回调不触发;v1.9.5 下返回 502 并正常触发失败回调。

连带升级 go-mitmproxy 要求的最低版本:x/net 0.61.0、x/crypto
0.58.0、x/text 0.43.0、x/sys 0.49.0、x/mod 0.41.0、x/sync 0.24.0、
x/tools 0.51.0。

验证:go build ./... 与 go vet ./... 在 -mod=readonly 下通过;
按包隔离数据库逐包对比升级前后,失败集合完全一致(db、llmrec
为本次改动前既有失败),traffic/evidence/intercept/sidequestion/
guard/notify/server 全绿。
2026-11-14 13:36:57 +08:00
nftpixels 4d66a84ac5 fix(intercept): match tool_input rules against decoded values
tool_input rules were only matched against the raw JSON of the tool call.
JSON escapes newlines and tabs, so a multi-line command reaches the rules as
`cd /tmp\nrm -rf /var/www`. There is no word boundary between the `n` of `\n`
and `rm`, so the built-in `\brm\b` / `\bmkfs\b` / `\bDROP ...` rules never
fire on a dangerous command that starts a new line (heredocs, small scripts).
Quotes (`\"`) and, for Go-encoded input, & < > (`&` etc.) have the same
problem.

Decode the input once per Match and also try each string value. The raw
JSON is still checked first, so existing rules written against the JSON
form keep matching; the change can only add matches.
2026-11-14 11:16:27 +13:00
ji ruo 570dc1a14c fix: expand standard browser user agent pool to twenty samples 2026-11-11 19:21:05 +08:00
ji ruo 171db405dd fix: select random browser user agents for outbound HTTP 2026-11-11 19:13:38 +08:00
ji ruo 17c2fc5e08 fix: make HTTP user agents configurable and remove tool identifiers 2026-11-11 18:56:33 +08:00
ji ruo 40746901db chore(agent): baseline — the task worktree started here 2026-11-11 18:42:23 +08:00
Bhao e45fa6e7b3 docs(readme): 修复前端构建产物复制命令的目录嵌套问题 2026-11-05 22:04:16 +08:00
55 changed files with 2643 additions and 154 deletions

No files matched your search

+3
View File
@@ -13,3 +13,6 @@ ARTEX_LLM_PROVIDER=
ARTEX_LLM_MODEL=
ARTEX_LLM_BASE_URL=
ARTEX_LLM_PROXY=
# HTTP User-Agent;留空时从浏览器 UA 样本中随机抽选,填写则使用固定值
ARTEX_HTTP_USER_AGENT=
+41 -1
View File
@@ -4,8 +4,48 @@
## [Unreleased]
- 新增 MCP 服务端:在「系统设置 → MCP 服务端」开启后,外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务、补充提示。新增访问令牌(权限范围、限定企业、任务限额、有效期、吊销)与调用审计。支持来源白名单(IP/CIDR,可配置可信反向代理),在令牌校验之前生效。默认关闭;`/mcp` 只认访问令牌,不经过也无法使用 HTTP Basic Auth。详见 README「MCP 服务端」。
### 新增与改进
- 支持热更新运行中任务的 Worker 并发:任务详情可设置 0(跟随全局)或 1–50;全局工作并发与 Worker 池变更会同步至跟随全局的任务。增容立即补起 Worker,缩容等待当前意图完成后退出,不中断正在执行的工作;详情展示实际存活 Worker 数与目标数。配置持久化,重启恢复时沿用。
- `PATCH /api/tasks/{id}` 新增 `worker_count`;任务接口返回 `worker_actual_count`(存活池循环,含缩容等待中的 Worker)与 `worker_target_count`(有效目标数),与 `in_flight`(执行中意图数)区分。
## [0.3.30] - 2026-11-14
### 新增与改进
- 新增 MCP 服务端:外部智能体可通过 `/mcp`(Streamable HTTP)查看任务与漏洞、创建任务、暂停/恢复任务及补充提示。提供访问令牌的权限范围、企业限制、任务限额、有效期和吊销,以及调用审计、来源 IP/CIDR 白名单与可信代理配置。默认关闭;该端点使用独立访问令牌认证,不使用 HTTP Basic Auth。
- 统一出站 HTTP 请求标识:Go 请求与两个 Python 采集模板从 20 个标准桌面浏览器 User-Agent 中随机选择,保留环境变量和显式请求头覆盖,同一请求的重定向与重试复用选定值。此变化仅涉及 HTTP 请求头。
### 修复
- 命令拦截除匹配原始工具输入 JSON 外,也匹配解码后的嵌套字符串值,修复换行、制表符和转义引号导致危险命令漏拦的问题;保留原始 JSON 规则及非 JSON 输入的兼容行为。
- 升级 `go-mitmproxy` v1.9.2 → v1.9.5,修复上游 SOCKS5 握手失败时的空指针 panic;失败时返回 502,并正常触发代理失败回调。
- 修正 README 中前端构建产物的复制命令,避免目录嵌套并创建目标目录。
## [0.3.29] - 2026-11-11
### 新增与改进
- 新增可选「无成果止损」:按实际运行时长、独立漏洞数量与最低等级做一次检查,未达标时自动收尾并显示「无成果停止」;暂停、排队及计划窗口外不计时,复活后重新计时,默认关闭。停止状态复用 `timeout`,已有 `on_task_timeout` 触发器仍会执行并可能消耗 token。
- 任务详情新增只读文件浏览,支持目录导航、最近文件、文本与图片预览、单文件下载及目录 ZIP 下载;展示意图目录对应的标题和状态,并提供中英文界面。
### 修复
- 无成果检查与漏洞写入串行化,并在同一事务内重新计算独立漏洞数量,避免临界时刻已达到成果要求的任务被误停止。
- 旁路问答摘要为空、被截断或过长时自动重试一次,并细分失败原因;失败摘要不写入缓存,重试计入原有调用次数、用量及超时限制。
## [0.3.28] - 2026-11-10
### 新增与改进
- 新增英文界面和语言切换,覆盖任务、审批、设置、通知及计划任务等页面,并统一常用状态、日期与错误提示的翻译。
- 流量页面展示完整流量 ID,支持按 ID 搜索及在 HTTP 环境复制;Agent 可直接使用该 ID 读取已有流量。
- 漏洞统计与列表使用相同筛选条件,区分独立漏洞和原始上报记录数量。
### 修复
- Basic Auth 会话失效时在当前标签页恢复认证,避免后台请求反复弹出浏览器认证框,并保留应用登录状态。
- 修复英文界面预览、覆盖度图谱生命周期及部分页面运行时错误,并为认证、出站探测等错误补充可翻译的错误码。
## [0.3.18] - 2026-10-12
+87
View File
@@ -0,0 +1,87 @@
# HTTP 请求标识排查
本次检查覆盖仓库中的 Go HTTP 请求、Python / JavaScript 脚本、安装脚本、前端示例,以及实际使用的 norma v0.4.3 的 WebFetch / WebSearch 实现。排查范围包括 User-Agent 的大小写变体、请求头构造、代理转发、重定向及代理失败后的直连重试。
## 已处理
| 通道 | 排查结果与处理 |
|---|---|
| 内置 WebFetch | SDK 将 `norma/0.4` 写死在首发及重试请求。worker、planner、main、chat 通过统一 session 入口使用本地适配器;保留 SDK 的工具 schema、权限、静态资源限制、代理、CA 验证及 HTML / RECON 输出。 |
| 资产自动补全 | 移除 `artex-enrich/1.0`,HTTP 探测使用统一策略。 |
| 出口连通性探测 | 额外发现 `artex-egress-probe/1.0`,已移除并接入统一策略。 |
| 静态采集 / 同源爬虫模板 | 移除 `spa-api-recon` / `spa-api-recon spider`,避免 urllib 自动填入自己的默认 UA;入口、下载、重定向及重试沿用同一配置。 |
| HTTP 自定义工具 | 原先没有 ARTEX 专属 UA,但未设置时会由 Go 填充默认值。现采用统一策略,模板显式请求头优先。 |
| 请求重放器 | 原先没有 ARTEX 专属 UA。现采用统一策略,用户显式请求头(包括空值)优先,保持原请求的配置能力。 |
| 记录代理 | 转发缺失 UA 的请求时,Go 会补入默认客户端标识。现保留显式省略语义,录制中不展示内部用于抑制默认值的空头。保留客户端已选择的随机或固定 UA。 |
| GitHub 自更新 | 额外发现查询版本、下载校验和与发布包共用的两处 `artex-selfupdate`。均已移除;与其他请求统一采用随机浏览器样本或显式配置的非空 UA,不含默认 ARTEX 名称。 |
| 前端 mock | `ARTEX-worker/1.0` 仅为示例 HTTP 文本,不会发出请求;已删除该行,避免误导。 |
## 配置
默认采用随机 UA 池方案,Go 与两个独立 Python 模板内置相同的 20 个完整浏览器 UA 样本,覆盖 Chrome / Edge / Firefox / Safari 的标准桌面格式。随机选择完整样本,允许重复,不拼接随机版本,不调用 AI 服务。
2026-11-11 核对并更新样本,采用主流浏览器近期稳定版本的标准格式:
| 浏览器 | 版本 | 平台 | 数量 |
|---|---|---|---|
| Chrome | 154 / 155 / 156 / 157 | Windows / macOS / Linux | 12 |
| Edge | 156 / 157 | Windows / macOS | 4 |
| Firefox | 159 | Windows / macOS / Linux | 3 |
| Safari | 27.1 | macOS | 1 |
保留浏览器正常的 UA 缩减字段(如 Chrome 的 `0.0.0` 和 macOS 的冻结版本),不添加工具名称、随机设备型号、扩展标记或自定义后缀。“常见”指主流浏览器的标准 UA 格式,不代表已测量这 20 项在目标站点中的使用占比;浏览器类型和版本仍可从 UA 中读出。
格式与版本参考:
- [Chromium UA Reduction](https://www.chromium.org/updates/ua-reduction/)
- [Chrome 稳定版发布记录](https://chromereleases.googleblog.com/search/label/Stable%20updates)
- [Edge UA 格式](https://learn.microsoft.com/en-us/microsoft-edge/web-platform/user-agent-guidance)
- [Edge 稳定版发布记录](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-relnote-stable-channel)
- [Firefox UA 格式](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent/Firefox)
- [Firefox 159 发布记录](https://www.firefox.com/en-US/firefox/159.0/releasenotes/)
- [Safari 27.1 发布记录](https://developer.apple.com/documentation/safari-release-notes/safari-27_1-release-notes)
- Go 每次 HTTP 逻辑请求抽选一次;WebFetch 每次工具调用重新抽选,重定向、代理请求及失败后的直连重试复用同一值。
- Python 脚本每次运行抽选一次,入口、并发资源下载及重试复用同一值;独立调用 `fetch()` 时每次调用单独选择。
- `ARTEX_HTTP_USER_AGENT` 非空且合法时覆盖随机值;留空、仅空格或非法值回退随机池。Go 使用系统加密随机源;若随机源出错,保留池中首个非空有效样本。
- Python 的 `--user-agent VALUE` 优先于环境变量,`--user-agent ''` 仍显式省略;自定义 HTTP 工具 / 重放器显式提供的 UA(包括空值)同样优先,保留原请求的控制能力。
- 固定值只接受可打印 ASCII,去除首尾空格;控制字符和非 ASCII 值回退随机池,防止头注入。
Go 服务重启后使用新配置;Docker 在 `.env` 设置后重建容器。直接运行 Python 模板时继承同名环境变量。已同步 agent OPSEC 提示,避免旧提示与平台随机策略冲突;遇到限速等异常时仍停止,不以换 UA 继续当前方向。
UA 随机化不会同步改变 TLS、Cookie 或其他浏览器特征,不能保证客户端不可识别;第三方出口代理也可能再次改写请求头。
## 检查过的其他通道与边界
| 通道 | 结果 |
|---|---|
| SDK 联网搜索 | norma 的 DDGS 后端含固定 Firefox 128 浏览器 UA,没有 ARTEX / norma 名称;仅发往搜索服务,属于依赖的兼容策略,未改动,也不受此环境变量控制。Brave / Tavily 及 DeepSeek 搜索未发现 ARTEX 专属 UA。 |
| 浏览器自动化 / runtime_harvest.js | 沿用浏览器请求头;运行时转发复制捕获到的请求头,没有硬编码 ARTEX UA。 |
| LLM 请求 / 模型列表 / JEV | 未发现硬编码 ARTEX UA;仍可能有 HTTP 库、SDK、服务协议和用户自定义标识,不套用目标流量配置。 |
| MCP HTTP / SSE | 未发现 ARTEX UA;显式自定义头保留。初始化协议体仍有 `clientInfo.name = artex`,这是对用户配置的 MCP 服务的协议自报身份,本次不改动。 |
| Webhook / 通知 | 未发现硬编码 ARTEX UA;保留用户自定义头及客户端默认行为。 |
| 安装脚本 | 未发现 ARTEX UA;获取 Docker 安装脚本的 curl 仍使用其原生身份。外部命令及用户自建脚本不由本次策略接管。 |
| phpfinger | 仓库未包含其源码,无法核验默认值;需要在独立工具中通过 `--user-agent` 配置。 |
服务端响应中的 `X-Artex-Auth` 仍用于前端识别 Basic Auth 未认证状态,属于站点自身对访问者的响应协议;它不是向目标站点发送的请求头。本次不改变该协议,也不承诺消除服务端页面、API、TLS 或其他流量特征。
## 回归验证
- Go:检查真实 HTTP 线上头、随机样本选择 / 固定值覆盖 / 显式空值、代理及失败直连重试、重定向、CA 信任、静态资源限制、HTML / RECON 解析、agent session 注册、自定义工具 / 重放器覆盖、自更新查询及下载和代理录制。
- Python:本地 HTTP 服务验证默认随机 / 环境变量 / CLI 覆盖 / 显式空值、非法头值、重定向、Cookie 保留及产物生成;不访问真实目标。
本地命令:
```sh
go test ./internal/httpidentity ./internal/webfetch ./enrich ./agent ./selfupdate ./traffic
go test ./server -run 'TestProbe|TestRepeater|TestRunHTTPTool|TestHTTPTool|TestTargetHTTPIdentity' -count=1
go test -race ./internal/httpidentity ./internal/webfetch ./enrich ./traffic
python3 -m unittest discover -s skills/api-recon/scripts -p 'test_*.py' -v
```
以上专项回归、Go race 检查及 Python 3 项测试均通过。池扩展后的回归还检查恰好 20 个不重复样本、逐项选择以及 Go 与两个 Python 模板的样本一致性。
2026-11-11 使用项目提供的本地 PostgreSQL,在新建的独立测试库中执行了 `go test ./... -count=1 -p 1 -timeout 180s`,测试库用后删除。完整套件未全部通过:
- `agent` 的 `TestGraphOverviewExpandsAssociatedCompanyScope`、`TestCaptureApprovalLifecycle` 和 `db` 的 `TestInterceptApprovalFilters`、`TestSideTaskArchiveVersions` 失败;在原提交 `17c2fc5` 的独立副本和另一新建测试库中复现了相同失败。
- `server` 首次运行有 3 个用例在临时目录清理阶段报 `directory not empty`,涉及 `TestInheritedActivityDetailAndRelationDeletion`、`TestTaskMetadataPatchReturnsRenameAndPin`、`TestTaskMetadataPatchGlobalAssetSharing`,不是 UA 断言失败。再次在新建隔离库中运行完整 `server` 套件时这 3 项通过,但 `TestCoreTaskLifecyclePG` 报 `expected goal nodes seeded, got 0`。原提交的完整 `server` 套件通过;修改后完整 `server` 套件仍未获得全绿结果。
+18 -6
View File
@@ -12,8 +12,13 @@ AI 自主渗透测试系统(Go 后端 + Next.js 前端)
本仓库为独立维护的 [RuoJi6/ARTEX](https://github.com/RuoJi6/ARTEX),保留 https://github.com/Autumn-27/ARTEX 的原始提交历史与 AGPL-3.0 许可证。程序发布包从本仓库 Releases 下载,Docker 镜像 [ruoji6/artex](https://hub.docker.com/r/ruoji6/artex) 由本仓库 GitHub Actions 构建并发布。
---
## 致谢
致谢米斯特安全团队
## 致谢
米斯特安全团队
洺熙
wowo
## 截图预览
@@ -157,9 +162,11 @@ cp config.example.json config.json # 填好 database 连接
```bash
# 1) 前端静态导出
cd web && npm ci && npm run build:static && cd ..
# 2) 拷进内嵌目录
cp -r web/out server/webui/dist
# 3) 编译(-tags embedui 才内嵌前端)
# 2) 创建内嵌目录
mkdir -p server/webui/dist
# 3) 拷进内嵌目录
cp -r web/out/. server/webui/dist
# 4) 编译(-tags embedui 才内嵌前端)
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
./start.sh
```
@@ -290,7 +297,8 @@ cp -r <解压目录>/skills ./ && cp <解压目录>/artex ./
```bash
git pull
cd web && npm ci && npm run build:static && cd ..
cp -r web/out server/webui/dist
mkdir -p server/webui/dist
cp -r web/out/. server/webui/dist
CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
# 重启 ./start.sh
```
@@ -314,6 +322,10 @@ CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
**LLM**:`export ANTHROPIC_API_KEY=sk-...`(或 `OPENAI_API_KEY`),也可在 UI 的「LLM 配置」页填写。
可选:`ARTEX_LLM_PROVIDER` / `ARTEX_LLM_MODEL` / `ARTEX_LLM_BASE_URL` / `ARTEX_LLM_PROXY`。
**HTTP User-Agent**:内置 WebFetch(worker / planner / main / chat)、资产 HTTP 自动补全、出口连通性探测、HTTP 自定义工具、请求重放器、GitHub 自更新和可选的 `harvest_static.py` / `spider_mpa.py` 默认从 20 个浏览器 UA 样本中随机抽选,不携带工具名称。Go 每次逻辑请求选择一次,重定向及 WebFetch 的代理失败直连重试复用同一个值;Python 脚本每次运行选择一次,入口、下载与重试保持一致。随机抽选允许重复,不依赖 AI 服务。通过 `ARTEX_HTTP_USER_AGENT` 可指定固定值;留空或非法值使用随机策略。直接运行时导出环境变量并重启服务,Docker 在 `.env` 配置后重建容器。Python 的 `--user-agent VALUE`、自定义工具和重放器中的显式 UA 优先,显式空值仍可省略此头。值去除首尾空格,仅接受可打印 ASCII。记录代理保留已有 UA 及显式省略语义。UA 随机化不代表 TLS、Cookie 或其他浏览器特征同步改变,也不保证无法识别客户端。此设置不影响浏览器、SDK 联网搜索、LLM / MCP / Webhook 服务通信或外部独立工具;本仓库未包含 phpfinger,其 `--user-agent` 需在该工具中单独配置。
完整排查范围、额外发现及未接管的服务通信见 [HTTP 请求标识排查](HTTP_USER_AGENT.md)。
**并发**:每个任务的 work agent 数在「系统设置」里配置(默认 3)。
**常用参数**:`./start.sh -addr :8787 -proxy :8788`(`-addr` 前端+API,`-proxy` 流量录制代理)。启动脚本会把参数原样透传给 `artex`。
+1 -1
View File
@@ -26,7 +26,7 @@ import (
// contiguous run is coalesced into a single record so the trace shows whole
// messages, not dozens of fragments.
func captureRun(ctx context.Context, opts agentcore.Options, input string, emit func(db.Activity)) (string, harness.TerminalReason, error) {
s := agentcore.NewSession(opts)
s := newAgentSession(opts)
defer s.Close() // release the session's background-task manager (temp dir + processes)
return captureRunSession(ctx, s, input, emit)
}
+1 -1
View File
@@ -173,7 +173,7 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
// 实验功能:开启后由 noa 接管上下文压缩(归档集中在 <workDir>/noa/<SessionID> 下,持久)。
enableNoa(&opts, c.noaEnabledFn, c.workDir, "chat-"+sessionID, noaWarn("chat-"+sessionID))
ctx = attachSideCapture(ctx, &opts)
s := agentcore.NewSession(opts)
s := newAgentSession(opts)
defer s.Close()
// reload prior conversation so the agent has context across turns (each Chat is
// a fresh session). First turn: no file yet → Resume loads nothing and proceeds.
+1 -1
View File
@@ -184,7 +184,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
}
enableNoa(&opts, m.noaEnabledFn, m.workDir, noaSession, noaWarn(noaSession))
ctx = attachSideCapture(ctx, &opts)
s := agentcore.NewSession(opts)
s := newAgentSession(opts)
defer s.Close()
// reload the prior conversation from the transcript so the agent has context
// across turns (each Chat is a fresh session; without this it can't see earlier
+2 -2
View File
@@ -21,9 +21,9 @@ const opsecPromptBlock = `【OPSEC 准则(硬约束,仅次于授权范围和
- 先复用已有流量、浏览器会话和已验证证据;能通过 traffic_search/traffic_get 或浏览器完成时,不要重复直接请求。
- 所有目标交互必须经过平台配置的记录代理或浏览器工具,确保请求可追溯。
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。命令行请求必须尽量复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;没有真实依据时不要伪造身份、设备或浏览器指纹。
- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。优先复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征;平台 HTTP 工具和采集模板的默认 UA 由系统随机策略提供,同一次请求的重定向、重试必须保持一致,明确配置的 UA 优先。
- 不得使用 -k、--ignore-https-errors、verify=False 等方式绕过证书校验;平台代理 CA 已配置时应正常验证。
- 不得随机轮换 User-Agent、批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向。
- 不得批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描;遇到限速、验证码、WAF 或异常响应时立即停止当前方向,不得通过更换 UA 继续当前方向。
- OPSEC 不扩大授权范围;目标、主机、端口、路径或账号不在授权范围内时不得访问。
- 请求和响应是否留痕由平台代理和浏览器负责;不要额外复制完整请求头、Cookie 或响应正文。`
+24
View File
@@ -0,0 +1,24 @@
package agent
import (
"github.com/Autumn-27/artex/internal/webfetch"
"github.com/Autumn-27/norma/agentcore"
actool "github.com/Autumn-27/norma/tool"
)
// newAgentSession installs the configurable WebFetch before the SDK builds the
// registry, so every agent role gets one tool and the same proxy/TLS settings.
func newAgentSession(opts agentcore.Options) *agentcore.Session {
if opts.EnableWebFetch {
opts.EnableWebFetch = false
if opts.Tools == nil {
opts.Tools = actool.DefaultTools()
}
opts.Tools = append(opts.Tools, webfetch.New(actool.WebFetchConfig{
Proxy: opts.WebFetchProxy,
CACert: opts.WebFetchCACert,
InsecureTLS: opts.WebFetchInsecureTLS,
}))
}
return agentcore.NewSession(opts)
}
+70
View File
@@ -0,0 +1,70 @@
package agent
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/Autumn-27/norma/agentcore"
"github.com/Autumn-27/norma/llm"
"github.com/Autumn-27/norma/permission"
)
type fetchFixtureProvider struct {
autoTitleProvider
t *testing.T
url string
calls int
}
func (p *fetchFixtureProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
p.calls++
if p.calls == 1 {
counts := map[string]int{}
for _, tool := range req.Tools {
counts[tool.Name]++
}
if counts["WebFetch"] != 1 || counts["Read"] != 1 || counts["Bash"] != 1 {
p.t.Errorf("session tool registration = %v", counts)
}
input, _ := json.Marshal(map[string]string{"url": p.url})
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{{Type: llm.BlockToolUse, ID: "fetch-fixture", Name: "WebFetch", Input: input}}}, "tool_use", llm.Usage{}, nil
}
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("finished")}}, "end_turn", llm.Usage{}, nil
}
func TestAgentSessionUsesConfiguredWebFetch(t *testing.T) {
for _, ua := range []string{"", "Captured-UA/1.0"} {
t.Run(ua, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if ua == "" && !strings.HasPrefix(r.UserAgent(), "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", r.UserAgent())
} else if ua != "" && r.UserAgent() != ua {
t.Errorf("session sent User-Agent %q, want %q", r.UserAgent(), ua)
}
w.Write([]byte("fixture"))
}))
defer server.Close()
provider := &fetchFixtureProvider{t: t, url: server.URL}
session := newAgentSession(agentcore.Options{
Provider: provider, NonStreaming: true, EnableWebFetch: true,
PermissionMode: permission.ModeBypass, MaxTurns: 3,
})
defer session.Close()
for _, err := range session.Prompt(t.Context(), "fetch local fixture") {
if err != nil {
t.Fatal(err)
}
}
if calls != 1 {
t.Fatalf("WebFetch calls = %d, want 1", calls)
}
})
}
}
+1 -1
View File
@@ -488,7 +488,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
noaSession := WorkerSessionID(ts.ID(), intent.ID)
enableNoa(&opts, w.noaEnabledFn, w.workDir, noaSession, noaWarn(noaSession))
ctx = attachSideCapture(ctx, &opts)
s := agentcore.NewSession(opts)
s := newAgentSession(opts)
defer s.Close() // release the session's background-task manager (temp dir + processes)
// Resume prior conversation if this intent was paused/blocked/exhausted and is
+38
View File
@@ -102,3 +102,41 @@ func TestDeleteConversationsReturnsExistingIDs(t *testing.T) {
t.Fatalf("deleted=%v, want existing ids only", deleted)
}
}
func TestTaskPatchWorkerCountPersistsAndValidates(t *testing.T) {
d, err := Open(testDSN(t))
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
}
defer d.Close()
task, err := d.CreateTask("worker count patch", "goal", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
defer func() { _ = d.DeleteTask(task.ID) }()
for _, count := range []int{1, MaxTaskWorkerCount, 0, 4} {
updated, err := d.UpdateTask(task.ID, TaskPatch{WorkerCount: &count})
if err != nil || updated == nil || updated.WorkerCount != count {
t.Fatalf("patch worker_count=%d task=%+v err=%v", count, updated, err)
}
persisted, err := d.GetTask(task.ID)
if err != nil || persisted == nil || persisted.WorkerCount != count {
t.Fatalf("read worker_count=%d task=%+v err=%v", count, persisted, err)
}
}
name := "renamed without worker override"
updated, err := d.UpdateTask(task.ID, TaskPatch{Name: &name})
if err != nil || updated == nil || updated.WorkerCount != 4 {
t.Fatalf("omitted worker_count changed override: task=%+v err=%v", updated, err)
}
for _, count := range []int{-1, MaxTaskWorkerCount + 1} {
invalidName := "must not commit"
if updated, err := d.UpdateTask(task.ID, TaskPatch{Name: &invalidName, WorkerCount: &count}); err == nil || updated != nil {
t.Fatalf("accepted invalid worker_count=%d: task=%+v err=%v", count, updated, err)
}
persisted, err := d.GetTask(task.ID)
if err != nil || persisted == nil || persisted.WorkerCount != 4 || persisted.Name != name {
t.Fatalf("invalid patch changed task=%+v err=%v", persisted, err)
}
}
}
+14 -4
View File
@@ -165,6 +165,9 @@ func normalizeWorkerCount(n int) int {
}
func validateWorkerCount(n int) error {
if n < 0 {
return fmt.Errorf("Worker 运行数量不能为负数")
}
if n > MaxTaskWorkerCount {
return fmt.Errorf("Worker 运行数量最多 %d", MaxTaskWorkerCount)
}
@@ -471,17 +474,23 @@ ORDER BY (pinned_at IS NOT NULL) DESC, pinned_at DESC NULLS LAST, id DESC`)
return out, nil
}
// TaskPatch updates editable metadata and global asset visibility.
// TaskPatch updates editable metadata, asset visibility, and worker concurrency.
type TaskPatch struct {
Name *string
Remark *string
Pinned *bool
ShareGlobalAssets *bool
WorkerCount *int
}
// UpdateTask applies a partial metadata/visibility mutation and returns the updated row.
// UpdateTask applies a partial metadata/runtime-policy mutation and returns the updated row.
// Re-pinning an already pinned task preserves its original position.
func (d *DB) UpdateTask(id int64, patch TaskPatch) (*Task, error) {
if patch.WorkerCount != nil {
if err := validateWorkerCount(*patch.WorkerCount); err != nil {
return nil, err
}
}
task, err := scanTask(d.QueryRow(`UPDATE tasks SET
name = CASE WHEN $2::boolean THEN $3 ELSE name END,
remark = CASE WHEN $5::boolean THEN $6 ELSE remark END,
@@ -490,9 +499,10 @@ func (d *DB) UpdateTask(id int64, patch TaskPatch) (*Task, error) {
WHEN $4::boolean THEN COALESCE(pinned_at, now())
ELSE NULL
END,
share_global_assets = COALESCE($7::boolean, share_global_assets)
share_global_assets = COALESCE($7::boolean, share_global_assets),
worker_count = COALESCE($8::integer, worker_count)
WHERE id=$1 AND deleted_at IS NULL
RETURNING `+taskCols, id, patch.Name != nil, patch.Name, patch.Pinned, patch.Remark != nil, patch.Remark, patch.ShareGlobalAssets))
RETURNING `+taskCols, id, patch.Name != nil, patch.Name, patch.Pinned, patch.Remark != nil, patch.Remark, patch.ShareGlobalAssets, patch.WorkerCount))
if err == sql.ErrNoRows {
return nil, nil
}
+1
View File
@@ -28,6 +28,7 @@ services:
ARTEX_LLM_MODEL: ${ARTEX_LLM_MODEL:-}
ARTEX_LLM_BASE_URL: ${ARTEX_LLM_BASE_URL:-}
ARTEX_LLM_PROXY: ${ARTEX_LLM_PROXY:-}
ARTEX_HTTP_USER_AGENT: ${ARTEX_HTTP_USER_AGENT:-}
ports:
- "8787:8787"
- "127.0.0.1:8788:8788"
+2 -1
View File
@@ -20,6 +20,7 @@ import (
"time"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/internal/httpidentity"
"github.com/miekg/dns"
"github.com/projectdiscovery/dnsx/libs/dnsx"
@@ -230,7 +231,7 @@ func (e *Engine) doHTTP(id int64, rawURL string) {
if err != nil {
return
}
req.Header.Set("User-Agent", "artex-enrich/1.0")
httpidentity.Apply(req)
resp, err := e.client.Do(req)
if err != nil {
return
+38
View File
@@ -0,0 +1,38 @@
package enrich
import (
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
// Stop after receiving the probe so this request regression needs no database.
func TestProbeUserAgent(t *testing.T) {
for _, ua := range []string{"", "Captured-UA/1.0"} {
t.Run(ua, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", got)
} else if ua != "" && got != ua {
t.Errorf("User-Agent = %q, want %q", got, ua)
}
http.Redirect(w, r, "/stop", http.StatusFound)
}))
defer server.Close()
client := buildClient(nil)
client.CheckRedirect = func(*http.Request, []*http.Request) error {
return errors.New("probe captured")
}
e := &Engine{client: client}
e.doHTTP(1, server.URL)
if calls != 1 {
t.Fatalf("probe calls = %d, want 1", calls)
}
})
}
}
+9 -9
View File
@@ -3,19 +3,19 @@ module github.com/Autumn-27/artex
go 1.26.10
require (
mvdan.cc/sh/v3 v3.10.0
github.com/Autumn-27/norma v0.4.3
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.10.0
github.com/klauspost/compress v1.18.6
github.com/lqqyt2423/go-mitmproxy v1.9.2
github.com/lqqyt2423/go-mitmproxy v1.9.5
github.com/miekg/dns v1.1.72
github.com/projectdiscovery/dnsx v1.2.3
golang.org/x/crypto v0.55.0
golang.org/x/net v0.58.0
golang.org/x/text v0.41.0
golang.org/x/crypto v0.58.0
golang.org/x/net v0.61.0
golang.org/x/text v0.43.0
modernc.org/sqlite v1.52.0
mvdan.cc/sh/v3 v3.10.0
)
require (
@@ -52,10 +52,10 @@ require (
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
golang.org/x/mod v0.40.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/sys v0.47.0 // indirect
golang.org/x/tools v0.49.0 // indirect
golang.org/x/mod v0.41.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/tools v0.51.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.72.3 // indirect
modernc.org/mathutil v1.7.1 // indirect
+16 -2
View File
@@ -48,8 +48,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/lqqyt2423/go-mitmproxy v1.9.2 h1:+QTbDyUUaVRmEavPq0/eWzJ2K+K4JdL4ff91s/adnxk=
github.com/lqqyt2423/go-mitmproxy v1.9.2/go.mod h1:6H5PR+nbVsiGQJC/Y98GHJItyFcmP/Z0txAgjhG2a80=
github.com/lqqyt2423/go-mitmproxy v1.9.5 h1:4ar1qWYlbXF+7HhG06AVjKkDILXX1FS0mNy+hkX4uX4=
github.com/lqqyt2423/go-mitmproxy v1.9.5/go.mod h1:74Z1IyzuUu/pq1IpdDd555f1xXM7Gl9KtUhWn/mzU7E=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
@@ -108,21 +108,35 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=
golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
+50 -9
View File
@@ -572,8 +572,9 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
if err != nil || len(rules) == 0 {
return Decision{}, false
}
values := inputStrings(input)
for _, r := range rules {
if ruleMatches(r, toolName, input) {
if ruleMatches(r, toolName, input, values) {
msg := r.Message
if msg == "" {
msg = defaultMessage(r.Action, r.Name)
@@ -593,20 +594,60 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
return Decision{}, false
}
func ruleMatches(r compiledRule, toolName string, input []byte) bool {
var subject string
// ruleMatches reports whether r applies to the call. tool_input rules are tried
// against the raw JSON first, then against each decoded string value in values.
// The raw form alone is not enough: JSON escapes newlines, tabs and quotes, and
// Go's encoder also escapes & < >, so a rule like `\brm\b` never fires on
// "cd /tmp\nrm -rf /var/www" (the raw bytes read `\nrm`, with no word boundary).
func ruleMatches(r compiledRule, toolName string, input []byte, values []string) bool {
switch r.MatchTarget {
case "tool_name":
subject = toolName
return r.matchString(toolName)
case "tool_input":
subject = string(input)
default:
return false
if r.matchString(string(input)) {
return true
}
for _, v := range values {
if r.matchString(v) {
return true
}
}
}
return false
}
func (r compiledRule) matchString(s string) bool {
if r.MatchType == "regex" {
return r.re != nil && r.re.MatchString(subject)
return r.re != nil && r.re.MatchString(s)
}
return strings.Contains(subject, r.Pattern)
return strings.Contains(s, r.Pattern)
}
// inputStrings returns every string value in a JSON tool input, at any depth.
// Object keys are skipped. It returns nil if the input is not valid JSON.
func inputStrings(input []byte) []string {
var doc any
if json.Unmarshal(input, &doc) != nil {
return nil
}
var out []string
var walk func(any)
walk = func(v any) {
switch v := v.(type) {
case string:
out = append(out, v)
case []any:
for _, e := range v {
walk(e)
}
case map[string]any:
for _, e := range v {
walk(e)
}
}
}
walk(doc)
return out
}
func defaultMessage(action, name string) string {
+138
View File
@@ -0,0 +1,138 @@
package intercept
import (
"encoding/json"
"regexp"
"testing"
"github.com/Autumn-27/artex/db"
)
// Patterns copied from the built-in rules seeded in db/db.go.
const (
builtinRmRf = `(?i)\brm\b.{0,80}(?:-[a-z]*r[a-z]*f[a-z]*|-[a-z]*f[a-z]*r[a-z]*|--recursive|--no-preserve-root)`
builtinMkfs = `\bmkfs\b`
builtinDrop = `(?i)\bDROP\s+(?:DATABASE|TABLE|SCHEMA|INDEX|VIEW|TABLESPACE|USER|ROLE)\b`
builtinFlush = `(?i)\b(?:FLUSHALL|FLUSHDB)\b`
builtinV2 = `(?i)\b(rm\s+-rf\s+/|mkfs|dd\s+if=|:\(\)\s*\{|shutdown|reboot|>\s*/dev/sd)`
)
func testRule(target, matchType, pattern string) compiledRule {
r := compiledRule{InterceptRule: db.InterceptRule{
ID: 1, Name: "test", Enabled: true, Action: "deny",
MatchTarget: target, MatchType: matchType, Pattern: pattern,
}}
if matchType == "regex" {
r.re = regexp.MustCompile(pattern)
}
return r
}
// interceptorWith returns an Interceptor whose rule cache is pre-filled, so
// Match runs without a database.
func interceptorWith(rules ...compiledRule) *Interceptor {
return &Interceptor{cached: rules}
}
func toolInput(t *testing.T, v any) []byte {
t.Helper()
b, err := json.Marshal(v)
if err != nil {
t.Fatal(err)
}
return b
}
func TestMatchToolInputSeesDecodedValues(t *testing.T) {
cases := []struct {
name string
pattern string
tool string
input any
}{
{"rm -rf on its own line", builtinRmRf, "Bash", map[string]string{"command": "cd /tmp\nrm -rf /var/www"}},
{"rm -rf after a tab", builtinRmRf, "Bash", map[string]string{"command": "echo done\trm -rf /var/www"}},
{"mkfs on its own line", builtinMkfs, "Bash", map[string]string{"command": "lsblk\nmkfs.ext4 /dev/sdb"}},
{"DROP inside a heredoc", builtinDrop, "Bash", map[string]string{"command": "psql <<EOF\nDROP TABLE users;\nEOF"}},
{"FLUSHALL inside a heredoc", builtinFlush, "Bash", map[string]string{"command": "redis-cli <<EOF\nFLUSHALL\nEOF"}},
{"shutdown on its own line", builtinV2, "Bash", map[string]string{"command": "sync\nshutdown -h now"}},
// encoding/json writes > as >, so this only matches the decoded value.
{"redirect to a disk", `>\s*/dev/sd`, "Bash", map[string]string{"command": "cat img > /dev/sda"}},
{"interactive shell text", builtinRmRf, "shell_send", map[string]string{"text": "ls\nrm -rf /srv/data\n"}},
{"nested edit value", builtinDrop, "MultiEdit", map[string]any{
"file_path": "/tmp/m.sql",
"edits": []map[string]string{{"old_string": "-- x", "new_string": "-- x\nDROP TABLE users;"}},
}},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
ic := interceptorWith(testRule("tool_input", "regex", tc.pattern))
in := toolInput(t, tc.input)
if _, ok := ic.Match(tc.tool, in); !ok {
t.Fatalf("rule %q did not match input %s", tc.pattern, in)
}
})
}
}
func TestMatchToolInputStringRuleSeesDecodedValues(t *testing.T) {
// The raw JSON holds rm -rf \"/var/www\", so the quotes only line up once decoded.
ic := interceptorWith(testRule("tool_input", "string", `rm -rf "/var/www"`))
in := toolInput(t, map[string]string{"command": `rm -rf "/var/www"`})
if _, ok := ic.Match("Bash", in); !ok {
t.Fatalf("string rule did not match input %s", in)
}
}
// Rules written against the raw JSON form must keep working.
func TestMatchToolInputStillMatchesRawJSON(t *testing.T) {
ic := interceptorWith(testRule("tool_input", "regex", `"command":"rm `))
in := toolInput(t, map[string]string{"command": "rm notes.txt"})
if _, ok := ic.Match("Bash", in); !ok {
t.Fatalf("raw JSON rule did not match input %s", in)
}
}
func TestMatchToolInputInvalidJSONFallsBackToRaw(t *testing.T) {
ic := interceptorWith(testRule("tool_input", "regex", builtinMkfs))
if _, ok := ic.Match("Bash", []byte(`not json: mkfs /dev/sdb`)); !ok {
t.Fatal("rule did not match non-JSON input")
}
}
func TestMatchToolInputNoFalsePositives(t *testing.T) {
ic := interceptorWith(
testRule("tool_input", "regex", builtinRmRf),
testRule("tool_input", "regex", builtinMkfs),
testRule("tool_input", "regex", builtinDrop),
)
for _, cmd := range []string{
"cd /tmp\nls -la",
"nmap -sV 10.0.0.5\ncurl -s http://10.0.0.5/",
"echo firm\nrmdir build", // "firm" and "rmdir" must not look like rm
} {
in := toolInput(t, map[string]string{"command": cmd})
if dec, ok := ic.Match("Bash", in); ok {
t.Fatalf("unexpected match %q for input %s", dec.RuleName, in)
}
}
}
// The match key ("command") must not be matched as a value.
func TestMatchToolInputIgnoresObjectKeys(t *testing.T) {
ic := interceptorWith(testRule("tool_input", "regex", `^mkfs$`))
in := toolInput(t, map[string]string{"mkfs": "ls"})
if _, ok := ic.Match("Bash", in); ok {
t.Fatalf("object key matched as a value for input %s", in)
}
}
func TestMatchToolNameUnchanged(t *testing.T) {
ic := interceptorWith(testRule("tool_name", "string", "WebFetch"))
if _, ok := ic.Match("WebFetch", []byte(`{"url":"http://x"}`)); !ok {
t.Fatal("tool_name rule did not match")
}
if _, ok := ic.Match("Bash", []byte(`{"command":"WebFetch"}`)); ok {
t.Fatal("tool_name rule matched tool input")
}
}
+67
View File
@@ -0,0 +1,67 @@
// Package httpidentity defines the explicit identity for target-facing HTTP requests.
package httpidentity
import (
"crypto/rand"
"io"
"math/big"
"net/http"
"os"
"strings"
)
// Keep these 20 standard desktop browser UA samples in sync with the standalone
// Python templates. Reviewed 2026-11-11; see HTTP_USER_AGENT.md for versions and
// vendor format references. Keep normal reduced platform/version fields and no
// tool identifiers, device-specific details or custom suffixes.
var userAgents = [...]string{
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0",
"Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15",
}
// UserAgent returns the configured printable ASCII value, or a random browser
// sample when unset, empty or malformed. Call once per logical request and reuse
// the selected value for redirects and retries.
func UserAgent() string {
value := strings.Trim(os.Getenv("ARTEX_HTTP_USER_AGENT"), " ")
for _, c := range value {
if c < 32 || c > 126 {
return randomUserAgent(rand.Reader)
}
}
if value != "" {
return value
}
return randomUserAgent(rand.Reader)
}
func randomUserAgent(source io.Reader) string {
index, err := rand.Int(source, big.NewInt(int64(len(userAgents))))
if err != nil {
return userAgents[0] // Keep a valid nonempty UA if system entropy fails.
}
return userAgents[index.Int64()]
}
// Apply selects an identity for this request; net/http preserves it on redirects.
func Apply(req *http.Request) {
req.Header.Set("User-Agent", UserAgent())
}
+65
View File
@@ -0,0 +1,65 @@
package httpidentity
import (
"bytes"
"net/http"
"net/http/httptest"
"slices"
"testing"
)
func TestApplyOnWire(t *testing.T) {
for _, tc := range []struct{ name, value, want string }{
{"empty", "", ""},
{"configured", " Captured-UA/1.0 ", "Captured-UA/1.0"},
{"whitespace", " ", ""},
{"injection", "ok\r\nX-Injected: yes", ""},
{"non-ascii", "客户端", ""},
} {
t.Run(tc.name, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.value)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
got := r.UserAgent()
if tc.want == "" && !slices.Contains(userAgents[:], got) {
t.Errorf("default User-Agent is not a browser sample: %q", got)
} else if tc.want != "" && got != tc.want {
t.Errorf("User-Agent = %q, want %q", got, tc.want)
}
if r.Header.Get("X-Injected") != "" {
t.Error("header injection")
}
}))
defer server.Close()
req, err := http.NewRequest(http.MethodGet, server.URL, nil)
if err != nil {
t.Fatal(err)
}
Apply(req)
resp, err := server.Client().Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
})
}
}
func TestRandomUserAgentSelection(t *testing.T) {
if len(userAgents) != 20 {
t.Fatalf("pool size = %d, want 20", len(userAgents))
}
seen := make(map[string]bool, len(userAgents))
// Exercise every choice deterministically without a statistical/flaky test.
for index, want := range userAgents {
if seen[want] {
t.Errorf("duplicate sample: %q", want)
}
seen[want] = true
if got := randomUserAgent(bytes.NewReader([]byte{byte(index)})); got != want {
t.Errorf("choice %d = %q, want %q", index, got, want)
}
}
if got := randomUserAgent(bytes.NewReader(nil)); got != userAgents[0] {
t.Errorf("entropy failure fallback = %q", got)
}
}
+467
View File
@@ -0,0 +1,467 @@
// Adapted from github.com/Autumn-27/norma/tool/webfetch.go at v0.4.3.
// Keep the SDK tool metadata and fetch/render behavior; use ARTEX's explicit
// HTTP identity for both the proxy attempt and its direct fallback. This local
// adapter is needed until the SDK exposes request headers in actool.WebFetchConfig.
package webfetch
import (
"context"
"crypto/tls"
"crypto/x509"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"os"
"regexp"
"strings"
"time"
"github.com/Autumn-27/artex/internal/httpidentity"
actool "github.com/Autumn-27/norma/tool"
"golang.org/x/net/html"
)
// New keeps the SDK's schema, permission checks and tool metadata.
func New(cfg actool.WebFetchConfig) actool.CoreTool {
base := actool.NewWebFetch(cfg)
return actool.Build(actool.Spec{
Name: base.Name(), Description: base.Description(), Prompt: base.Prompt(),
Schema: base.InputSchema(), Permissions: base.CheckPermissions,
ReadOnly: base.IsReadOnly, Concurrent: base.IsConcurrencySafe,
Run: webFetchRunner(cfg),
})
}
const maxFetchBytes = 2 << 20 // 2 MiB
// newFetchClient returns an HTTP client honoring the proxy / TLS config.
func newFetchClient(cfg actool.WebFetchConfig) *http.Client {
tr := &http.Transport{}
if cfg.Proxy != "" {
if pu, err := url.Parse(cfg.Proxy); err == nil {
tr.Proxy = http.ProxyURL(pu)
}
}
switch {
case cfg.InsecureTLS:
tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
case cfg.CACert != "":
// Trust the proxy's CA in addition to the system roots, so certs it
// re-signs verify while every other connection stays fully checked.
if pem, err := os.ReadFile(cfg.CACert); err == nil {
pool, _ := x509.SystemCertPool()
if pool == nil {
pool = x509.NewCertPool()
}
if pool.AppendCertsFromPEM(pem) {
tr.TLSClientConfig = &tls.Config{RootCAs: pool}
}
}
}
return &http.Client{Transport: tr}
}
// proxyLikelyErr reports whether a fetch error is the kind a recording/MITM proxy
// introduces — a closed tunnel (EOF / connection reset), a failed CONNECT, or a
// TLS/HTTP2 mismatch — so a direct (no-proxy) retry may succeed. Name-resolution
// failures and caller cancellation/timeout are target- or caller-side problems a
// direct retry won't fix, so those return false (no retry).
func proxyLikelyErr(err error) bool {
if err == nil {
return false
}
s := strings.ToLower(err.Error())
for _, n := range []string{"no such host", "context canceled", "context deadline", "timeout"} {
if strings.Contains(s, n) {
return false
}
}
for _, p := range []string{
"eof", "connection reset", "connection refused", "proxyconnect",
"broken pipe", "forcibly closed", "malformed", "http2", "http/2",
"tls:", "protocol error",
} {
if strings.Contains(s, p) {
return true
}
}
return false
}
// staticAssetExts are file extensions WebFetch refuses: linked resources (scripts,
// styles, source maps, images, fonts, media, wasm) that are not page content.
var staticAssetExts = map[string]bool{
".js": true, ".mjs": true, ".cjs": true, ".css": true, ".map": true,
".png": true, ".jpg": true, ".jpeg": true, ".gif": true, ".svg": true,
".webp": true, ".avif": true, ".ico": true, ".bmp": true,
".woff": true, ".woff2": true, ".ttf": true, ".otf": true, ".eot": true,
".wasm": true, ".mp4": true, ".webm": true, ".mp3": true, ".wav": true, ".ogg": true,
}
// staticAssetExt returns the offending extension (with dot) if rawURL points at a
// static asset, or "" otherwise. The query string and fragment are ignored, so
// "/app.js?v=3" is still caught.
func staticAssetExt(rawURL string) string {
path := rawURL
if u, err := url.Parse(rawURL); err == nil {
path = u.Path
}
if i := strings.LastIndexByte(path, '/'); i >= 0 {
path = path[i+1:] // last segment only
}
dot := strings.LastIndexByte(path, '.')
if dot < 0 {
return ""
}
if ext := strings.ToLower(path[dot:]); staticAssetExts[ext] {
return ext
}
return ""
}
func webFetchRunner(cfg actool.WebFetchConfig) func(context.Context, json.RawMessage, *actool.ToolContext) (actool.Result, error) {
client := newFetchClient(cfg)
// direct is the no-proxy fallback used when a proxied fetch fails for a
// proxy-caused reason (closed tunnel / failed CONNECT / TLS or HTTP2 mismatch):
// it bypasses the configured proxy so a recording/MITM hiccup doesn't fail an
// otherwise-reachable URL. nil when no proxy is configured (nothing to bypass).
var direct *http.Client
if cfg.Proxy != "" {
direct = newFetchClient(actool.WebFetchConfig{InsecureTLS: cfg.InsecureTLS})
}
return func(ctx context.Context, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
var in struct {
URL string `json:"url"`
Extract bool `json:"extract"`
}
if err := json.Unmarshal(input, &in); err != nil {
return actool.Result{}, err
}
if !strings.HasPrefix(in.URL, "http://") && !strings.HasPrefix(in.URL, "https://") {
return actool.Errorf("Error: url must start with http:// or https://"), nil
}
if ext := staticAssetExt(in.URL); ext != "" {
return actool.Errorf("Error: WebFetch does not fetch static assets (" + ext + " file). It reads HTML pages, documentation, and API responses — not linked resources like scripts, styles, images, or fonts. Fetch the page that references this asset instead."), nil
}
cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil)
if err != nil {
return actool.Errorf("Error: " + err.Error()), nil
}
userAgent := httpidentity.UserAgent()
req.Header.Set("User-Agent", userAgent)
resp, err := client.Do(req)
// One direct retry when the proxied attempt failed for a proxy-caused reason.
// A non-proxy error (bad host, caller timeout/cancel) is returned as-is — a
// direct retry wouldn't fix it. The direct response is NOT captured by the proxy.
if err != nil && direct != nil && proxyLikelyErr(err) {
if r2, e2 := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil); e2 == nil {
r2.Header.Set("User-Agent", userAgent)
if resp2, err2 := direct.Do(r2); err2 == nil {
resp, err = resp2, nil
} else {
err = err2 // surface the direct failure (more representative of the target)
}
}
}
if err != nil {
return actool.Errorf("Error fetching URL: " + err.Error()), nil
}
defer resp.Body.Close()
body, _ := io.ReadAll(io.LimitReader(resp.Body, maxFetchBytes))
header := fmt.Sprintf("[%d %s] %s\n", resp.StatusCode, http.StatusText(resp.StatusCode), in.URL)
// Cheap fingerprint signal that HTML→Markdown would otherwise hide.
if sv := resp.Header.Get("Server"); sv != "" {
header += "Server: " + sv + "\n"
}
if xp := resp.Header.Get("X-Powered-By"); xp != "" {
header += "X-Powered-By: " + xp + "\n"
}
var out string
if strings.Contains(strings.ToLower(resp.Header.Get("content-type")), "html") {
doc, perr := html.Parse(strings.NewReader(string(body)))
if perr != nil { // html.Parse is lenient; this is effectively unreachable
out = string(body)
} else {
out = htmlToMarkdown(doc)
if in.Extract {
if recon := extractRecon(doc); recon != "" {
out += "\n\n" + recon
}
}
}
} else {
out = string(body)
}
return actool.Text(truncate(header+strings.TrimSpace(out), 50000)), nil
}
}
var (
reWS = regexp.MustCompile(`[ \t]+`)
reLineTrim = regexp.MustCompile(` *\n *`) // strip spaces hugging a newline
reBlankLines = regexp.MustCompile(`\n{2,}`) // collapse blank lines → one newline
// reEndpoint matches quoted absolute URLs and root-relative paths in inline JS.
reEndpoint = regexp.MustCompile("[\"'`](https?://[^\"'`\\s]+|/[A-Za-z0-9._~$&'()*+,;=:@%/?#\\[\\]-]{2,})[\"'`]")
)
// skipMarkdown are elements whose subtree carries no readable prose.
var skipMarkdown = map[string]bool{"script": true, "style": true, "noscript": true, "svg": true, "template": true}
// htmlToMarkdown renders the visible content of a parsed document as Markdown.
// Text nodes are already entity-decoded by the parser, so no unescaping is needed.
func htmlToMarkdown(doc *html.Node) string {
var b strings.Builder
renderNode(&b, doc)
s := reWS.ReplaceAllString(b.String(), " ")
s = reLineTrim.ReplaceAllString(s, "\n") // drop indentation / trailing spaces per line
s = reBlankLines.ReplaceAllString(s, "\n") // no blank lines — compact, token-lean output
return strings.TrimSpace(s)
}
func renderNode(b *strings.Builder, n *html.Node) {
switch n.Type {
case html.TextNode:
b.WriteString(strings.ReplaceAll(n.Data, "\n", " "))
return
case html.ElementNode:
if skipMarkdown[n.Data] {
return
}
default:
if n.Type != html.DocumentNode {
return
}
}
tag := ""
if n.Type == html.ElementNode {
tag = n.Data
}
switch tag {
case "h1", "h2", "h3", "h4", "h5", "h6":
b.WriteString("\n\n" + strings.Repeat("#", int(tag[1]-'0')) + " ")
renderChildren(b, n)
b.WriteString("\n\n")
case "title":
b.WriteString("\n\n# ")
renderChildren(b, n)
b.WriteString("\n\n")
case "p", "div", "section", "article", "header", "footer", "main", "ul", "ol", "table", "tr", "blockquote":
b.WriteString("\n\n")
renderChildren(b, n)
b.WriteString("\n\n")
case "br":
b.WriteString("\n")
case "hr":
b.WriteString("\n\n---\n\n")
case "li":
b.WriteString("\n- ")
renderChildren(b, n)
case "a":
var inner strings.Builder
renderChildren(&inner, n)
text := strings.TrimSpace(inner.String())
if href := attr(n, "href"); href != "" && text != "" {
fmt.Fprintf(b, "[%s](%s)", text, href)
} else {
b.WriteString(text)
}
case "img":
if src := attr(n, "src"); src != "" {
fmt.Fprintf(b, "![%s](%s)", attr(n, "alt"), src)
}
case "strong", "b":
b.WriteString("**")
renderChildren(b, n)
b.WriteString("**")
case "em", "i":
b.WriteString("*")
renderChildren(b, n)
b.WriteString("*")
case "code":
b.WriteString("`")
renderChildren(b, n)
b.WriteString("`")
case "pre":
var inner strings.Builder
textOnly(&inner, n)
b.WriteString("\n\n```\n" + strings.TrimSpace(inner.String()) + "\n```\n\n")
default:
renderChildren(b, n)
}
}
func renderChildren(b *strings.Builder, n *html.Node) {
for c := n.FirstChild; c != nil; c = c.NextSibling {
renderNode(b, c)
}
}
// extractRecon walks the document for security-relevant signal that Markdown
// rendering drops: comments, script sources, inline endpoints, forms, meta, links.
func extractRecon(doc *html.Node) string {
var comments, scriptSrcs, inlineEndpoints, forms, metas, links []string
seenLink := map[string]bool{}
var walk func(n *html.Node)
walk = func(n *html.Node) {
switch n.Type {
case html.CommentNode:
if c := oneLine(n.Data, 300); c != "" {
comments = append(comments, c)
}
case html.ElementNode:
switch n.Data {
case "script":
if src := attr(n, "src"); src != "" {
scriptSrcs = append(scriptSrcs, src)
} else {
var body strings.Builder
textOnly(&body, n)
for _, m := range reEndpoint.FindAllStringSubmatch(body.String(), 40) {
inlineEndpoints = append(inlineEndpoints, m[1])
}
}
case "form":
forms = append(forms, describeForm(n))
case "meta":
name := attr(n, "name")
if name == "" {
name = attr(n, "property")
}
if content := attr(n, "content"); name != "" && content != "" {
metas = append(metas, name+"="+oneLine(content, 120))
}
case "a", "link":
if h := attr(n, "href"); h != "" && !seenLink[h] {
seenLink[h] = true
links = append(links, h)
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(doc)
var b strings.Builder
b.WriteString("=== RECON ===")
reconSection(&b, "COMMENTS", comments)
reconSection(&b, "SCRIPTS (src)", dedup(scriptSrcs))
reconSection(&b, "INLINE ENDPOINTS", dedup(inlineEndpoints))
reconSection(&b, "FORMS", forms)
reconSection(&b, "META", metas)
reconSection(&b, "LINKS", capSlice(links, 100))
if b.Len() == len("=== RECON ===") {
return "" // nothing worth reporting
}
return b.String()
}
// describeForm summarizes a <form> as "METHOD action → name(type[=value]) …".
// Hidden inputs keep their value (CSRF tokens, parameter defaults matter in recon).
func describeForm(form *html.Node) string {
method := strings.ToUpper(attr(form, "method"))
if method == "" {
method = "GET"
}
var inputs []string
var walk func(n *html.Node)
walk = func(n *html.Node) {
if n.Type == html.ElementNode {
switch n.Data {
case "input", "textarea", "select":
typ := attr(n, "type")
if typ == "" {
typ = n.Data
}
desc := attr(n, "name") + "(" + typ
if typ == "hidden" {
if v := attr(n, "value"); v != "" {
desc += "=" + oneLine(v, 60)
}
}
inputs = append(inputs, desc+")")
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
walk(c)
}
}
walk(form)
return fmt.Sprintf("%s %s → %s", method, attr(form, "action"), strings.Join(inputs, " "))
}
// --- small helpers ---
func attr(n *html.Node, key string) string {
for _, a := range n.Attr {
if a.Key == key {
return a.Val
}
}
return ""
}
func textOnly(b *strings.Builder, n *html.Node) {
if n.Type == html.TextNode {
b.WriteString(n.Data)
return
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
textOnly(b, c)
}
}
func oneLine(s string, max int) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) > max {
return s[:max] + "…"
}
return s
}
func reconSection(b *strings.Builder, title string, items []string) {
if len(items) == 0 {
return
}
b.WriteString("\n\n## " + title + "\n")
for _, it := range items {
b.WriteString("- " + it + "\n")
}
}
func dedup(in []string) []string {
seen := map[string]bool{}
out := in[:0:0]
for _, s := range in {
if s == "" || seen[s] {
continue
}
seen[s] = true
out = append(out, s)
}
return out
}
func capSlice(in []string, n int) []string {
if len(in) <= n {
return in
}
return append(in[:n:n], fmt.Sprintf("… (%d more)", len(in)-n))
}
// truncate retains the SDK's head/tail output limit.
func truncate(s string, max int) string {
if max <= 0 || len(s) <= max {
return s
}
half := max / 2
return s[:half] + fmt.Sprintf("\n\n... [%d characters truncated] ...\n\n", len(s)-max) + s[len(s)-half:]
}
+175
View File
@@ -0,0 +1,175 @@
package webfetch
import (
"context"
"encoding/json"
"encoding/pem"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"sync/atomic"
"testing"
actool "github.com/Autumn-27/norma/tool"
)
func callFetch(t *testing.T, tool actool.CoreTool, target string, extract bool) actool.Result {
t.Helper()
input, err := json.Marshal(map[string]any{"url": target, "extract": extract})
if err != nil {
t.Fatal(err)
}
result, err := tool.Call(context.Background(), input, nil)
if err != nil {
t.Fatal(err)
}
return result
}
func TestIdentityAcrossRoutes(t *testing.T) {
for _, ua := range []string{"", "Captured-UA/1.0"} {
for _, route := range []string{"direct", "proxy", "fallback"} {
t.Run(route+"/"+ua, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
var targetCalls, proxyCalls atomic.Int32
var mu sync.Mutex
var selected string
checkUA := func(r *http.Request) {
mu.Lock()
defer mu.Unlock()
got := r.UserAgent()
if ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", got)
} else if ua != "" && got != ua {
t.Errorf("User-Agent = %q, want %q", got, ua)
}
if selected != "" && selected != got {
t.Errorf("redirect/retry UA changed from %q to %q", selected, got)
}
selected = got
}
target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
targetCalls.Add(1)
checkUA(r)
if r.URL.Path == "/start" {
http.Redirect(w, r, "/page", http.StatusFound)
return
}
w.Header().Set("Content-Type", "text/html")
fmt.Fprint(w, `<h1>Page</h1><!--keep--><script src="/app.js"></script><form action="/submit"><input name="csrf" type="hidden" value="token"></form>`)
}))
defer target.Close()
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxyCalls.Add(1)
checkUA(r)
if route == "fallback" {
conn, _, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Error(err)
return
}
conn.Close() // Deterministic EOF, causing the SDK-compatible retry.
return
}
req := r.Clone(r.Context())
req.RequestURI = ""
// A forwarding proxy must preserve absence, like ARTEX's sink.
if _, present := req.Header["User-Agent"]; !present {
req.Header.Set("User-Agent", "")
}
resp, err := target.Client().Do(req)
if err != nil {
t.Error(err)
w.WriteHeader(http.StatusBadGateway)
return
}
defer resp.Body.Close()
for key, values := range resp.Header {
w.Header()[key] = values
}
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}))
defer proxy.Close()
cfg := actool.WebFetchConfig{}
if route != "direct" {
cfg.Proxy = proxy.URL
}
result := callFetch(t, New(cfg), target.URL+"/start", true)
for _, want := range []string{"[200 OK]", "# Page", "=== RECON ===", "keep", "/app.js", "csrf(hidden=token)"} {
if result.IsError || !strings.Contains(result.Flatten(), want) {
t.Fatalf("missing %q in result: %+v", want, result)
}
}
if targetCalls.Load() != 2 {
t.Errorf("target calls = %d, want 2", targetCalls.Load())
}
if route != "direct" && proxyCalls.Load() == 0 {
t.Error("configured proxy was bypassed")
}
})
}
}
}
func TestTLSAndStaticAssetGuard(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", "Captured-UA/1.0")
var calls atomic.Int32
server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
if r.UserAgent() != "Captured-UA/1.0" {
t.Errorf("User-Agent = %q", r.UserAgent())
}
fmt.Fprint(w, "verified")
}))
defer server.Close()
certPath := filepath.Join(t.TempDir(), "ca.pem")
cert := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})
if err := os.WriteFile(certPath, cert, 0600); err != nil {
t.Fatal(err)
}
tool := New(actool.WebFetchConfig{CACert: certPath})
if result := callFetch(t, tool, server.URL, false); result.IsError || !strings.Contains(result.Flatten(), "verified") {
t.Fatalf("CA trust lost: %+v", result)
}
if result := callFetch(t, tool, server.URL+"/app.js?v=1", false); !result.IsError {
t.Fatalf("static asset guard lost: %+v", result)
}
if calls.Load() != 1 {
t.Errorf("static asset triggered network request: calls = %d", calls.Load())
}
}
func TestIdentitySelectedForEachCall(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", "Before-Tool-Creation")
tool := New(actool.WebFetchConfig{})
seen := make(chan string, 2)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
seen <- r.UserAgent()
fmt.Fprint(w, "ok")
}))
defer server.Close()
for _, want := range []string{"First-Call", "Second-Call"} {
t.Setenv("ARTEX_HTTP_USER_AGENT", want)
if result := callFetch(t, tool, server.URL, false); result.IsError {
t.Fatal(result.Flatten())
}
if got := <-seen; got != want {
t.Fatalf("identity cached across calls: got %q, want %q", got, want)
}
}
}
func TestSDKMetadataPreserved(t *testing.T) {
base := actool.NewWebFetch(actool.WebFetchConfig{})
local := New(actool.WebFetchConfig{})
if local.Name() != base.Name() || local.Description() != base.Description() || !reflect.DeepEqual(local.InputSchema(), base.InputSchema()) {
t.Fatal("SDK tool contract changed")
}
}
+3 -1
View File
@@ -9,6 +9,8 @@ import (
"strconv"
"strings"
"time"
"github.com/Autumn-27/artex/internal/httpidentity"
)
// Repo 是发布源。写死而不是做成配置项:更新源可配等于给任何能改配置的人一条
@@ -125,7 +127,7 @@ func FetchLatest(ctx context.Context, c *http.Client) (*Release, error) {
return nil, err
}
req.Header.Set("Accept", "application/vnd.github+json")
req.Header.Set("User-Agent", "artex-selfupdate")
httpidentity.Apply(req)
resp, err := c.Do(req)
if err != nil {
+35
View File
@@ -56,3 +56,38 @@ func TestFetchLatestMissingRelease(t *testing.T) {
t.Fatalf("unexpected missing-release error: %v", err)
}
}
func TestUpdateUserAgent(t *testing.T) {
for _, tc := range []struct{ env, want string }{
{"", ""},
{"Captured-UA/1.0", "Captured-UA/1.0"},
} {
t.Run(tc.env, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
calls := 0
client := &http.Client{Transport: releaseTransport(func(req *http.Request) (*http.Response, error) {
calls++
if got := req.UserAgent(); tc.want == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", got)
} else if tc.want != "" && got != tc.want {
t.Errorf("User-Agent = %q, want %q", got, tc.want)
}
body := `{"tag_name":"v0.3.17"}`
return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(body))}, nil
})}
if _, err := FetchLatest(context.Background(), client); err != nil {
t.Fatal(err)
}
for _, name := range []string{"SHA256SUMS", "artex.zip"} {
body, err := get(context.Background(), client, "https://github.com/RuoJi6/ARTEX/releases/download/v0.3.17/"+name)
if err != nil {
t.Fatal(err)
}
body.Close()
}
if calls != 3 {
t.Fatalf("calls = %d, want 3", calls)
}
})
}
}
+3 -1
View File
@@ -13,6 +13,8 @@ import (
"runtime"
"strings"
"time"
"github.com/Autumn-27/artex/internal/httpidentity"
)
// sumsAsset 是 release.yml 生成的校验和清单,覆盖 Release 里全部 zip。
@@ -229,7 +231,7 @@ func get(ctx context.Context, c *http.Client, rawURL string) (io.ReadCloser, err
if err := checkURL(req.URL); err != nil {
return nil, err
}
req.Header.Set("User-Agent", "artex-selfupdate")
httpidentity.Apply(req)
resp, err := c.Do(req)
if err != nil {
return nil, err
+2
View File
@@ -18,6 +18,7 @@ import (
"time"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/artex/internal/httpidentity"
"github.com/Autumn-27/norma/permission"
actool "github.com/Autumn-27/norma/tool"
)
@@ -438,6 +439,7 @@ func (s *Server) runHTTPTool(ctx context.Context, execRaw json.RawMessage, param
if err != nil {
return actool.Errorf(maskSecrets(err.Error())), nil
}
httpidentity.Apply(req)
for k, v := range spec.Headers {
req.Header.Set(k, renderTemplate(v, params, identity))
}
+4 -2
View File
@@ -70,7 +70,9 @@ type TaskDTO struct {
CompanyIDs []int64 `json:"company_ids"`
CoverageEnabled bool `json:"coverage_enabled"` // 资产覆盖度功能开关(创建时定)
ShareGlobalAssets bool `json:"share_global_assets"`
WorkerCount int `json:"worker_count"` // 本任务 Worker 数;0=用全局默认
WorkerCount int `json:"worker_count"` // 本任务 Worker 数;0=用全局默认
WorkerActualCount int `json:"worker_actual_count"` // live pool loops, including draining workers
WorkerTargetCount int `json:"worker_target_count"` // desired worker slots
NoResultSeconds int `json:"no_result_seconds"`
NoResultMinFindings int `json:"no_result_min_findings"`
@@ -170,7 +172,7 @@ func taskDTO(t *Task, status string) TaskDTO {
CompanyIDs: lifecycle.CompanyIDs,
CoverageEnabled: t.CoverageEnabled,
ShareGlobalAssets: lifecycle.ShareGlobalAssets,
WorkerCount: t.WorkerCount,
WorkerCount: lifecycle.WorkerCount,
NoResultSeconds: t.NoResultSeconds,
NoResultMinFindings: t.NoResultMinFindings,
+2 -1
View File
@@ -23,6 +23,7 @@ import (
"syscall"
"time"
"github.com/Autumn-27/artex/internal/httpidentity"
"github.com/Autumn-27/artex/traffic"
)
@@ -305,7 +306,7 @@ func probeOne(ctx context.Context, client *http.Client, target string) (string,
if err != nil {
return "", err
}
req.Header.Set("User-Agent", "artex-egress-probe/1.0")
httpidentity.Apply(req)
resp, err := client.Do(req)
if err != nil {
+21
View File
@@ -27,6 +27,27 @@ func fakeIPServer(ip string) *httptest.Server {
}))
}
func TestProbeOneUserAgent(t *testing.T) {
for _, ua := range []string{"", "Captured-UA/1.0"} {
t.Run(ua, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", got)
} else if ua != "" && got != ua {
t.Errorf("User-Agent = %q, want %q", got, ua)
}
fmt.Fprint(w, `{"ip":"203.0.113.7"}`)
}))
defer server.Close()
ip, err := probeOne(context.Background(), server.Client(), server.URL)
if err != nil || ip != "203.0.113.7" {
t.Fatalf("ip = %q, err = %v", ip, err)
}
})
}
}
// TestProbeEgress 覆盖 probeEgress 的输入/响应形态:成功、非法协议、死代理、坏响应、回落。
// 每条用例自带服务端与断言,失败时 t.Run 的名字直接指出是哪一条。
func TestProbeEgress(t *testing.T) {
+42 -47
View File
@@ -111,8 +111,9 @@ type Engine struct {
// Every long-lived task goroutine (planner, workers and deadline coordinator)
// runs under one task-scoped context. Successful deletion cancels that context,
// waits for all goroutines, then releases every task-level Engine reference.
runtimeMu sync.Mutex
runtimes map[string]*taskRuntime
runtimeMu sync.Mutex
runtimes map[string]*taskRuntime
workerPools sync.Map // taskID -> *taskWorkerPool; each pool serializes claims with resizing
// per-task execution context: each planner.Plan / worker.Execute runs under it,
// so pausing can CANCEL an in-flight run (not just skip the next one). Recreated
@@ -150,7 +151,7 @@ type Engine struct {
resolveAuthoritative bool
workerResolve func(t *Task, slot WorkerSlot) (*agent.Worker, error)
slotState sync.Map // task/slot -> slotAvailability
slotPlans sync.Map // task -> map[slot name]profile ID; immutable for a run
slotPlans sync.Map // task -> immutable snapshot of slot name/profile routing
// readiness reports whether a global LLM provider is configured — the signal behind
// Ready()/the llm_configured indicator. Wired once at startup; nil → not ready.
readiness func() bool
@@ -297,6 +298,7 @@ func (e *Engine) StopTask(taskID string) {
e.inflight.Delete(taskID)
e.coordStarted.Delete(taskID)
e.slotPlans.Delete(taskID)
e.workerPools.Delete(taskID)
e.slotState.Range(func(key, _ any) bool {
keyText, ok := key.(string)
if ok && strings.HasPrefix(keyText, taskID+"/") {
@@ -406,10 +408,14 @@ func (e *Engine) ActiveLLMCalls(taskID string) int64 {
func (e *Engine) touch(taskID string) { e.lastAct.Store(taskID, time.Now().Unix()) }
func NewEngine(m *Manager) *Engine {
return &Engine{m: m, debounce: 800 * time.Millisecond, bc: NewBroadcaster(),
e := &Engine{m: m, debounce: 800 * time.Millisecond, bc: NewBroadcaster(),
execCancel: map[string]context.CancelCauseFunc{}, execCtx: map[string]context.Context{},
work: map[int64]*workExecution{}, steerBox: map[int64][]string{},
runtimes: map[string]*taskRuntime{}}
if m != nil {
m.SetWorkerConfigChangedCallback(e.ReconcileDefaultWorkers)
}
return e
}
// registerWork records the cancel for the work currently running intentID.
@@ -796,11 +802,6 @@ func (e *Engine) ReadyFor(t *Task) bool {
// but no-op until an LLM is configured (so a task created while idle picks up
// automatically once LLM is set from the UI).
func (e *Engine) Run(ctx context.Context, t *Task) {
slots, err := workerSlotsForTask(t, e.m.WorkerPools(), e.m.Workers())
if err != nil {
log.Printf("[engine] task %s worker pool invalid, using legacy workers: %v", t.ID, err)
slots, _ = ExpandWorkerSlots(nil, workerCountForTask(t, e.m.Workers()))
}
e.deleteMu.RLock()
if e.IsDeleting(t.ID) {
e.deleteMu.RUnlock()
@@ -811,19 +812,13 @@ func (e *Engine) Run(ctx context.Context, t *Task) {
t.Notify() // already running — just nudge a planning round
return
}
rt := e.registerTaskRoutines(ctx, t.ID, 1+len(slots))
plan := make(map[string]int64, len(slots))
for _, slot := range slots {
plan[slot.Name] = slot.ProfileID
}
e.slotPlans.Store(t.ID, plan)
rt := e.registerTaskRoutines(ctx, t.ID, 1)
pool := &taskWorkerPool{runtime: rt, task: t, loops: make(map[string]*workerSlotRuntime)}
e.workerPools.Store(t.ID, pool)
e.deleteMu.RUnlock()
e.touch(t.ID)
runTaskRoutine(rt, func(loopCtx context.Context) { e.plannerLoop(loopCtx, t) })
for _, slot := range slots {
slot := slot
runTaskRoutine(rt, func(loopCtx context.Context) { e.workerLoop(loopCtx, t, slot) })
}
e.ReconcileWorkers(t)
e.startDeadlineCoordinator(ctx, t) // 任务级超时定时器(仅 timeout>0;去重)
// 仅在「完全没有活动意图(open+running)」时才 kick 首轮规划。带种子意图的任务:种子已
// 是 open,或已被上面刚起的 worker 抢先 claim 成 running——两种都算「有活干」,一律跳过
@@ -838,20 +833,20 @@ func (e *Engine) Run(ctx context.Context, t *Task) {
// workerSlotsForTask preserves an explicit task concurrency override. Tasks
// using the global default inherit its model pools and their pinned profiles.
func workerSlotsForTask(t *Task, pools []WorkerPool, global int) ([]WorkerSlot, error) {
if t != nil && t.WorkerCount > 0 {
if t != nil && t.lifecycleSnapshot().WorkerCount > 0 {
return ExpandWorkerSlots(nil, workerCountForTask(t, global))
}
return ExpandWorkerSlots(pools, global)
}
func workerCountForTask(t *Task, global int) int {
if t != nil && t.WorkerCount > 0 {
// Direct database edits can bypass validation; cap those values so the
// engine never starts more than the supported per-task worker count.
if t.WorkerCount > db.MaxTaskWorkerCount {
return db.MaxTaskWorkerCount
if t != nil {
if count := t.lifecycleSnapshot().WorkerCount; count > 0 {
if count > db.MaxTaskWorkerCount {
return db.MaxTaskWorkerCount
}
return count
}
return t.WorkerCount
}
return global
}
@@ -1004,11 +999,12 @@ func (e *Engine) plannerLoop(ctx context.Context, t *Task) {
}
}
func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot, control *workerSlotRuntime) {
name := slot.Name
idleCtx := control.idleCtx
for {
select {
case <-ctx.Done():
case <-idleCtx.Done():
return
default:
}
@@ -1031,13 +1027,13 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
_, worker = e.snapshotFor(t)
}
if worker == nil {
if sleepCtx(ctx, 1500*time.Millisecond) {
if sleepCtx(idleCtx, 1500*time.Millisecond) {
return
}
continue
}
if e.IsPaused(t.ID) {
if sleepCtx(ctx, 1000*time.Millisecond) {
if sleepCtx(idleCtx, 1000*time.Millisecond) {
return
}
continue // user-paused: don't claim/execute intents
@@ -1046,13 +1042,13 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
return
}
if e.isSettling(t.ID) {
if sleepCtx(ctx, 1000*time.Millisecond) {
if sleepCtx(idleCtx, 1000*time.Millisecond) {
return
}
continue // 任务超时收尾中:不再领新意图(在跑的自行收尾,协调器等其 drain)
}
if isTerminalStatus(e.m.TaskStatus(t.ID)) {
if sleepCtx(ctx, 1000*time.Millisecond) {
if sleepCtx(idleCtx, 1000*time.Millisecond) {
return
}
continue // 任务已终态(done/failed/timeout):停止领取遗留意图,别在完成后空跑 frontier
@@ -1060,28 +1056,27 @@ func (e *Engine) workerLoop(ctx context.Context, t *Task, slot WorkerSlot) {
if !e.beginTaskOperation(t.ID) {
return
}
claimed := e.runWorkerStep(ctx, t, name, worker)
// A resize and the next claim share the pool lock. Once retirement is
// requested this loop cannot claim again; execution keeps the task context.
control.pool.mu.Lock()
var intent *db.Node
if idleCtx.Err() == nil {
intent = e.claimNext(t, name)
}
control.pool.mu.Unlock()
if intent != nil {
log.Printf("[worker %s] task %s 领取意图 #%d", name, t.ID, intent.ID)
e.runIntent(ctx, t, name, worker, intent, "", "")
}
e.decInflight(t.ID)
if !claimed && sleepCtx(ctx, 800*time.Millisecond) {
if intent == nil && sleepCtx(idleCtx, 800*time.Millisecond) {
return
}
}
}
// runWorkerStep claims one intent from the frontier and fully settles it via
// runIntent. Returns false when nothing was claimable. The pool worker loop is its
// only caller.
func (e *Engine) runWorkerStep(ctx context.Context, t *Task, name string, worker *agent.Worker) bool {
intent := e.claimNext(t, name)
if intent == nil {
return false
}
log.Printf("[worker %s] task %s 领取意图 #%d", name, t.ID, intent.ID)
return e.runIntent(ctx, t, name, worker, intent, "", "")
}
// runIntent executes and fully settles one already-claimed (state=running) intent.
// Both the pool worker loop (via runWorkerStep) and the human-message handler (via
// Both the pool worker loop and the human-message handler (via
// runDetachedIntent, a dedicated goroutine outside the worker pool) call it, so the
// execute/retry/state-write logic lives in exactly one place. A non-empty message
// is injected as this turn's input through ExecuteWithMessage; requestID keys the
+131
View File
@@ -0,0 +1,131 @@
package server
import (
"context"
"log"
"sync"
)
// taskWorkerPool owns the desired plan and all live loops, including retirees.
// A slot name is never reused until its previous loop has fully settled its work.
type taskWorkerPool struct {
mu sync.Mutex
runtime *taskRuntime
task *Task
loops map[string]*workerSlotRuntime
desired map[string]WorkerSlot
}
type workerSlotRuntime struct {
pool *taskWorkerPool
slot WorkerSlot
idleCtx context.Context
retire context.CancelFunc // stops claiming/sleeping, never cancels Execute
}
func (e *Engine) desiredWorkerSlots(t *Task) []WorkerSlot {
slots, err := workerSlotsForTask(t, e.m.WorkerPools(), e.m.Workers())
if err != nil {
log.Printf("[engine] task %s worker pool invalid, using legacy workers: %v", t.ID, err)
slots, _ = ExpandWorkerSlots(nil, workerCountForTask(t, e.m.Workers()))
}
return slots
}
// ReconcileWorkers applies the latest persisted settings without restarting a task.
// Claims and retirement share the pool lock, so no retired slot takes new work.
func (e *Engine) ReconcileWorkers(t *Task) {
e.deleteMu.RLock()
defer e.deleteMu.RUnlock()
if e.IsDeleting(t.ID) {
return
}
value, ok := e.workerPools.Load(t.ID)
if !ok {
return // queued/not started: Run will read the latest configuration
}
pool := value.(*taskWorkerPool)
pool.mu.Lock()
defer pool.mu.Unlock()
if pool.runtime.ctx.Err() != nil {
return
}
slots := e.desiredWorkerSlots(t)
pool.desired = make(map[string]WorkerSlot, len(slots))
for _, slot := range slots {
pool.desired[slot.Name] = slot
}
for name, loop := range pool.loops {
if slot, keep := pool.desired[name]; !keep || slot != loop.slot {
loop.retire()
}
}
for _, slot := range slots {
if pool.loops[slot.Name] == nil {
e.startWorkerSlotLocked(pool, slot)
}
}
}
// ReconcileDefaultWorkers is called after a global workers/pool save. Explicit
// task overrides keep both their concurrency and legacy task-router profiles.
func (e *Engine) ReconcileDefaultWorkers() {
e.workerPools.Range(func(_, value any) bool {
t := value.(*taskWorkerPool).task
if t.lifecycleSnapshot().WorkerCount == 0 {
e.ReconcileWorkers(t)
}
return true
})
}
// WorkerCounts distinguishes live pool loops (including draining workers) from
// the effective configured target. Neither count is the number of busy intents.
func (e *Engine) WorkerCounts(t *Task) (actual, target int) {
if value, ok := e.workerPools.Load(t.ID); ok {
pool := value.(*taskWorkerPool)
pool.mu.Lock()
defer pool.mu.Unlock()
return len(pool.loops), len(pool.desired)
}
return 0, len(e.desiredWorkerSlots(t))
}
// Caller holds deleteMu.RLock and pool.mu. The delete barrier protects wg.Add
// from StopTask.Wait, including replacements started as a retiree exits.
func (e *Engine) startWorkerSlotLocked(pool *taskWorkerPool, slot WorkerSlot) {
idleCtx, retire := context.WithCancel(pool.runtime.ctx)
loop := &workerSlotRuntime{pool: pool, slot: slot, idleCtx: idleCtx, retire: retire}
pool.loops[slot.Name] = loop
// Publish a fresh immutable routing snapshot. Retired slot routes remain
// available for human continuation of paused intents until the name is reused.
plan := make(map[string]int64)
if old, ok := e.slotPlans.Load(pool.task.ID); ok {
for name, profile := range old.(map[string]int64) {
plan[name] = profile
}
}
plan[slot.Name] = slot.ProfileID
e.slotPlans.Store(pool.task.ID, plan)
pool.runtime.wg.Add(1)
runTaskRoutine(pool.runtime, func(ctx context.Context) {
defer e.finishWorkerSlot(loop)
e.workerLoop(ctx, pool.task, slot, loop)
})
}
func (e *Engine) finishWorkerSlot(loop *workerSlotRuntime) {
loop.retire()
e.deleteMu.RLock()
defer e.deleteMu.RUnlock()
pool := loop.pool
pool.mu.Lock()
defer pool.mu.Unlock()
delete(pool.loops, loop.slot.Name)
e.slotState.Delete(pool.task.ID + "/" + loop.slot.Name)
if !e.IsDeleting(pool.task.ID) && pool.runtime.ctx.Err() == nil {
if slot, needed := pool.desired[loop.slot.Name]; needed {
e.startWorkerSlotLocked(pool, slot)
}
}
}
+254
View File
@@ -0,0 +1,254 @@
package server
import (
"context"
"fmt"
"iter"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
"github.com/Autumn-27/artex/agent"
"github.com/Autumn-27/artex/db"
"github.com/Autumn-27/norma/llm"
)
type resizingCall struct {
ctx context.Context
release chan struct{}
}
type resizingProvider struct{ calls chan resizingCall }
func (p *resizingProvider) Complete(ctx context.Context, _ llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
call := resizingCall{ctx: ctx, release: make(chan struct{})}
select {
case p.calls <- call:
case <-ctx.Done():
return llm.Message{}, "", llm.Usage{}, ctx.Err()
}
select {
case <-call.release:
return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("fixture completed")}}, "end_turn", llm.Usage{}, nil
case <-ctx.Done():
return llm.Message{}, "", llm.Usage{}, ctx.Err()
}
}
func (*resizingProvider) Stream(context.Context, llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
panic("resizing fixture uses non-streaming calls")
}
func waitWorkerCondition(t *testing.T, condition func() bool) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) {
if condition() {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatal("timed out waiting for worker state")
}
// Exercises the real worker claim/Execute/settlement path. Both slots remain
// blocked in provider calls while shrinking; only the retained slot may claim
// another intent, and re-expansion must reuse a name without overlapping loops.
func TestWorkerHotResizeDrainsBeforeReusingSlot(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Skipf("postgres unavailable: %v", err)
}
defer m.Close()
task, err := m.CreateTask("resize workers", "fixture", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
defer m.DeleteTask(task.ID, DeleteTaskOptions{})
count := 2
if _, err := m.UpdateTaskMetadata(task.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
t.Fatal(err)
}
for range 2 {
if _, err := task.Store.AddIntent(map[string]any{"text": "fixture"}, 0, nil, "test"); err != nil {
t.Fatal(err)
}
}
e := NewEngine(m)
defer e.StopTask(task.ID)
var mu sync.Mutex
providers := make(map[string]*resizingProvider)
e.SetAuthoritativeAgentResolver(func(*Task) (*agent.Planner, *agent.Worker) { return nil, nil })
// Use a pinned pool so each slot's blocked call can be released independently.
profileID, err := m.pg.SaveProfile(&db.LLMProfile{Name: "resize-fixture", Format: "openai", Model: "fixture"})
if err != nil {
t.Fatal(err)
}
defer m.pg.DeleteProfile(profileID)
oldPools, _, _ := m.pg.GetSetting(settingWorkerPools)
oldWorkers, _, _ := m.pg.GetSetting(settingWorkers)
defer m.pg.SetSetting(settingWorkerPools, oldPools)
defer m.pg.SetSetting(settingWorkers, oldWorkers)
count = 0
if _, err := m.UpdateTaskMetadata(task.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
t.Fatal(err)
}
workDir := t.TempDir()
e.SetWorkerResolver(func(_ *Task, slot WorkerSlot) (*agent.Worker, error) {
mu.Lock()
defer mu.Unlock()
p := providers[slot.Name]
if p == nil {
p = &resizingProvider{calls: make(chan resizingCall, 4)}
providers[slot.Name] = p
}
w := agent.NewWorker(p, "fixture", workDir, nil, 0, 0)
w.SetNonStreaming(func() bool { return true })
return w, nil
})
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
t.Fatal(err)
}
e.Run(t.Context(), task)
getCall := func(name string) resizingCall {
t.Helper()
var p *resizingProvider
waitWorkerCondition(t, func() bool {
mu.Lock()
defer mu.Unlock()
p = providers[name]
return p != nil
})
select {
case call := <-p.calls:
return call
case <-time.After(5 * time.Second):
t.Fatalf("%s did not execute an intent", name)
return resizingCall{}
}
}
first, second := getCall("work#1"), getCall("work#2")
thirdID, err := task.Store.AddIntent(map[string]any{"text": "next"}, 0, nil, "test")
if err != nil {
t.Fatal(err)
}
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 1}}); err != nil {
t.Fatal(err)
}
if actual, target := e.WorkerCounts(task); actual != 2 || target != 1 {
t.Fatalf("draining counts=%d/%d, want 2/1", actual, target)
}
if second.ctx.Err() != nil || first.ctx.Err() != nil {
t.Fatal("resize cancelled an executing intent")
}
close(second.release)
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(task); return actual == 1 && target == 1 })
third, err := task.Store.GetNode(thirdID)
if err != nil || third.State != "open" {
t.Fatalf("retired slot claimed queued intent: node=%+v err=%v", third, err)
}
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
t.Fatal(err)
}
thirdCall := getCall("work#2")
third, err = task.Store.GetNode(thirdID)
if err != nil || third.State != "running" || third.Owner != "work#2" {
t.Fatalf("expanded slot did not claim queued intent: node=%+v err=%v", third, err)
}
// A rapid shrink/grow must not start a duplicate work#2 while it drains.
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 1}}); err != nil {
t.Fatal(err)
}
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
t.Fatal(err)
}
if actual, target := e.WorkerCounts(task); actual != 2 || target != 2 || thirdCall.ctx.Err() != nil {
t.Fatalf("rapid resize interrupted work or duplicated slot: %d/%d err=%v", actual, target, thirdCall.ctx.Err())
}
close(thirdCall.release)
close(first.release)
waitWorkerCondition(t, func() bool {
node, _ := task.Store.GetNode(thirdID)
return node != nil && node.State != "running" && e.inflightCount(task.ID) == 0
})
}
func TestWorkerHotResizeTaskOverrideAndGlobalDefault(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Skipf("postgres unavailable: %v", err)
}
defer m.Close()
oldPools, _, _ := m.pg.GetSetting(settingWorkerPools)
oldWorkers, _, _ := m.pg.GetSetting(settingWorkers)
defer m.pg.SetSetting(settingWorkerPools, oldPools)
defer m.pg.SetSetting(settingWorkers, oldWorkers)
if err := m.SetWorkerPools(nil, 2); err != nil {
t.Fatal(err)
}
e := NewEngine(m)
global, err := m.CreateTask("global resize fixture", "fixture", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
defer m.DeleteTask(global.ID, DeleteTaskOptions{})
defer e.StopTask(global.ID)
override, err := m.CreateTask("explicit resize fixture", "fixture", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
defer m.DeleteTask(override.ID, DeleteTaskOptions{})
defer e.StopTask(override.ID)
count := 3
if _, err := m.UpdateTaskMetadata(override.ID, db.TaskPatch{WorkerCount: &count}); err != nil {
t.Fatal(err)
}
e.Run(t.Context(), global)
e.Run(t.Context(), override)
if err := m.SetWorkers(4); err != nil {
t.Fatal(err)
}
if actual, target := e.WorkerCounts(global); actual != 4 || target != 4 {
t.Fatalf("global default did not expand: %d/%d", actual, target)
}
if actual, target := e.WorkerCounts(override); actual != 3 || target != 3 {
t.Fatalf("global setting changed explicit override: %d/%d", actual, target)
}
if err := m.SetWorkers(1); err != nil {
t.Fatal(err)
}
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(global); return actual == 1 && target == 1 })
if actual, target := e.WorkerCounts(override); actual != 3 || target != 3 {
t.Fatalf("global shrink changed explicit override: %d/%d", actual, target)
}
// Exercise the live PATCH path, then restore global inheritance through PATCH.
s := &Server{m: m, engine: e, ctx: t.Context()}
patch := func(count int) {
t.Helper()
req := httptest.NewRequest(http.MethodPatch, "/api/tasks/"+override.ID, strings.NewReader(fmt.Sprintf(`{"worker_count":%d}`, count)))
req.SetPathValue("id", override.ID)
rec := httptest.NewRecorder()
s.updateTaskMetadata(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("live patch=%d: %s", rec.Code, rec.Body.String())
}
}
patch(5)
if actual, target := e.WorkerCounts(override); actual != 5 || target != 5 {
t.Fatalf("task PATCH did not expand: %d/%d", actual, target)
}
patch(0)
waitWorkerCondition(t, func() bool { actual, target := e.WorkerCounts(override); return actual == 1 && target == 1 })
persisted, err := m.pg.GetTask(taskIDInt(t, override.ID))
if err != nil || persisted.WorkerCount != 0 {
t.Fatalf("inheritance not persisted: task=%+v err=%v", persisted, err)
}
// Reconstructing a task after restart must use the persisted override policy.
restored := taskFromPG(persisted, override.Store, m.interceptor)
if slots, err := workerSlotsForTask(restored, m.WorkerPools(), m.Workers()); err != nil || len(slots) != 1 {
t.Fatalf("restored task lost concurrency policy: slots=%v err=%v", slots, err)
}
}
+54
View File
@@ -0,0 +1,54 @@
package server
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestTargetHTTPIdentityOverrides(t *testing.T) {
t.Setenv(repeaterAllowLocalEnv, "1")
for _, route := range []string{"custom", "repeater"} {
for _, tc := range []struct {
name, env, want string
headers map[string]string
}{
{name: "default"},
{name: "environment", env: "Captured-UA/1.0", want: "Captured-UA/1.0"},
{name: "override", env: "Environment-UA/1.0", want: "Explicit-UA/1.0", headers: map[string]string{"user-agent": "Explicit-UA/1.0"}},
{name: "explicit-empty", env: "Environment-UA/1.0", headers: map[string]string{"User-Agent": ""}},
} {
t.Run(route+"/"+tc.name, func(t *testing.T) {
t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.UserAgent(); tc.name == "default" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
t.Errorf("random UA missing: %q", got)
} else if tc.name != "default" && got != tc.want {
t.Errorf("User-Agent = %q, want %q", got, tc.want)
}
if _, present := r.Header["User-Agent"]; tc.name == "explicit-empty" && present {
t.Error("empty User-Agent present on wire")
}
w.Write([]byte("ok"))
}))
defer server.Close()
payload := map[string]any{"url": server.URL, "headers": tc.headers, "use_recording_proxy": false}
if route == "custom" {
raw, _ := json.Marshal(payload)
result, err := (&Server{}).runHTTPTool(context.Background(), raw, nil, nil)
if err != nil || result.IsError {
t.Fatalf("custom tool: %v %+v", err, result)
}
} else {
status, result, body := repeaterTestSend(t, &Server{}, payload)
if status != http.StatusOK || result.Status != http.StatusOK {
t.Fatalf("repeater: %d %s", status, body)
}
}
})
}
}
}
+55 -19
View File
@@ -99,6 +99,7 @@ type Task struct {
// directly after the task has been published by Manager.
type taskLifecycleState struct {
ShareGlobalAssets bool
WorkerCount int
Name string
Remark string
PinnedAt int64
@@ -140,6 +141,7 @@ func (t *Task) lifecycleSnapshot() taskLifecycleState {
func (t *Task) lifecycleSnapshotLocked() taskLifecycleState {
return taskLifecycleState{
ShareGlobalAssets: t.ShareGlobalAssets,
WorkerCount: t.WorkerCount,
Name: t.Name,
Remark: t.Remark,
PinnedAt: t.PinnedAt,
@@ -171,6 +173,7 @@ func (t *Task) updateLifecycle(update func(*taskLifecycleState)) {
state := t.lifecycleSnapshotLocked()
update(&state)
t.ShareGlobalAssets = state.ShareGlobalAssets
t.WorkerCount = state.WorkerCount
t.Name = state.Name
t.Remark = state.Remark
t.PinnedAt = state.PinnedAt
@@ -282,13 +285,14 @@ type Manager struct {
// taskStateMu preserves commit order between PostgreSQL lifecycle writes and
// their in-memory mirrors. lifecycleMu makes snapshots race-free, but without
// this outer write lock an older request could commit first and publish last.
taskStateMu sync.Mutex
controlMu *sync.Mutex // Server's admission/task-control mutex, when attached.
mu sync.RWMutex
tasks map[string]*Task
active string
trafficOn bool // 流量捕获开关(默认关;settings.traffic_capture)
llmRecOn bool // LLM 录制开关(默认关;settings.llm_record)
taskStateMu sync.Mutex
controlMu *sync.Mutex // Server's admission/task-control mutex, when attached.
mu sync.RWMutex
tasks map[string]*Task
workerConfigChanged func()
active string
trafficOn bool // 流量捕获开关(默认关;settings.traffic_capture)
llmRecOn bool // LLM 录制开关(默认关;settings.llm_record)
// 联网搜索开关与来源(默认关;settings.web_search_*)。brave-free 需要 braveKey;tavily 需要 tavilyKey。
// webSearchProxy 是独立出口代理(http/https/socks5),与记录流量的 MITM 代理无关。
webSearchOn bool
@@ -310,6 +314,23 @@ type Manager struct {
// participate in the Server's admission/task-control critical section.
func (m *Manager) SetTaskControlMutex(mu *sync.Mutex) { m.controlMu = mu }
// SetWorkerConfigChangedCallback installs the engine's live worker reconciler.
// Successful worker-setting writes invoke it after releasing Manager locks.
func (m *Manager) SetWorkerConfigChangedCallback(callback func()) {
m.mu.Lock()
m.workerConfigChanged = callback
m.mu.Unlock()
}
func (m *Manager) notifyWorkerConfigChanged() {
m.mu.RLock()
callback := m.workerConfigChanged
m.mu.RUnlock()
if callback != nil {
callback()
}
}
// Settings keys the UI toggles at runtime.
const (
settingTrafficCapture = "traffic_capture"
@@ -433,6 +454,9 @@ func ExpandWorkerSlots(pools []WorkerPool, legacy int) ([]WorkerSlot, error) {
if legacy <= 0 {
legacy = defaultWorkers
}
if legacy > pgdb.MaxTaskWorkerCount {
legacy = pgdb.MaxTaskWorkerCount
}
slots := make([]WorkerSlot, legacy)
for i := range slots {
slots[i] = WorkerSlot{Name: fmt.Sprintf("work#%d", i+1)}
@@ -478,8 +502,8 @@ func (m *Manager) SetConcurrency(enabled bool, limit int) error {
return m.pg.SetSetting(settingConcurrencyOn, strconv.FormatBool(enabled))
}
// Workers returns the configured concurrent work-agent count (default 3). Read
// per-task at engine.Run, so a change applies to tasks started afterwards.
// Workers returns the configured concurrent work-agent count (default 3),
// capped at the supported maximum even for settings saved by older versions.
func (m *Manager) Workers() int {
v, ok, err := m.pg.GetSetting(settingWorkers)
if err != nil || !ok {
@@ -489,6 +513,9 @@ func (m *Manager) Workers() int {
if err != nil || n <= 0 {
return defaultWorkers
}
if n > pgdb.MaxTaskWorkerCount {
return pgdb.MaxTaskWorkerCount
}
return n
}
@@ -542,23 +569,31 @@ func (m *Manager) SetWorkerPools(pools []WorkerPool, legacyWorkers ...int) error
workerCount += pool.Workers
}
} else if len(legacyWorkers) > 0 {
if legacyWorkers[0] <= 0 {
return fmt.Errorf("workers 必须 >0")
if legacyWorkers[0] <= 0 || legacyWorkers[0] > pgdb.MaxTaskWorkerCount {
return fmt.Errorf("workers 必须在 1 到 %d 之间", pgdb.MaxTaskWorkerCount)
}
workerCount = legacyWorkers[0]
}
return m.pg.SetSettingsAtomic(context.Background(), map[string]string{
if err := m.pg.SetSettingsAtomic(context.Background(), map[string]string{
settingWorkerPools: string(b),
settingWorkers: strconv.Itoa(workerCount),
})
}); err != nil {
return err
}
m.notifyWorkerConfigChanged()
return nil
}
// SetWorkers persists the concurrent work-agent count. Values <=0 are rejected.
// SetWorkers persists the concurrent work-agent count and reconciles live tasks.
func (m *Manager) SetWorkers(n int) error {
if n <= 0 {
return fmt.Errorf("workers 必须 >0")
if n <= 0 || n > pgdb.MaxTaskWorkerCount {
return fmt.Errorf("workers 必须在 1 到 %d 之间", pgdb.MaxTaskWorkerCount)
}
return m.pg.SetSetting(settingWorkers, strconv.Itoa(n))
if err := m.pg.SetSetting(settingWorkers, strconv.Itoa(n)); err != nil {
return err
}
m.notifyWorkerConfigChanged()
return nil
}
// Enrich returns the asset auto-completion engine (may be nil if init failed).
@@ -1179,8 +1214,8 @@ func taskFromPG(pt *pgdb.Task, store *pgdb.ExplorationStore, ic *intercept.Inter
}
}
// UpdateTaskMetadata changes task metadata and asset visibility without interrupting any
// planner, main-agent, or worker call.
// UpdateTaskMetadata changes task metadata, asset visibility, and worker policy
// without interrupting any planner, main-agent, or worker call.
func (m *Manager) UpdateTaskMetadata(taskID string, patch pgdb.TaskPatch) (*Task, error) {
m.taskStateMu.Lock()
defer m.taskStateMu.Unlock()
@@ -1200,6 +1235,7 @@ func (m *Manager) UpdateTaskMetadata(taskID string, patch pgdb.TaskPatch) (*Task
}
task.updateLifecycle(func(state *taskLifecycleState) {
state.ShareGlobalAssets = updated.ShareGlobalAssets
state.WorkerCount = updated.WorkerCount
state.Name = updated.Name
state.Remark = updated.Remark
state.PinnedAt = unixOrZero(updated.PinnedAt)
+18 -1
View File
@@ -55,6 +55,7 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
state.CompletedAt = generation
state.FirstRunAt = generation
state.DeadlineAt = generation
state.WorkerCount = int(generation % 51)
state.SourceTaskIDs = []int64{generation, generation + 1}
state.CompanyIDs = []int64{generation, generation + 1}
})
@@ -93,7 +94,7 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
}
generation := state.QueuedAt
if state.CompletedAt != generation || state.FirstRunAt != generation || state.DeadlineAt != generation ||
len(state.SourceTaskIDs) != 2 || len(state.CompanyIDs) != 2 ||
state.WorkerCount != int(generation%51) || len(state.SourceTaskIDs) != 2 || len(state.CompanyIDs) != 2 ||
state.SourceTaskIDs[0] != generation || state.CompanyIDs[0] != generation {
errCh <- fmt.Errorf("mixed lifecycle snapshot: %+v", state)
return
@@ -107,3 +108,19 @@ func TestTaskLifecycleSnapshotConcurrentConsistency(t *testing.T) {
t.Fatal(err)
}
}
func TestTaskLifecycleWorkerCountSnapshot(t *testing.T) {
task := &Task{WorkerCount: 2}
if got := taskDTO(task, "created").WorkerCount; got != 2 {
t.Fatalf("DTO worker_count=%d, want 2", got)
}
task.updateLifecycle(func(state *taskLifecycleState) {
state.WorkerCount = 7
})
if got := task.lifecycleSnapshot().WorkerCount; got != 7 {
t.Fatalf("snapshot worker_count=%d, want 7", got)
}
if got := taskDTO(task, "created").WorkerCount; got != 7 {
t.Fatalf("DTO worker_count=%d, want 7", got)
}
}
+3
View File
@@ -16,6 +16,8 @@ import (
"strings"
"syscall"
"time"
"github.com/Autumn-27/artex/internal/httpidentity"
)
const (
@@ -141,6 +143,7 @@ func (s *Server) sendRepeater(w http.ResponseWriter, r *http.Request) {
writeErr(w, http.StatusBadRequest, "构造请求失败")
return
}
httpidentity.Apply(req)
for key, value := range in.Headers {
name := strings.TrimSpace(key)
if name == "" {
+4 -6
View File
@@ -1285,6 +1285,7 @@ func (s *Server) listTasks(w http.ResponseWriter, r *http.Request) {
dtos := make([]TaskDTO, 0, len(list))
for _, t := range list {
dto := taskDTO(t, s.resolvedTaskStatus(t))
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(t)
applyTaskArchiveBlocker(&dto, archiveBlockers)
metric := metrics[t.ExpID]
dto.Tokens = tokenTotalDTO(metric.Tokens)
@@ -2068,6 +2069,7 @@ func (s *Server) getTask(w http.ResponseWriter, r *http.Request) {
return
}
dto := taskDTO(t, s.resolvedTaskStatus(t))
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(t)
archiveBlockers, _ := s.m.PG().TaskArchiveBlockers()
applyTaskArchiveBlocker(&dto, archiveBlockers)
writeJSON(w, 200, dto)
@@ -3740,7 +3742,7 @@ func (s *Server) settingsPayload() map[string]any {
"show_egress_ip": s.m.pg.GetBool(settingShowEgressIP, false),
"egress_probe_url": s.m.EgressProbeURL(),
"python_interpreter": strings.TrimSpace(pyStored), // 用户/自动设的值(空=用运行时检测)
"workers": s.m.Workers(), // 并发工作 agent 数(默认3);对之后启动的任务生效
"workers": s.m.Workers(), // 并发工作 agent 数(默认3);实时更新使用全局默认的任务
"worker_pools": s.m.WorkerPools(), // 非空时按池顺序绑定 worker 会话
"task_concurrency_enabled": concOn, // 任务并发上限开关(默认关)
"task_concurrency_limit": concLimit, // 同时运行任务上限(开启后默认5)
@@ -3833,7 +3835,7 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request) {
ShowEgressIP *bool `json:"show_egress_ip"` // 在顶栏显示直连/全局代理出口 IP;默认关
EgressProbeURL *string `json:"egress_probe_url"` // null=不改;空串=默认公共探测服务
PythonInterp *string `json:"python_interpreter"` // 自定义脚本工具的 python 解释器路径
Workers *int `json:"workers"` // 并发工作 agent 数(>0);对之后启动的任务生效
Workers *int `json:"workers"` // 并发工作 agent 数(1..50);实时更新使用全局默认的任务
WorkerPools *[]WorkerPool `json:"worker_pools"`
// 任务并发上限:同时「运行中」的任务数上限。关闭=不限;开启后新建任务超限则排队,有空位自动启动。
ConcurrencyEnabled *bool `json:"task_concurrency_enabled"`
@@ -3923,10 +3925,6 @@ func (s *Server) putSettings(w http.ResponseWriter, r *http.Request) {
// synchronizes workers to its total. An empty list clears pools; when workers
// is also present it sets the legacy count for the newly restored legacy mode.
if req.WorkerPools != nil {
if len(*req.WorkerPools) == 0 && req.Workers != nil && *req.Workers <= 0 {
writeErr(w, 400, "workers 必须 >0")
return
}
legacyWorkers := []int(nil)
if req.Workers != nil {
legacyWorkers = []int{*req.Workers}
+15 -5
View File
@@ -15,7 +15,7 @@ const (
maxTaskMetadataRequestSize = 16 << 10
)
// updateTaskMetadata updates editable metadata and the live asset-sharing policy.
// updateTaskMetadata updates editable metadata and live task policies.
func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
taskID := r.PathValue("id")
if _, ok := s.m.Task(taskID); !ok {
@@ -28,6 +28,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
Remark *string `json:"remark"`
Pinned *bool `json:"pinned"`
ShareGlobalAssets *bool `json:"share_global_assets"`
WorkerCount *int `json:"worker_count"`
}
if err := decode(r, &request); err != nil {
var tooLarge *http.MaxBytesError
@@ -38,8 +39,12 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
}
return
}
if request.Name == nil && request.Pinned == nil && request.Remark == nil && request.ShareGlobalAssets == nil {
writeErr(w, http.StatusBadRequest, "至少需要提供 name、pinned、remark 或 share_global_assets")
if request.Name == nil && request.Pinned == nil && request.Remark == nil && request.ShareGlobalAssets == nil && request.WorkerCount == nil {
writeErr(w, http.StatusBadRequest, "至少需要提供 name、pinned、remark、share_global_assets 或 worker_count")
return
}
if request.WorkerCount != nil && (*request.WorkerCount < 0 || *request.WorkerCount > db.MaxTaskWorkerCount) {
writeErr(w, http.StatusBadRequest, fmt.Sprintf("worker_count 必须在 0 到 %d 之间", db.MaxTaskWorkerCount))
return
}
if request.Name != nil {
@@ -63,7 +68,7 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
}
request.Remark = &remark
}
task, err := s.m.UpdateTaskMetadata(taskID, db.TaskPatch{Name: request.Name, Pinned: request.Pinned, Remark: request.Remark, ShareGlobalAssets: request.ShareGlobalAssets})
task, err := s.m.UpdateTaskMetadata(taskID, db.TaskPatch{Name: request.Name, Pinned: request.Pinned, Remark: request.Remark, ShareGlobalAssets: request.ShareGlobalAssets, WorkerCount: request.WorkerCount})
if err != nil {
writeErr(w, http.StatusInternalServerError, err.Error())
return
@@ -72,5 +77,10 @@ func (s *Server) updateTaskMetadata(w http.ResponseWriter, r *http.Request) {
writeErr(w, http.StatusNotFound, "task not found")
return
}
writeJSON(w, http.StatusOK, taskDTO(task, s.resolvedTaskStatus(task)))
if request.WorkerCount != nil {
s.engine.ReconcileWorkers(task)
}
dto := taskDTO(task, s.resolvedTaskStatus(task))
dto.WorkerActualCount, dto.WorkerTargetCount = s.engine.WorkerCounts(task)
writeJSON(w, http.StatusOK, dto)
}
+108
View File
@@ -159,3 +159,111 @@ func TestConversationBatchDeleteReportsMissing(t *testing.T) {
t.Fatalf("missing conversation %s must include an error", fmt.Sprint(missing))
}
}
func TestTaskMetadataPatchWorkerCount(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Skipf("postgres unavailable (%v) — skipping", err)
}
defer m.Close()
task, err := m.CreateTask("worker count patch", "goal", nil, 0, 0)
if err != nil {
t.Fatal(err)
}
taskID, _ := strconv.ParseInt(task.ID, 10, 64)
defer func() { _ = m.pg.DeleteTask(taskID) }()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
s := &Server{m: m, engine: NewEngine(m), ctx: ctx}
mux := http.NewServeMux()
mux.HandleFunc("PATCH /api/tasks/{id}", s.updateTaskMetadata)
mux.HandleFunc("GET /api/tasks/{id}", s.getTask)
mux.HandleFunc("GET /api/tasks", s.listTasks)
request := func(method, path, body string) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequest(method, path, strings.NewReader(body))
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
return rec
}
assertCounts := func(body []byte, want int) {
t.Helper()
var dto struct {
WorkerCount int `json:"worker_count"`
WorkerActualCount *int `json:"worker_actual_count"`
WorkerTargetCount *int `json:"worker_target_count"`
}
if err := json.Unmarshal(body, &dto); err != nil {
t.Fatal(err)
}
actual, target := s.engine.WorkerCounts(task)
if dto.WorkerCount != want || dto.WorkerActualCount == nil || dto.WorkerTargetCount == nil {
t.Fatalf("missing or incorrect worker counts: %s", body)
}
if *dto.WorkerActualCount != actual || *dto.WorkerTargetCount != target {
t.Fatalf("worker counts=%d/%d, engine=%d/%d", *dto.WorkerActualCount, *dto.WorkerTargetCount, actual, target)
}
if actual != 0 || (want > 0 && target != want) {
t.Fatalf("inactive task worker counts=%d/%d, want actual=0 and configured target=%d", actual, target, want)
}
}
for _, count := range []int{1, 50, 0} {
rec := request(http.MethodPatch, "/api/tasks/"+task.ID, fmt.Sprintf(`{"worker_count":%d}`, count))
if rec.Code != http.StatusOK {
t.Fatalf("patch worker_count=%d status=%d body=%s", count, rec.Code, rec.Body.String())
}
assertCounts(rec.Body.Bytes(), count)
if got := task.lifecycleSnapshot().WorkerCount; got != count {
t.Fatalf("in-memory worker count=%d, want %d", got, count)
}
persisted, err := m.pg.GetTask(taskID)
if err != nil || persisted == nil || persisted.WorkerCount != count {
t.Fatalf("persisted task=%+v err=%v, want worker_count=%d", persisted, err, count)
}
rec = request(http.MethodGet, "/api/tasks/"+task.ID, "")
if rec.Code != http.StatusOK {
t.Fatalf("get task status=%d body=%s", rec.Code, rec.Body.String())
}
assertCounts(rec.Body.Bytes(), count)
}
for _, body := range []string{
`{"worker_count":-1,"name":"must not commit"}`,
`{"worker_count":51,"name":"must not commit"}`,
`{"worker_count":1.5}`,
`{"worker_count":"2"}`,
`{"worker_count":null}`,
} {
rec := request(http.MethodPatch, "/api/tasks/"+task.ID, body)
if rec.Code != http.StatusBadRequest {
t.Fatalf("invalid patch %s status=%d body=%s", body, rec.Code, rec.Body.String())
}
}
state := task.lifecycleSnapshot()
persisted, err := m.pg.GetTask(taskID)
if err != nil || persisted == nil || persisted.WorkerCount != 0 || persisted.Name != state.Name || state.Name == "must not commit" || state.WorkerCount != 0 {
t.Fatalf("invalid patch changed state: memory=%+v persisted=%+v err=%v", state, persisted, err)
}
rec := request(http.MethodGet, "/api/tasks", "")
if rec.Code != http.StatusOK {
t.Fatalf("list tasks status=%d body=%s", rec.Code, rec.Body.String())
}
var list struct {
Tasks []json.RawMessage `json:"tasks"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &list); err != nil {
t.Fatal(err)
}
for _, raw := range list.Tasks {
var dto struct {
ID string `json:"id"`
}
if err := json.Unmarshal(raw, &dto); err != nil {
t.Fatal(err)
}
if dto.ID == task.ID {
assertCounts(raw, 0)
return
}
}
t.Fatal("updated task missing from list")
}
+98 -1
View File
@@ -26,6 +26,7 @@ func TestExpandWorkerSlotsLegacy(t *testing.T) {
{"configured", 2, 2},
{"zero defaults", 0, 3},
{"negative defaults", -1, 3},
{"oversized caps", db.MaxTaskWorkerCount + 1, db.MaxTaskWorkerCount},
} {
t.Run(tc.name, func(t *testing.T) {
slots, err := ExpandWorkerSlots(nil, tc.count)
@@ -170,6 +171,13 @@ func TestWorkerPoolSettingsRoundTrip(t *testing.T) {
if clearResult.Code != http.StatusOK || m.Workers() != 4 || len(m.WorkerPools()) != 0 {
t.Fatalf("PUT legacy clear status=%d body=%s workers=%d pools=%+v", clearResult.Code, clearResult.Body.String(), m.Workers(), m.WorkerPools())
}
for _, body := range []string{`{"workers":51}`, `{"workers":51,"worker_pools":[]}`, `{"workers":0}`} {
invalid := httptest.NewRecorder()
s.putSettings(invalid, httptest.NewRequest(http.MethodPut, "/api/settings", strings.NewReader(body)))
if invalid.Code != http.StatusBadRequest || m.Workers() != 4 || len(m.WorkerPools()) != 0 {
t.Fatalf("invalid legacy workers %s status=%d body=%s workers=%d", body, invalid.Code, invalid.Body.String(), m.Workers())
}
}
res = httptest.NewRecorder()
s.getSettings(res, httptest.NewRequest(http.MethodGet, "/api/settings", nil))
var cleared map[string]any
@@ -280,7 +288,7 @@ func TestPinnedWorkerLoopDoesNotConsultTaskRouterAndReportsFailureOnce(t *testin
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
e.workerLoop(ctx, task, WorkerSlot{Name: "work#4", ProfileID: 12})
e.workerLoop(ctx, task, WorkerSlot{Name: "work#4", ProfileID: 12}, &workerSlotRuntime{idleCtx: ctx})
close(done)
}()
time.Sleep(100 * time.Millisecond)
@@ -391,3 +399,92 @@ func TestDetachedIntentPinnedOwnerUsesPinnedResolver(t *testing.T) {
t.Fatalf("intent changed after unavailable resolver: node=%+v err=%v", node, err)
}
}
func TestWorkerSettingsDirectSettersNotifyAfterPersistence(t *testing.T) {
m, err := NewManager(t.TempDir(), "")
if err != nil {
t.Skipf("postgres unavailable: %v", err)
}
defer m.Close()
oldPools, poolsSet, _ := m.pg.GetSetting(settingWorkerPools)
oldWorkers, workersSet, _ := m.pg.GetSetting(settingWorkers)
defer func() {
for key, saved := range map[string]struct {
value string
set bool
}{
settingWorkerPools: {oldPools, poolsSet},
settingWorkers: {oldWorkers, workersSet},
} {
if saved.set {
_ = m.pg.SetSetting(key, saved.value)
} else {
_, _ = m.pg.Exec("DELETE FROM settings WHERE key=$1", key)
}
}
}()
profileID, err := m.pg.SaveProfile(&db.LLMProfile{Name: "worker-notify-test", Format: "openai", Model: "worker-notify-test", APIKey: "worker-notify-test-key"})
if err != nil {
t.Fatal(err)
}
defer m.pg.DeleteProfile(profileID)
var notifications int
wantWorkers, wantPools := 0, 0
callback := func() {
notifications++
// Re-enter Manager APIs: callbacks must run after the write and outside mu.
if got := m.Workers(); got != wantWorkers {
t.Errorf("callback observed workers=%d, want %d", got, wantWorkers)
}
if got := len(m.WorkerPools()); got != wantPools {
t.Errorf("callback observed %d pools, want %d", got, wantPools)
}
m.SetWorkerConfigChangedCallback(nil)
}
m.SetWorkerConfigChangedCallback(callback)
check := func(want int) {
t.Helper()
if notifications != want {
t.Fatalf("notifications=%d, want %d", notifications, want)
}
}
wantWorkers, wantPools = 50, len(m.WorkerPools())
if err := m.SetWorkers(50); err != nil {
t.Fatal(err)
}
check(1)
m.SetWorkerConfigChangedCallback(callback)
for _, count := range []int{-1, 0, 51} {
if err := m.SetWorkers(count); err == nil {
t.Fatalf("SetWorkers(%d) accepted", count)
}
if err := m.SetWorkerPools(nil, count); err == nil {
t.Fatalf("SetWorkerPools(nil, %d) accepted", count)
}
check(1)
}
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: 9223372036854775807, Workers: 1}}); err == nil {
t.Fatal("missing profile accepted")
}
check(1)
if got := m.Workers(); got != 50 {
t.Fatalf("rejected worker settings changed persisted workers=%d, want 50", got)
}
wantWorkers, wantPools = 2, 1
if err := m.SetWorkerPools([]WorkerPool{{ProfileID: profileID, Workers: 2}}); err != nil {
t.Fatal(err)
}
check(2)
m.SetWorkerConfigChangedCallback(callback)
wantWorkers, wantPools = 3, 0
if err := m.SetWorkerPools(nil, 3); err != nil {
t.Fatal(err)
}
check(3)
if err := m.pg.SetSetting(settingWorkers, "1000"); err != nil {
t.Fatal(err)
}
if got := m.Workers(); got != db.MaxTaskWorkerCount {
t.Fatalf("old oversized workers=%d, want %d", got, db.MaxTaskWorkerCount)
}
}
+52 -11
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
harvest_static.py <BASE_URL> <OUTDIR>
harvest_static.py <BASE_URL> <OUTDIR> [--user-agent VALUE]
参考模板 — 非通用成品。执行前须按目标站点调整,常见改动:
- extract_endpoints 正则(endpoint 方言)
@@ -16,16 +16,54 @@ Static SPA bundle harvester. Framework-agnostic; tuned for webpack + Vite.
Stdlib only. TLS verification is disabled (recon against self-signed/internal hosts).
"""
import sys, os, re, ssl, json, urllib.request, urllib.parse
import sys, os, re, ssl, secrets, json, argparse, urllib.request, urllib.parse
from concurrent.futures import ThreadPoolExecutor
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
UA = "Mozilla/5.0 (spa-api-recon)"
USER_AGENTS = (
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
)
def fetch(url, binary=False):
def user_agent(value=None):
"""Explicit value > environment > random sample; explicit empty omits UA."""
explicit = value is not None
if value is None:
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
value = value.strip(" ")
if not all(32 <= ord(c) <= 126 for c in value):
return secrets.choice(USER_AGENTS)
if value or explicit:
return value
return secrets.choice(USER_AGENTS)
def fetch(url, binary=False, user_agent_value=None):
try:
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req, context=CTX, timeout=25) as r:
ua = user_agent(user_agent_value)
req = urllib.request.Request(url, headers={"User-Agent": ua} if ua else {})
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
with opener.open(req, timeout=25) as r:
data = r.read()
return (data if binary else data.decode("utf-8", "ignore")), r.status
except Exception as e:
@@ -112,14 +150,17 @@ def extract_routes(js_text):
# ---- main -------------------------------------------------------------------
def main():
if len(sys.argv) < 3:
print("usage: harvest_static.py <BASE_URL> <OUTDIR>"); sys.exit(1)
base, outdir = sys.argv[1], sys.argv[2]
ap = argparse.ArgumentParser()
ap.add_argument("base"); ap.add_argument("outdir")
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
args = ap.parse_args()
base, outdir = args.base, args.outdir
ua = user_agent(args.user_agent)
if not base.startswith("http"): base = "https://" + base
jsdir = os.path.join(outdir, "js"); os.makedirs(jsdir, exist_ok=True)
print(f"[*] entry: {base}")
html, status = fetch(base)
html, status = fetch(base, user_agent_value=ua)
open(os.path.join(outdir, "index.html"), "w").write(html)
origin = origin_of(base)
@@ -137,7 +178,7 @@ def main():
if not fn.endswith(".js"): return None
lp = os.path.join(jsdir, fn)
if url in have: return have[url]
data, st = fetch(url, binary=True)
data, st = fetch(url, binary=True, user_agent_value=ua)
if st == 200 and data and not data[:15].lstrip().startswith(b"<"):
open(lp, "wb").write(data); have[url] = lp; return lp
return None
+47 -7
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4]
spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4] [--user-agent VALUE]
参考模板 — 非通用成品。执行前须按目标调整 --exclude、cookie、depth/max 等同域策略。
@@ -14,20 +14,58 @@ the same origin and extracts:
Stdlib only. Same-origin, bounded, polite. Provide --cookie for an authed crawl.
"""
import sys, os, re, ssl, argparse, urllib.request, urllib.parse, time
import sys, os, re, ssl, secrets, argparse, urllib.request, urllib.parse, time
from html.parser import HTMLParser
from collections import deque
CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
UA = "Mozilla/5.0 (spa-api-recon spider)"
USER_AGENTS = (
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
)
def user_agent(value=None):
"""Explicit value > environment > random sample; explicit empty omits UA."""
explicit = value is not None
if value is None:
value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
value = value.strip(" ")
if not all(32 <= ord(c) <= 126 for c in value):
return secrets.choice(USER_AGENTS)
if value or explicit:
return value
return secrets.choice(USER_AGENTS)
SKIP_EXT = re.compile(r'\.(css|png|jpe?g|gif|svg|ico|woff2?|ttf|pdf|zip|mp4|webp|js)(\?|$)', re.I)
API_HINT = re.compile(r'/(?:api|rest|service|ajax|action|do|rpc|graphql|v\d|admin|backend)\b', re.I)
def fetch(url, cookie):
h = {"User-Agent": UA}
def fetch(url, cookie, user_agent_value=None):
ua = user_agent(user_agent_value)
h = {"User-Agent": ua} if ua else {}
if cookie: h["Cookie"] = cookie
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=h), context=CTX, timeout=20) as r:
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
opener.addheaders = [] # Do not reintroduce Python-urllib's default UA.
with opener.open(urllib.request.Request(url, headers=h), timeout=20) as r:
ct = r.headers.get("Content-Type", "")
if "html" not in ct and "xml" not in ct: return "", ct
return r.read().decode("utf-8", "ignore"), ct
@@ -70,7 +108,9 @@ def main():
ap.add_argument("--cookie", default=""); ap.add_argument("--max", type=int, default=200)
ap.add_argument("--depth", type=int, default=4); ap.add_argument("--delay", type=float, default=0.1)
ap.add_argument("--include", default=""); ap.add_argument("--exclude", default="logout|signout|delete|remove|destroy")
ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
args = ap.parse_args()
ua = user_agent(args.user_agent)
base = args.base if args.base.startswith("http") else "https://" + args.base
os.makedirs(args.outdir, exist_ok=True)
origin = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(base))
@@ -82,7 +122,7 @@ def main():
n = 0
while q and n < args.max:
url, d = q.popleft(); n += 1
html, ct = fetch(url, args.cookie)
html, ct = fetch(url, args.cookie, user_agent_value=ua)
if args.delay: time.sleep(args.delay)
if not html: continue
p = Page()
+124
View File
@@ -0,0 +1,124 @@
"""Offline regressions: python3 -m unittest discover -s skills/api-recon/scripts."""
import contextlib
import http.server
import importlib.util
import os
from pathlib import Path
import re
import subprocess
import sys
import tempfile
import threading
import unittest
from unittest.mock import patch
SCRIPTS = Path(__file__).resolve().parent
# Keep the offline fixtures independent of the runner's outbound/recording proxy.
DIRECT_ENV = {"no_proxy": "*", "NO_PROXY": "*"}
def load_script(name):
spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
@contextlib.contextmanager
def capture_server():
seen = []
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
seen.append((self.path, self.headers.get("User-Agent"), self.headers.get("Cookie")))
if self.path == "/start":
self.send_response(302)
self.send_header("Location", "/page")
self.end_headers()
return
self.send_response(200)
self.send_header("Content-Type", "text/html")
self.end_headers()
self.wfile.write(b'<h1>Page</h1><script src="/app.js"></script><form action="/submit"></form>')
def log_message(self, *args):
pass
with http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
thread = threading.Thread(target=server.serve_forever)
thread.start()
try:
yield f"http://127.0.0.1:{server.server_port}", seen
finally:
server.shutdown()
thread.join()
class UserAgentTests(unittest.TestCase):
def assert_identity(self, seen, expected, samples):
self.assertTrue(seen)
if expected == "random":
self.assertIn(seen[0][1], samples)
expected = seen[0][1]
self.assertTrue(all(ua == expected for _, ua, _ in seen), seen)
def test_random_selection_and_explicit_override(self):
go_source = (SCRIPTS.parents[2] / "internal/httpidentity/useragent.go").read_text()
go_pool = tuple(re.findall(r'^\s*"(Mozilla/5\.0 [^"]+)",$', go_source, re.MULTILINE))
self.assertEqual(len(go_pool), 20)
self.assertEqual(len(set(go_pool)), 20)
for name in ("harvest_static", "spider_mpa"):
module = load_script(name)
self.assertEqual(module.USER_AGENTS, go_pool)
for sample in module.USER_AGENTS:
with self.subTest(script=name, sample=sample):
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": ""}), patch.object(module.secrets, "choice", return_value=sample) as choose:
self.assertEqual(module.user_agent(), sample)
choose.assert_called_once_with(module.USER_AGENTS)
with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": "Configured-UA"}), patch.object(module.secrets, "choice") as choose:
self.assertEqual(module.user_agent(), "Configured-UA")
self.assertEqual(module.user_agent("Override-UA"), "Override-UA")
self.assertEqual(module.user_agent(""), "")
choose.assert_not_called()
def test_fetch_and_redirect(self):
for name in ("harvest_static", "spider_mpa"):
module = load_script(name)
for value, expected in (("", "random"), (" Captured-UA/1.0 ", "Captured-UA/1.0"),
("ok\r\nX-Injected: yes", "random"), ("客户端", "random")):
with self.subTest(script=name, value=value), capture_server() as (base, seen):
with patch.dict(os.environ, {**DIRECT_ENV, "ARTEX_HTTP_USER_AGENT": value}):
if name == "harvest_static":
body, status = module.fetch(base + "/start")
self.assertEqual(status, 200)
else:
body, _ = module.fetch(base + "/start", "session=sample")
self.assertTrue(all(cookie == "session=sample" for _, _, cookie in seen))
self.assertIn("Page", body)
self.assertEqual([path for path, _, _ in seen], ["/start", "/page"])
self.assert_identity(seen, expected, module.USER_AGENTS)
def test_cli_precedence_and_outputs(self):
for name in ("harvest_static", "spider_mpa"):
module = load_script(name)
for env_ua, flags, expected in (("", [], "random"),
("Environment-UA/1.0", [], "Environment-UA/1.0"),
("Environment-UA/1.0", ["--user-agent", "Override-UA/1.0"], "Override-UA/1.0"),
("Environment-UA/1.0", ["--user-agent", ""], None)):
with self.subTest(script=name, flags=flags), capture_server() as (base, seen):
with tempfile.TemporaryDirectory() as outdir:
env = dict(os.environ, **DIRECT_ENV, ARTEX_HTTP_USER_AGENT=env_ua)
command = [sys.executable, str(SCRIPTS / f"{name}.py"), base + "/start", outdir] + flags
if name == "spider_mpa":
command += ["--max", "1", "--delay", "0"]
result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=15)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertGreaterEqual(len(seen), 2)
self.assert_identity(seen, expected, module.USER_AGENTS)
artifact = "api_static.txt" if name == "harvest_static" else "forms.txt"
self.assertTrue((Path(outdir) / artifact).exists())
if __name__ == "__main__":
unittest.main()
+17
View File
@@ -412,6 +412,20 @@ type sink struct {
t *Traffic
}
// Preserve an absent User-Agent when go-mitmproxy creates its outbound Go
// request. An explicit empty value suppresses net/http's automatic default.
func (s *sink) Requestheaders(f *mproxy.Flow) {
if f.Request == nil {
return
}
if f.Request.Header == nil {
f.Request.Header = make(http.Header)
}
if _, present := f.Request.Header["User-Agent"]; !present {
f.Request.Header.Set("User-Agent", "")
}
}
func (s *sink) Response(f *mproxy.Flow) {
if f.Request == nil || f.Response == nil {
return
@@ -683,6 +697,9 @@ func headerLines(h map[string][]string) string {
func requestHeaderLines(req *mproxy.Request) string {
headers := req.Header.Clone()
headers.Del("Host")
if headers.Get("User-Agent") == "" {
headers.Del("User-Agent") // Empty suppression value is not sent on the wire.
}
host := req.URL.Host
if raw := req.Raw(); raw != nil && strings.TrimSpace(raw.Host) != "" {
host = raw.Host
+48
View File
@@ -0,0 +1,48 @@
package traffic
import (
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
)
func TestProxyPreservesUserAgent(t *testing.T) {
for _, ua := range []string{"", "Captured-UA/1.0"} {
t.Run(ua, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.UserAgent(); got != ua {
t.Errorf("forwarded User-Agent = %q, want %q", got, ua)
}
if _, present := r.Header["User-Agent"]; ua == "" && present {
t.Error("proxy reintroduced a User-Agent")
}
}))
defer server.Close()
u, _ := url.Parse(server.URL)
flow := &mproxy.Flow{Request: &mproxy.Request{URL: u, Header: make(http.Header)}}
if ua != "" {
flow.Request.Header.Set("User-Agent", ua)
}
(&sink{}).Requestheaders(flow)
// go-mitmproxy rebuilds the outbound request by adding every value.
req, _ := http.NewRequest(http.MethodGet, server.URL, nil)
for key, values := range flow.Request.Header {
for _, value := range values {
req.Header.Add(key, value)
}
}
resp, err := server.Client().Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
if ua == "" && strings.Contains(requestHeaderLines(flow.Request), "User-Agent:") {
t.Error("recording contains a synthetic header that was not sent")
}
})
}
}
@@ -24,6 +24,8 @@ import {
AlertDialogTrigger,
} from "@/components/ui/alert-dialog";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import {
Popover,
PopoverContent,
@@ -40,6 +42,7 @@ import { Tooltip, TooltipContent, TooltipTrigger } from "@/components/ui/tooltip
import { type UITranslate, useUITranslation } from "@/i18n/use-ui-translation";
import { api } from "@/lib/api";
import type { LLMProfile, Task } from "@/lib/types";
import { MAX_TASK_WORKER_COUNT, parseWorkerCountInput } from "@/lib/types";
import { AssetsTab } from "./_tabs/assets-tab";
import { BroadcastTab } from "./_tabs/broadcast-tab";
@@ -212,6 +215,87 @@ function TaskLLMControl({ task, profiles, onUpdated }: { task: Task; profiles: L
);
}
function TaskWorkerControl({ task, onUpdated }: { task: Task; onUpdated: (task: Task) => void }) {
const { ui } = useUITranslation();
const [open, setOpen] = React.useState(false);
const [draft, setDraft] = React.useState(String(task.worker_count ?? 0));
const [saving, setSaving] = React.useState(false);
const actual = task.worker_actual_count;
const target = task.worker_target_count;
const draining = actual !== undefined && target !== undefined && actual > target;
const save = async (event: React.FormEvent<HTMLFormElement>) => {
event.preventDefault();
const count = parseWorkerCountInput(draft);
if (count === null) {
toast.error(ui("taskUI.invalidWorkers", { max: MAX_TASK_WORKER_COUNT }));
return;
}
setSaving(true);
try {
const updated = await api.updateTask(task.id, { worker_count: count });
onUpdated(updated);
setOpen(false);
toast.success(ui("taskWorkers.saved"));
} catch (error) {
toast.error(ui("taskWorkers.saveFailed", { message: (error as Error).message }));
} finally {
setSaving(false);
}
};
return (
<div className="flex flex-wrap items-center gap-x-3 gap-y-1 text-xs text-muted-foreground">
<span className="tabular-nums">{ui("taskWorkers.counts", { actual: actual ?? "—", target: target ?? "—" })}</span>
<Popover
open={open}
onOpenChange={(next) => {
setOpen(next);
if (next) setDraft(String(task.worker_count ?? 0));
}}
>
<PopoverTrigger asChild>
<Button size="sm" variant="outline" className="h-7 text-xs">
{ui("taskWorkers.edit")}
</Button>
</PopoverTrigger>
<PopoverContent align="start" className="w-[min(25rem,calc(100vw-2rem))] gap-4 p-4">
<PopoverHeader>
<PopoverTitle>{ui("taskWorkers.edit")}</PopoverTitle>
<PopoverDescription>{ui("taskWorkers.help")}</PopoverDescription>
</PopoverHeader>
<form onSubmit={save} className="space-y-3">
<div className="space-y-2">
<Label htmlFor="task-worker-count">{ui("taskWorkers.label")}</Label>
<Input
id="task-worker-count"
type="number"
min={0}
max={MAX_TASK_WORKER_COUNT}
step={1}
value={draft}
disabled={saving}
onChange={(event) => setDraft(event.target.value)}
/>
</div>
<p className="text-xs text-muted-foreground">{ui("taskWorkers.transitionHelp")}</p>
<div className="flex justify-end gap-2">
<Button type="button" variant="outline" size="sm" onClick={() => setOpen(false)} disabled={saving}>
{ui("screen.3fd47edce45b36")}
</Button>
<Button type="submit" size="sm" disabled={saving}>
{saving && <Spinner data-icon="inline-start" />}
{ui(saving ? "screen.a1a1720d3eb58a" : "screen.a3030bf8f16dc6")}
</Button>
</div>
</form>
</PopoverContent>
</Popover>
{draining && <span className="text-amber-600">{ui("taskWorkers.draining")}</span>}
</div>
);
}
function TaskDetailInner() {
const { ui, liveUI } = useUITranslation();
@@ -437,6 +521,14 @@ function TaskDetailInner() {
</Button>
</div>
<p className="truncate text-xs text-muted-foreground">{task.goal}</p>
<TaskWorkerControl
key={task.id}
task={task}
onUpdated={(updated) => {
setTask(updated);
load();
}}
/>
{task.paused && task.pause_reason && (
<div className="flex flex-wrap items-center gap-2 text-xs text-amber-600">
<span>
+2 -2
View File
@@ -1731,8 +1731,8 @@ export default function SystemSettingsPage() {
</CardTitle>
<CardDescription>
{t("settings.workersHelpStart")}
<b>{t("settings.futureTasksOnly")}</b>
{t("settings.runningTasksUnchanged")}
<b>{t("settings.workersLiveUpdate")}</b>
{t("settings.workersDrainHelp")}
</CardDescription>
</CardHeader>
<CardContent className="flex flex-col gap-3">
+12 -6
View File
@@ -12,14 +12,14 @@ export const settingsMessages = {
"settings.workerLimit": ["Worker 总数不能超过 50", "The total number of Workers cannot exceed 50"],
"settings.duplicatePoolProfile": ["不能重复选择同一个 LLM 配置", "Each LLM profile can only be selected once"],
"settings.workerPoolsSaved": [
"已保存多模型 Worker 池(对之后启动的任务生效)",
"Model pools saved; applies to tasks started afterward",
"已保存多模型 Worker 池,跟随全局配置的任务已热更新",
"Model pools saved; tasks following the global configuration have been updated live",
],
"settings.saveFailed": ["保存失败:{{message}}", "Could not save: {{message}}"],
"settings.positiveConcurrency": ["并发数必须是大于 0 的整数", "Concurrency must be a positive whole number"],
"settings.workersSaved": [
"已保存并发工作 agent 数(对之后启动的任务生效)",
"Worker count saved; applies to tasks started afterward",
"已保存工作并发,跟随全局配置的任务已热更新",
"Worker count saved; tasks following the global configuration have been updated live",
],
"settings.saveFailedPrefix": ["保存失败:", "Could not save: "],
"settings.pythonSaved": ["已保存 Python 解释器配置", "Python interpreter settings saved"],
@@ -427,8 +427,14 @@ export const settingsMessages = {
"每个任务并发运行的工作 agent 数量(默认 3)。数值越大并发探测越多、消耗也越高。修改后",
"The number of Workers running concurrently per task (default: 3). More Workers mean more parallel probes and higher usage. Changes ",
],
"settings.futureTasksOnly": ["对之后启动的任务生效", "apply to tasks started afterward"],
"settings.runningTasksUnchanged": [",正在运行的任务不受影响。", "; running tasks are unaffected."],
"settings.workersLiveUpdate": [
"即时更新跟随全局配置(worker_count = 0)的任务",
"apply immediately to tasks following the global configuration (worker_count = 0)",
],
"settings.workersDrainHelp": [
";单任务覆盖值不受影响。增容启动新 Worker loops;缩容等待在跑意图完成后退出,不会强杀,实际数量可能暂时高于目标。",
"; per-task overrides are unchanged. Scaling up starts new Worker loops; scaling down lets running intents finish before loops exit, without force-cancelling them. Actual counts may temporarily exceed target.",
],
"settings.workerPools": ["多模型 Worker 池", "Model pools"],
"settings.addPool": ["添加模型池", "Add model pool"],
"settings.unavailableProfile": ["配置 #{{id}} 已删除/不可用", "Profile #{{id}} deleted or unavailable"],
+17
View File
@@ -1,4 +1,21 @@
export const taskMessages = {
"taskWorkers.edit": ["调整 Worker 并发", "Adjust Worker concurrency"],
"taskWorkers.label": ["Worker 数量(0–50)", "Worker count (0–50)"],
"taskWorkers.counts": ["Worker 实际 / 目标:{{actual}} / {{target}}", "Workers actual / target: {{actual}} / {{target}}"],
"taskWorkers.help": [
"0 表示跟随全局 Worker 配置;1–50 为本任务覆盖值,保存后热更新,无需重启任务。",
"Use 0 to follow the global Worker configuration, or 1–50 to override it for this task. Saving applies changes live without restarting the task.",
],
"taskWorkers.transitionHelp": [
"实际数量是存活的 Worker loops(含缩容退出中的 loops),不是正在执行的意图数。增容会启动新 loops;缩容等待正在执行的意图完成后退出,不会强杀意图,因此实际数量可能暂时高于目标。",
"Actual counts live Worker loops, including draining loops, not in-flight intents. Scaling up starts new loops. Scaling down lets running intents finish before loops exit, without force-cancelling them, so actual may temporarily exceed target.",
],
"taskWorkers.draining": [
"缩容过渡中:等待在跑意图完成,不会强杀。",
"Scaling down: waiting for running intents to finish; they will not be force-cancelled.",
],
"taskWorkers.saved": ["Worker 并发已更新", "Worker concurrency updated"],
"taskWorkers.saveFailed": ["Worker 并发更新失败:{{message}}", "Could not update Worker concurrency: {{message}}"],
"taskFiles.tab": ["文件", "Files"],
"taskFiles.tree": ["目录树", "Directory tree"],
"taskFiles.recent": ["最近文件", "Recent files"],
+1 -1
View File
@@ -483,7 +483,7 @@ export const api = {
taskCategories: () => get<{ categories: TaskCategory[] }>("/task-categories").then((r) => arr(r.categories)),
updateTask: (
id: string,
input: { name?: string; pinned?: boolean; remark?: string; share_global_assets?: boolean },
input: { name?: string; pinned?: boolean; remark?: string; share_global_assets?: boolean; worker_count?: number },
) => patch<Task>(`/tasks/${id}`, input),
renameTask: (id: string, name: string) => patch<Task>(`/tasks/${id}`, { name }),
pinTask: (id: string, pinned: boolean) => patch<Task>(`/tasks/${id}`, { pinned }),
+12 -1
View File
@@ -58,6 +58,9 @@ export const tasks: Task[] = [
description: "Acme 官网与后台外部渗透(acme.com)",
goal: "拿到 acme.com 后台管理权限,确认可读取用户敏感数据。",
status: "running",
worker_count: 0,
worker_actual_count: 3,
worker_target_count: 3,
created_at: T("2026-07-24T09:12:00Z"),
created_unix: 1785489120,
active_run_seconds: 28800,
@@ -87,6 +90,9 @@ export const tasks: Task[] = [
description: "api.acme.com 越权与注入测试",
goal: "评估 api.acme.com 订单/用户接口的越权(IDOR)与注入风险。",
status: "running",
worker_count: 0,
worker_actual_count: 3,
worker_target_count: 3,
created_at: T("2026-07-25T14:05:00Z"),
created_unix: 1785592500,
active_run_seconds: 14400,
@@ -116,6 +122,9 @@ export const tasks: Task[] = [
description: "shop.acme.com 支付与订单链路",
goal: "评估支付与订单接口的越权、金额篡改与竞态风险。",
status: "paused",
worker_count: 0,
worker_actual_count: 3,
worker_target_count: 3,
created_at: T("2026-07-22T08:30:00Z"),
created_unix: 1785313800,
active_run_seconds: 21600,
@@ -144,6 +153,9 @@ export const tasks: Task[] = [
description: "对外暴露面侦察(VPN / 边界服务)",
goal: "识别 acme.com 对外暴露的可利用边界服务。",
status: "done",
worker_count: 0,
worker_actual_count: 0,
worker_target_count: 3,
created_at: T("2026-07-18T08:00:00Z"),
created_unix: 1784966400,
active_run_seconds: 86400,
@@ -2921,7 +2933,6 @@ export const trafficDetail: TrafficDetail = {
req: `GET /v1/orders?id=1002 HTTP/1.1
Host: api.acme.com
Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
User-Agent: ARTEX-worker/1.0
Accept: application/json
`,
+35 -3
View File
@@ -48,6 +48,7 @@ const mockInterceptHistory = structuredClone(D.interceptHistory);
const mockInterceptPending = structuredClone(D.interceptPending);
const mockInterceptDetails = structuredClone(D.interceptDetails);
const mockTasks = structuredClone(D.tasks);
const mockWorkerDrainUntil = new Map<string, number>();
const mockFindings = structuredClone(D.findings);
const mockLLMRecords = structuredClone(D.llmRecords);
const mockTaskTemplates = structuredClone(D.taskTemplates);
@@ -256,7 +257,33 @@ function mockTaskArchiveBlocker(taskID: string): string | undefined {
)?.id;
}
function reconcileMockWorkers(task: Task): void {
const target = task.worker_count || D.settings.workers || 3;
const previousTarget = task.worker_target_count;
task.worker_target_count = target;
if (task.status !== "running" && task.status !== "paused") {
task.worker_actual_count = 0;
mockWorkerDrainUntil.delete(task.id);
return;
}
const actual = task.worker_actual_count ?? target;
if (actual <= target) {
task.worker_actual_count = target;
mockWorkerDrainUntil.delete(task.id);
} else {
// Demo loops drain independently of in_flight; retain them across polls briefly.
if (!mockWorkerDrainUntil.has(task.id) || previousTarget !== target) {
mockWorkerDrainUntil.set(task.id, Date.now() + 10_000);
}
if (Date.now() >= (mockWorkerDrainUntil.get(task.id) ?? 0)) {
task.worker_actual_count = target;
mockWorkerDrainUntil.delete(task.id);
}
}
}
function publicMockTask(task: Task): Task {
reconcileMockWorkers(task);
const item = structuredClone(task);
item.numeric_id = mockTaskAssetIDs.get(task.id);
item.share_global_assets = task.share_global_assets ?? false;
@@ -1323,6 +1350,7 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
company_ids: companyIDs,
share_global_assets: b.share_global_assets === true,
};
reconcileMockWorkers(created);
// Validate the attached preview schedule before mutating task state.
const numericTaskID = nextMockTaskAssetID;
if (b.schedule && typeof b.schedule === "object") {
@@ -1527,8 +1555,10 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
const namePresent = has("name") && b.name !== null;
const remarkPresent = has("remark") && b.remark !== null;
const pinnedPresent = has("pinned") && b.pinned !== null;
if (!namePresent && !remarkPresent && !pinnedPresent && !sharingPresent)
throw new Error("至少需要提供 name、pinned、remark 或 share_global_assets");
const workerCountPresent = has("worker_count") && b.worker_count !== null;
const nextWorkerCount = workerCountPresent ? mockWorkerCount(b.worker_count) : task.worker_count;
if (!namePresent && !remarkPresent && !pinnedPresent && !sharingPresent && !workerCountPresent)
throw new Error("至少需要提供 name、pinned、remark、share_global_assets 或 worker_count");
let nextName = task.name;
let nextRemark = task.remark;
let nextPinned = task.pinned;
@@ -1550,8 +1580,9 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
task.remark = nextRemark;
task.pinned = nextPinned;
task.pinned_at = nextPinnedAt;
task.worker_count = nextWorkerCount;
sortMockTasks();
return structuredClone(task);
return publicMockTask(task);
}
if (seg[0] === "tasks" && seg.length === 2 && m === "DELETE") {
const id = seg[1];
@@ -2463,6 +2494,7 @@ function route(m: string, path: string, seg: string[], q: URLSearchParams, b: Re
next.workers = next.worker_pools.reduce((sum, pool) => sum + Number(pool.workers), 0);
}
Object.assign(D.settings, next);
for (const task of mockTasks) reconcileMockWorkers(task);
return next;
}
if (path === "/settings/basic-auth" && m === "GET") return { enabled: false, username: "", password_set: false };
+3 -1
View File
@@ -26,6 +26,8 @@ export interface Task {
id: string;
remark?: string;
worker_count?: number; // 本任务 Worker 数;0/缺省=用全局默认
worker_actual_count?: number; // 当前存活的 Worker pool loops,含缩容退出中的 loops
worker_target_count?: number; // 当前期望的 Worker slots 数量
name?: string; // 可选任务名称;空/缺省=未命名,展示时回退到描述
category_id?: number;
category_name?: string;
@@ -1028,7 +1030,7 @@ export interface Settings {
show_egress_ip?: boolean; // 在顶栏显示直连 / 全局代理出口 IP;默认关闭
egress_probe_url?: string; // 自定义 IP 探测服务;空=默认公共回显服务
python_interpreter?: string; // 自定义脚本工具的 python 解释器路径(空=运行时检测)
workers?: number; // 并发工作 agent 数(默认3);对之后启动的任务生效
workers?: number; // 并发工作 agent 数(默认3);热更新跟随全局配置的任务
worker_pools: WorkerPool[];
// 任务并发上限:同时「运行中」的任务数上限。关闭=不限;开启后新建任务超限则排队,有空位自动启动。
task_concurrency_enabled?: boolean; // 默认 false