chore(release): prepare v0.3.30 release notes

全局出口代理配置为 socks5:// 时，SOCKS5 握手失败会刷出
"http: panic serving ...: nil pointer dereference"。

根因在依赖库 go-mitmproxy v1.9.2 的 internal/helper/proxy.go:40：
x/net 的 SOCKS5 DialContext 失败时返回 (nil, err)，库仍调用
conn.Close()，必然空指针。net/http 每连接 recover，进程不退出，
但每条 CONNECT 被直接掐断（客户端见到 EOF），且 panic 发生在
HTTPConnectError 回调之前，导致 reportUpstreamFailure 不触发，
代理池自愈与任务重连完全收不到失败信号。

上游已在 v1.9.4 修复（issue #110，commit 388f47a），本次升级
v1.9.2 -> v1.9.5。实测同一份 traffic 包：v1.9.2 下 CONNECT 返回
unexpected EOF 且回调不触发；v1.9.5 下返回 502 并正常触发失败回调。

连带升级 go-mitmproxy 要求的最低版本：x/net 0.61.0、x/crypto
0.58.0、x/text 0.43.0、x/sys 0.49.0、x/mod 0.41.0、x/sync 0.24.0、
x/tools 0.51.0。

验证：go build ./... 与 go vet ./... 在 -mod=readonly 下通过；
按包隔离数据库逐包对比升级前后，失败集合完全一致（db、llmrec
为本次改动前既有失败），traffic/evidence/intercept/sidequestion/
guard/notify/server 全绿。

.env.example

@@ -13,3 +13,6 @@ ARTEX_LLM_PROVIDER=
 ARTEX_LLM_MODEL=
 ARTEX_LLM_BASE_URL=
 ARTEX_LLM_PROXY=
+
+# HTTP User-Agent；留空时从浏览器 UA 样本中随机抽选，填写则使用固定值
+ARTEX_HTTP_USER_AGENT=

CHANGELOG.md

@@ -4,8 +4,43 @@
 
 ## [Unreleased]
 
-- 新增 MCP 服务端：在「系统设置 → MCP 服务端」开启后，外部智能体可通过 `/mcp`（Streamable HTTP）查看任务与漏洞、创建任务、暂停/恢复任务、补充提示。新增访问令牌（权限范围、限定企业、任务限额、有效期、吊销）与调用审计。支持来源白名单（IP/CIDR，可配置可信反向代理），在令牌校验之前生效。默认关闭；`/mcp` 只认访问令牌，不经过也无法使用 HTTP Basic Auth。详见 README「MCP 服务端」。
+## [0.3.30] - 2026-11-14
+
+### 新增与改进
+
+- 新增 MCP 服务端：外部智能体可通过 `/mcp`（Streamable HTTP）查看任务与漏洞、创建任务、暂停/恢复任务及补充提示。提供访问令牌的权限范围、企业限制、任务限额、有效期和吊销，以及调用审计、来源 IP/CIDR 白名单与可信代理配置。默认关闭；该端点使用独立访问令牌认证，不使用 HTTP Basic Auth。
+- 统一出站 HTTP 请求标识：Go 请求与两个 Python 采集模板从 20 个标准桌面浏览器 User-Agent 中随机选择，保留环境变量和显式请求头覆盖，同一请求的重定向与重试复用选定值。此变化仅涉及 HTTP 请求头。
+
+### 修复
+
+- 命令拦截除匹配原始工具输入 JSON 外，也匹配解码后的嵌套字符串值，修复换行、制表符和转义引号导致危险命令漏拦的问题；保留原始 JSON 规则及非 JSON 输入的兼容行为。
+- 升级 `go-mitmproxy` v1.9.2 → v1.9.5，修复上游 SOCKS5 握手失败时的空指针 panic；失败时返回 502，并正常触发代理失败回调。
+- 修正 README 中前端构建产物的复制命令，避免目录嵌套并创建目标目录。
+
+## [0.3.29] - 2026-11-11
+
+### 新增与改进
+
 - 新增可选「无成果止损」：按实际运行时长、独立漏洞数量与最低等级做一次检查，未达标时自动收尾并显示「无成果停止」；暂停、排队及计划窗口外不计时，复活后重新计时，默认关闭。停止状态复用 `timeout`，已有 `on_task_timeout` 触发器仍会执行并可能消耗 token。
+- 任务详情新增只读文件浏览，支持目录导航、最近文件、文本与图片预览、单文件下载及目录 ZIP 下载；展示意图目录对应的标题和状态，并提供中英文界面。
+
+### 修复
+
+- 无成果检查与漏洞写入串行化，并在同一事务内重新计算独立漏洞数量，避免临界时刻已达到成果要求的任务被误停止。
+- 旁路问答摘要为空、被截断或过长时自动重试一次，并细分失败原因；失败摘要不写入缓存，重试计入原有调用次数、用量及超时限制。
+
+## [0.3.28] - 2026-11-10
+
+### 新增与改进
+
+- 新增英文界面和语言切换，覆盖任务、审批、设置、通知及计划任务等页面，并统一常用状态、日期与错误提示的翻译。
+- 流量页面展示完整流量 ID，支持按 ID 搜索及在 HTTP 环境复制；Agent 可直接使用该 ID 读取已有流量。
+- 漏洞统计与列表使用相同筛选条件，区分独立漏洞和原始上报记录数量。
+
+### 修复
+
+- Basic Auth 会话失效时在当前标签页恢复认证，避免后台请求反复弹出浏览器认证框，并保留应用登录状态。
+- 修复英文界面预览、覆盖度图谱生命周期及部分页面运行时错误，并为认证、出站探测等错误补充可翻译的错误码。
 
 ## [0.3.18] - 2026-10-12
 

HTTP_USER_AGENT.md

@@ -0,0 +1,87 @@
+# HTTP 请求标识排查
+
+本次检查覆盖仓库中的 Go HTTP 请求、Python / JavaScript 脚本、安装脚本、前端示例，以及实际使用的 norma v0.4.3 的 WebFetch / WebSearch 实现。排查范围包括 User-Agent 的大小写变体、请求头构造、代理转发、重定向及代理失败后的直连重试。
+
+## 已处理
+
+| 通道 | 排查结果与处理 |
+|---|---|
+| 内置 WebFetch | SDK 将 `norma/0.4` 写死在首发及重试请求。worker、planner、main、chat 通过统一 session 入口使用本地适配器；保留 SDK 的工具 schema、权限、静态资源限制、代理、CA 验证及 HTML / RECON 输出。 |
+| 资产自动补全 | 移除 `artex-enrich/1.0`，HTTP 探测使用统一策略。 |
+| 出口连通性探测 | 额外发现 `artex-egress-probe/1.0`，已移除并接入统一策略。 |
+| 静态采集 / 同源爬虫模板 | 移除 `spa-api-recon` / `spa-api-recon spider`，避免 urllib 自动填入自己的默认 UA；入口、下载、重定向及重试沿用同一配置。 |
+| HTTP 自定义工具 | 原先没有 ARTEX 专属 UA，但未设置时会由 Go 填充默认值。现采用统一策略，模板显式请求头优先。 |
+| 请求重放器 | 原先没有 ARTEX 专属 UA。现采用统一策略，用户显式请求头（包括空值）优先，保持原请求的配置能力。 |
+| 记录代理 | 转发缺失 UA 的请求时，Go 会补入默认客户端标识。现保留显式省略语义，录制中不展示内部用于抑制默认值的空头。保留客户端已选择的随机或固定 UA。 |
+| GitHub 自更新 | 额外发现查询版本、下载校验和与发布包共用的两处 `artex-selfupdate`。均已移除；与其他请求统一采用随机浏览器样本或显式配置的非空 UA，不含默认 ARTEX 名称。 |
+| 前端 mock | `ARTEX-worker/1.0` 仅为示例 HTTP 文本，不会发出请求；已删除该行，避免误导。 |
+
+## 配置
+
+默认采用随机 UA 池方案，Go 与两个独立 Python 模板内置相同的 20 个完整浏览器 UA 样本，覆盖 Chrome / Edge / Firefox / Safari 的标准桌面格式。随机选择完整样本，允许重复，不拼接随机版本，不调用 AI 服务。
+
+2026-11-11 核对并更新样本，采用主流浏览器近期稳定版本的标准格式：
+
+| 浏览器 | 版本 | 平台 | 数量 |
+|---|---|---|---|
+| Chrome | 154 / 155 / 156 / 157 | Windows / macOS / Linux | 12 |
+| Edge | 156 / 157 | Windows / macOS | 4 |
+| Firefox | 159 | Windows / macOS / Linux | 3 |
+| Safari | 27.1 | macOS | 1 |
+
+保留浏览器正常的 UA 缩减字段（如 Chrome 的 `0.0.0` 和 macOS 的冻结版本），不添加工具名称、随机设备型号、扩展标记或自定义后缀。“常见”指主流浏览器的标准 UA 格式，不代表已测量这 20 项在目标站点中的使用占比；浏览器类型和版本仍可从 UA 中读出。
+
+格式与版本参考：
+
+- [Chromium UA Reduction](https://www.chromium.org/updates/ua-reduction/)
+- [Chrome 稳定版发布记录](https://chromereleases.googleblog.com/search/label/Stable%20updates)
+- [Edge UA 格式](https://learn.microsoft.com/en-us/microsoft-edge/web-platform/user-agent-guidance)
+- [Edge 稳定版发布记录](https://learn.microsoft.com/en-us/deployedge/microsoft-edge-relnote-stable-channel)
+- [Firefox UA 格式](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent/Firefox)
+- [Firefox 159 发布记录](https://www.firefox.com/en-US/firefox/159.0/releasenotes/)
+- [Safari 27.1 发布记录](https://developer.apple.com/documentation/safari-release-notes/safari-27_1-release-notes)
+
+- Go 每次 HTTP 逻辑请求抽选一次；WebFetch 每次工具调用重新抽选，重定向、代理请求及失败后的直连重试复用同一值。
+- Python 脚本每次运行抽选一次，入口、并发资源下载及重试复用同一值；独立调用 `fetch()` 时每次调用单独选择。
+- `ARTEX_HTTP_USER_AGENT` 非空且合法时覆盖随机值；留空、仅空格或非法值回退随机池。Go 使用系统加密随机源；若随机源出错，保留池中首个非空有效样本。
+- Python 的 `--user-agent VALUE` 优先于环境变量，`--user-agent ''` 仍显式省略；自定义 HTTP 工具 / 重放器显式提供的 UA（包括空值）同样优先，保留原请求的控制能力。
+- 固定值只接受可打印 ASCII，去除首尾空格；控制字符和非 ASCII 值回退随机池，防止头注入。
+
+Go 服务重启后使用新配置；Docker 在 `.env` 设置后重建容器。直接运行 Python 模板时继承同名环境变量。已同步 agent OPSEC 提示，避免旧提示与平台随机策略冲突；遇到限速等异常时仍停止，不以换 UA 继续当前方向。
+
+UA 随机化不会同步改变 TLS、Cookie 或其他浏览器特征，不能保证客户端不可识别；第三方出口代理也可能再次改写请求头。
+
+## 检查过的其他通道与边界
+
+| 通道 | 结果 |
+|---|---|
+| SDK 联网搜索 | norma 的 DDGS 后端含固定 Firefox 128 浏览器 UA，没有 ARTEX / norma 名称；仅发往搜索服务，属于依赖的兼容策略，未改动，也不受此环境变量控制。Brave / Tavily 及 DeepSeek 搜索未发现 ARTEX 专属 UA。 |
+| 浏览器自动化 / runtime_harvest.js | 沿用浏览器请求头；运行时转发复制捕获到的请求头，没有硬编码 ARTEX UA。 |
+| LLM 请求 / 模型列表 / JEV | 未发现硬编码 ARTEX UA；仍可能有 HTTP 库、SDK、服务协议和用户自定义标识，不套用目标流量配置。 |
+| MCP HTTP / SSE | 未发现 ARTEX UA；显式自定义头保留。初始化协议体仍有 `clientInfo.name = artex`，这是对用户配置的 MCP 服务的协议自报身份，本次不改动。 |
+| Webhook / 通知 | 未发现硬编码 ARTEX UA；保留用户自定义头及客户端默认行为。 |
+| 安装脚本 | 未发现 ARTEX UA；获取 Docker 安装脚本的 curl 仍使用其原生身份。外部命令及用户自建脚本不由本次策略接管。 |
+| phpfinger | 仓库未包含其源码，无法核验默认值；需要在独立工具中通过 `--user-agent` 配置。 |
+
+服务端响应中的 `X-Artex-Auth` 仍用于前端识别 Basic Auth 未认证状态，属于站点自身对访问者的响应协议；它不是向目标站点发送的请求头。本次不改变该协议，也不承诺消除服务端页面、API、TLS 或其他流量特征。
+
+## 回归验证
+
+- Go：检查真实 HTTP 线上头、随机样本选择 / 固定值覆盖 / 显式空值、代理及失败直连重试、重定向、CA 信任、静态资源限制、HTML / RECON 解析、agent session 注册、自定义工具 / 重放器覆盖、自更新查询及下载和代理录制。
+- Python：本地 HTTP 服务验证默认随机 / 环境变量 / CLI 覆盖 / 显式空值、非法头值、重定向、Cookie 保留及产物生成；不访问真实目标。
+
+本地命令：
+
+```sh
+go test ./internal/httpidentity ./internal/webfetch ./enrich ./agent ./selfupdate ./traffic
+go test ./server -run 'TestProbe|TestRepeater|TestRunHTTPTool|TestHTTPTool|TestTargetHTTPIdentity' -count=1
+go test -race ./internal/httpidentity ./internal/webfetch ./enrich ./traffic
+python3 -m unittest discover -s skills/api-recon/scripts -p 'test_*.py' -v
+```
+
+以上专项回归、Go race 检查及 Python 3 项测试均通过。池扩展后的回归还检查恰好 20 个不重复样本、逐项选择以及 Go 与两个 Python 模板的样本一致性。
+
+2026-11-11 使用项目提供的本地 PostgreSQL，在新建的独立测试库中执行了 `go test ./... -count=1 -p 1 -timeout 180s`，测试库用后删除。完整套件未全部通过：
+
+- `agent` 的 `TestGraphOverviewExpandsAssociatedCompanyScope`、`TestCaptureApprovalLifecycle` 和 `db` 的 `TestInterceptApprovalFilters`、`TestSideTaskArchiveVersions` 失败；在原提交 `17c2fc5` 的独立副本和另一新建测试库中复现了相同失败。
+- `server` 首次运行有 3 个用例在临时目录清理阶段报 `directory not empty`，涉及 `TestInheritedActivityDetailAndRelationDeletion`、`TestTaskMetadataPatchReturnsRenameAndPin`、`TestTaskMetadataPatchGlobalAssetSharing`，不是 UA 断言失败。再次在新建隔离库中运行完整 `server` 套件时这 3 项通过，但 `TestCoreTaskLifecyclePG` 报 `expected goal nodes seeded, got 0`。原提交的完整 `server` 套件通过；修改后完整 `server` 套件仍未获得全绿结果。

README.md

@@ -13,7 +13,12 @@ AI 自主渗透测试系统（Go 后端 + Next.js 前端）
 
 ---
 ## 致谢 
-致谢米斯特安全团队
+米斯特安全团队
+
+洺熙
+
+wowo
+
 
 ## 截图预览
 
@@ -157,9 +162,11 @@ cp config.example.json config.json   # 填好 database 连接
 ```bash
 # 1) 前端静态导出
 cd web && npm ci && npm run build:static && cd ..
-# 2) 拷进内嵌目录
-cp -r web/out server/webui/dist
-# 3) 编译（-tags embedui 才内嵌前端）
+# 2) 创建内嵌目录
+mkdir -p server/webui/dist
+# 3) 拷进内嵌目录
+cp -r web/out/. server/webui/dist
+# 4) 编译（-tags embedui 才内嵌前端）
 CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
 ./start.sh
 ```
@@ -290,7 +297,8 @@ cp -r <解压目录>/skills ./ && cp <解压目录>/artex ./
 ```bash
 git pull
 cd web && npm ci && npm run build:static && cd ..
-cp -r web/out server/webui/dist
+mkdir -p server/webui/dist
+cp -r web/out/. server/webui/dist
 CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
 # 重启 ./start.sh
 ```
@@ -314,6 +322,10 @@ CGO_ENABLED=0 go build -tags embedui -o artex ./cmd/artex
 **LLM**：`export ANTHROPIC_API_KEY=sk-...`（或 `OPENAI_API_KEY`），也可在 UI 的「LLM 配置」页填写。
 可选：`ARTEX_LLM_PROVIDER` / `ARTEX_LLM_MODEL` / `ARTEX_LLM_BASE_URL` / `ARTEX_LLM_PROXY`。
 
+**HTTP User-Agent**：内置 WebFetch（worker / planner / main / chat）、资产 HTTP 自动补全、出口连通性探测、HTTP 自定义工具、请求重放器、GitHub 自更新和可选的 `harvest_static.py` / `spider_mpa.py` 默认从 20 个浏览器 UA 样本中随机抽选，不携带工具名称。Go 每次逻辑请求选择一次，重定向及 WebFetch 的代理失败直连重试复用同一个值；Python 脚本每次运行选择一次，入口、下载与重试保持一致。随机抽选允许重复，不依赖 AI 服务。通过 `ARTEX_HTTP_USER_AGENT` 可指定固定值；留空或非法值使用随机策略。直接运行时导出环境变量并重启服务，Docker 在 `.env` 配置后重建容器。Python 的 `--user-agent VALUE`、自定义工具和重放器中的显式 UA 优先，显式空值仍可省略此头。值去除首尾空格，仅接受可打印 ASCII。记录代理保留已有 UA 及显式省略语义。UA 随机化不代表 TLS、Cookie 或其他浏览器特征同步改变，也不保证无法识别客户端。此设置不影响浏览器、SDK 联网搜索、LLM / MCP / Webhook 服务通信或外部独立工具；本仓库未包含 phpfinger，其 `--user-agent` 需在该工具中单独配置。
+
+完整排查范围、额外发现及未接管的服务通信见 [HTTP 请求标识排查](HTTP_USER_AGENT.md)。
+
 **并发**：每个任务的 work agent 数在「系统设置」里配置（默认 3）。
 
 **常用参数**：`./start.sh -addr :8787 -proxy :8788`（`-addr` 前端+API，`-proxy` 流量录制代理）。启动脚本会把参数原样透传给 `artex`。

agent/capture.go

@@ -26,7 +26,7 @@ import (
 // contiguous run is coalesced into a single record so the trace shows whole
 // messages, not dozens of fragments.
 func captureRun(ctx context.Context, opts agentcore.Options, input string, emit func(db.Activity)) (string, harness.TerminalReason, error) {
-	s := agentcore.NewSession(opts)
+	s := newAgentSession(opts)
 	defer s.Close() // release the session's background-task manager (temp dir + processes)
 	return captureRunSession(ctx, s, input, emit)
 }

agent/chat.go

@@ -173,7 +173,7 @@ func (c *ChatAgent) Chat(ctx context.Context, agentKey, sessionID, message strin
 	// 实验功能:开启后由 noa 接管上下文压缩(归档集中在 <workDir>/noa/<SessionID> 下,持久)。
 	enableNoa(&opts, c.noaEnabledFn, c.workDir, "chat-"+sessionID, noaWarn("chat-"+sessionID))
 	ctx = attachSideCapture(ctx, &opts)
-	s := agentcore.NewSession(opts)
+	s := newAgentSession(opts)
 	defer s.Close()
 	// reload prior conversation so the agent has context across turns (each Chat is
 	// a fresh session). First turn: no file yet → Resume loads nothing and proceeds.

agent/mainagent.go

@@ -184,7 +184,7 @@ func (m *MainAgent) Chat(ctx context.Context, taskID int64, mainSeg int, as *db.
 	}
 	enableNoa(&opts, m.noaEnabledFn, m.workDir, noaSession, noaWarn(noaSession))
 	ctx = attachSideCapture(ctx, &opts)
-	s := agentcore.NewSession(opts)
+	s := newAgentSession(opts)
 	defer s.Close()
 	// reload the prior conversation from the transcript so the agent has context
 	// across turns (each Chat is a fresh session; without this it can't see earlier

agent/prompt.go

@@ -21,9 +21,9 @@ const opsecPromptBlock = `【OPSEC 准则（硬约束，仅次于授权范围和
 
 - 先复用已有流量、浏览器会话和已验证证据；能通过 traffic_search/traffic_get 或浏览器完成时，不要重复直接请求。
 - 所有目标交互必须经过平台配置的记录代理或浏览器工具，确保请求可追溯。
-- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。命令行请求必须尽量复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征；没有真实依据时不要伪造身份、设备或浏览器指纹。
+- 不得使用裸 curl、urllib、requests 等工具的默认请求头直接访问目标。优先复用目标已有流量中的 User-Agent、Accept、Referer、Cookie 等请求特征；平台 HTTP 工具和采集模板的默认 UA 由系统随机策略提供，同一次请求的重定向、重试必须保持一致，明确配置的 UA 优先。
 - 不得使用 -k、--ignore-https-errors、verify=False 等方式绕过证书校验；平台代理 CA 已配置时应正常验证。
-- 不得随机轮换 User-Agent、批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描；遇到限速、验证码、WAF 或异常响应时立即停止当前方向。
+- 不得批量并发、快速循环、无节制重试、端口横扫、目录爆破或大规模扫描；遇到限速、验证码、WAF 或异常响应时立即停止当前方向，不得通过更换 UA 继续当前方向。
 - OPSEC 不扩大授权范围；目标、主机、端口、路径或账号不在授权范围内时不得访问。
 - 请求和响应是否留痕由平台代理和浏览器负责；不要额外复制完整请求头、Cookie 或响应正文。`
 

agent/webfetch.go

@@ -0,0 +1,24 @@
+package agent
+
+import (
+	"github.com/Autumn-27/artex/internal/webfetch"
+	"github.com/Autumn-27/norma/agentcore"
+	actool "github.com/Autumn-27/norma/tool"
+)
+
+// newAgentSession installs the configurable WebFetch before the SDK builds the
+// registry, so every agent role gets one tool and the same proxy/TLS settings.
+func newAgentSession(opts agentcore.Options) *agentcore.Session {
+	if opts.EnableWebFetch {
+		opts.EnableWebFetch = false
+		if opts.Tools == nil {
+			opts.Tools = actool.DefaultTools()
+		}
+		opts.Tools = append(opts.Tools, webfetch.New(actool.WebFetchConfig{
+			Proxy:       opts.WebFetchProxy,
+			CACert:      opts.WebFetchCACert,
+			InsecureTLS: opts.WebFetchInsecureTLS,
+		}))
+	}
+	return agentcore.NewSession(opts)
+}

agent/webfetch_test.go

@@ -0,0 +1,70 @@
+package agent
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+
+	"github.com/Autumn-27/norma/agentcore"
+	"github.com/Autumn-27/norma/llm"
+	"github.com/Autumn-27/norma/permission"
+)
+
+type fetchFixtureProvider struct {
+	autoTitleProvider
+	t     *testing.T
+	url   string
+	calls int
+}
+
+func (p *fetchFixtureProvider) Complete(_ context.Context, req llm.CompletionRequest) (llm.Message, string, llm.Usage, error) {
+	p.calls++
+	if p.calls == 1 {
+		counts := map[string]int{}
+		for _, tool := range req.Tools {
+			counts[tool.Name]++
+		}
+		if counts["WebFetch"] != 1 || counts["Read"] != 1 || counts["Bash"] != 1 {
+			p.t.Errorf("session tool registration = %v", counts)
+		}
+		input, _ := json.Marshal(map[string]string{"url": p.url})
+		return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{{Type: llm.BlockToolUse, ID: "fetch-fixture", Name: "WebFetch", Input: input}}}, "tool_use", llm.Usage{}, nil
+	}
+	return llm.Message{Role: llm.RoleAssistant, Content: []llm.ContentBlock{llm.TextBlock("finished")}}, "end_turn", llm.Usage{}, nil
+}
+
+func TestAgentSessionUsesConfiguredWebFetch(t *testing.T) {
+	for _, ua := range []string{"", "Captured-UA/1.0"} {
+		t.Run(ua, func(t *testing.T) {
+			t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
+			calls := 0
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				calls++
+				if ua == "" && !strings.HasPrefix(r.UserAgent(), "Mozilla/5.0 ") {
+					t.Errorf("random UA missing: %q", r.UserAgent())
+				} else if ua != "" && r.UserAgent() != ua {
+					t.Errorf("session sent User-Agent %q, want %q", r.UserAgent(), ua)
+				}
+				w.Write([]byte("fixture"))
+			}))
+			defer server.Close()
+			provider := &fetchFixtureProvider{t: t, url: server.URL}
+			session := newAgentSession(agentcore.Options{
+				Provider: provider, NonStreaming: true, EnableWebFetch: true,
+				PermissionMode: permission.ModeBypass, MaxTurns: 3,
+			})
+			defer session.Close()
+			for _, err := range session.Prompt(t.Context(), "fetch local fixture") {
+				if err != nil {
+					t.Fatal(err)
+				}
+			}
+			if calls != 1 {
+				t.Fatalf("WebFetch calls = %d, want 1", calls)
+			}
+		})
+	}
+}

agent/worker.go

@@ -488,7 +488,7 @@ func (w *Worker) execute(ctx context.Context, name string, taskID int64, as *db.
 	noaSession := WorkerSessionID(ts.ID(), intent.ID)
 	enableNoa(&opts, w.noaEnabledFn, w.workDir, noaSession, noaWarn(noaSession))
 	ctx = attachSideCapture(ctx, &opts)
-	s := agentcore.NewSession(opts)
+	s := newAgentSession(opts)
 	defer s.Close() // release the session's background-task manager (temp dir + processes)
 
 	// Resume prior conversation if this intent was paused/blocked/exhausted and is

docker-compose.yml

@@ -28,6 +28,7 @@ services:
       ARTEX_LLM_MODEL: ${ARTEX_LLM_MODEL:-}
       ARTEX_LLM_BASE_URL: ${ARTEX_LLM_BASE_URL:-}
       ARTEX_LLM_PROXY: ${ARTEX_LLM_PROXY:-}
+      ARTEX_HTTP_USER_AGENT: ${ARTEX_HTTP_USER_AGENT:-}
     ports:
       - "8787:8787"
       - "127.0.0.1:8788:8788"

enrich/enrich.go

@@ -20,6 +20,7 @@ import (
 	"time"
 
 	"github.com/Autumn-27/artex/db"
+	"github.com/Autumn-27/artex/internal/httpidentity"
 
 	"github.com/miekg/dns"
 	"github.com/projectdiscovery/dnsx/libs/dnsx"
@@ -230,7 +231,7 @@ func (e *Engine) doHTTP(id int64, rawURL string) {
 	if err != nil {
 		return
 	}
-	req.Header.Set("User-Agent", "artex-enrich/1.0")
+	httpidentity.Apply(req)
 	resp, err := e.client.Do(req)
 	if err != nil {
 		return

enrich/enrich_test.go

@@ -0,0 +1,38 @@
+package enrich
+
+import (
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+// Stop after receiving the probe so this request regression needs no database.
+func TestProbeUserAgent(t *testing.T) {
+	for _, ua := range []string{"", "Captured-UA/1.0"} {
+		t.Run(ua, func(t *testing.T) {
+			t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
+			calls := 0
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				calls++
+				if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
+					t.Errorf("random UA missing: %q", got)
+				} else if ua != "" && got != ua {
+					t.Errorf("User-Agent = %q, want %q", got, ua)
+				}
+				http.Redirect(w, r, "/stop", http.StatusFound)
+			}))
+			defer server.Close()
+			client := buildClient(nil)
+			client.CheckRedirect = func(*http.Request, []*http.Request) error {
+				return errors.New("probe captured")
+			}
+			e := &Engine{client: client}
+			e.doHTTP(1, server.URL)
+			if calls != 1 {
+				t.Fatalf("probe calls = %d, want 1", calls)
+			}
+		})
+	}
+}

go.mod

@@ -3,19 +3,19 @@ module github.com/Autumn-27/artex
 go 1.26.10
 
 require (
-	mvdan.cc/sh/v3 v3.10.0
 	github.com/Autumn-27/norma v0.4.3
 	github.com/golang-jwt/jwt/v5 v5.3.1
 	github.com/google/uuid v1.6.0
 	github.com/jackc/pgx/v5 v5.10.0
 	github.com/klauspost/compress v1.18.6
-	github.com/lqqyt2423/go-mitmproxy v1.9.2
+	github.com/lqqyt2423/go-mitmproxy v1.9.5
 	github.com/miekg/dns v1.1.72
 	github.com/projectdiscovery/dnsx v1.2.3
-	golang.org/x/crypto v0.55.0
-	golang.org/x/net v0.58.0
-	golang.org/x/text v0.41.0
+	golang.org/x/crypto v0.58.0
+	golang.org/x/net v0.61.0
+	golang.org/x/text v0.43.0
 	modernc.org/sqlite v1.52.0
+	mvdan.cc/sh/v3 v3.10.0
 )
 
 require (
@@ -52,10 +52,10 @@ require (
 	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 // indirect
-	golang.org/x/mod v0.40.0 // indirect
-	golang.org/x/sync v0.22.0 // indirect
-	golang.org/x/sys v0.47.0 // indirect
-	golang.org/x/tools v0.49.0 // indirect
+	golang.org/x/mod v0.41.0 // indirect
+	golang.org/x/sync v0.24.0 // indirect
+	golang.org/x/sys v0.49.0 // indirect
+	golang.org/x/tools v0.51.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 	modernc.org/libc v1.72.3 // indirect
 	modernc.org/mathutil v1.7.1 // indirect

go.sum

@@ -48,8 +48,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
 github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
-github.com/lqqyt2423/go-mitmproxy v1.9.2 h1:+QTbDyUUaVRmEavPq0/eWzJ2K+K4JdL4ff91s/adnxk=
-github.com/lqqyt2423/go-mitmproxy v1.9.2/go.mod h1:6H5PR+nbVsiGQJC/Y98GHJItyFcmP/Z0txAgjhG2a80=
+github.com/lqqyt2423/go-mitmproxy v1.9.5 h1:4ar1qWYlbXF+7HhG06AVjKkDILXX1FS0mNy+hkX4uX4=
+github.com/lqqyt2423/go-mitmproxy v1.9.5/go.mod h1:74Z1IyzuUu/pq1IpdDd555f1xXM7Gl9KtUhWn/mzU7E=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
@@ -108,21 +108,35 @@ go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
 go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
 golang.org/x/crypto v0.55.0 h1:+KWHjbgOaAQ66dh/YlkZKHlz9ZUlq61AFirAR9ntP8M=
 golang.org/x/crypto v0.55.0/go.mod h1:uq0V9dE/fzQuJtbnL+2EhWOE63vo164FY8xqEnV9xis=
+golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
+golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
 golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=
 golang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=
 golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
 golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
+golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c=
+golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o=
 golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To=
 golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU=
+golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
+golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
 golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
 golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
+golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
+golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
 golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
+golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
+golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
 golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
 golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
+golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
+golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
 golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
 golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
+golang.org/x/tools v0.51.0 h1:k4Xc/1Om9jwkBJBo4NVLMSARBoWtK10mx+W5BnXCeAI=
+golang.org/x/tools v0.51.0/go.mod h1:9eEncMayCV6zRMGhR5eZEC2iBx98qWcF1HZ9Z7wJOoA=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=

intercept/intercept.go

@@ -572,8 +572,9 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
 	if err != nil || len(rules) == 0 {
 		return Decision{}, false
 	}
+	values := inputStrings(input)
 	for _, r := range rules {
-		if ruleMatches(r, toolName, input) {
+		if ruleMatches(r, toolName, input, values) {
 			msg := r.Message
 			if msg == "" {
 				msg = defaultMessage(r.Action, r.Name)
@@ -593,20 +594,60 @@ func (i *Interceptor) Match(toolName string, input []byte) (Decision, bool) {
 	return Decision{}, false
 }
 
-func ruleMatches(r compiledRule, toolName string, input []byte) bool {
-	var subject string
+// ruleMatches reports whether r applies to the call. tool_input rules are tried
+// against the raw JSON first, then against each decoded string value in values.
+// The raw form alone is not enough: JSON escapes newlines, tabs and quotes, and
+// Go's encoder also escapes & < >, so a rule like `\brm\b` never fires on
+// "cd /tmp\nrm -rf /var/www" (the raw bytes read `\nrm`, with no word boundary).
+func ruleMatches(r compiledRule, toolName string, input []byte, values []string) bool {
 	switch r.MatchTarget {
 	case "tool_name":
-		subject = toolName
+		return r.matchString(toolName)
 	case "tool_input":
-		subject = string(input)
-	default:
-		return false
+		if r.matchString(string(input)) {
+			return true
+		}
+		for _, v := range values {
+			if r.matchString(v) {
+				return true
+			}
+		}
 	}
+	return false
+}
+
+func (r compiledRule) matchString(s string) bool {
 	if r.MatchType == "regex" {
-		return r.re != nil && r.re.MatchString(subject)
+		return r.re != nil && r.re.MatchString(s)
 	}
-	return strings.Contains(subject, r.Pattern)
+	return strings.Contains(s, r.Pattern)
+}
+
+// inputStrings returns every string value in a JSON tool input, at any depth.
+// Object keys are skipped. It returns nil if the input is not valid JSON.
+func inputStrings(input []byte) []string {
+	var doc any
+	if json.Unmarshal(input, &doc) != nil {
+		return nil
+	}
+	var out []string
+	var walk func(any)
+	walk = func(v any) {
+		switch v := v.(type) {
+		case string:
+			out = append(out, v)
+		case []any:
+			for _, e := range v {
+				walk(e)
+			}
+		case map[string]any:
+			for _, e := range v {
+				walk(e)
+			}
+		}
+	}
+	walk(doc)
+	return out
 }
 
 func defaultMessage(action, name string) string {

intercept/rule_match_test.go

@@ -0,0 +1,138 @@
+package intercept
+
+import (
+	"encoding/json"
+	"regexp"
+	"testing"
+
+	"github.com/Autumn-27/artex/db"
+)
+
+// Patterns copied from the built-in rules seeded in db/db.go.
+const (
+	builtinRmRf  = `(?i)\brm\b.{0,80}(?:-[a-z]*r[a-z]*f[a-z]*|-[a-z]*f[a-z]*r[a-z]*|--recursive|--no-preserve-root)`
+	builtinMkfs  = `\bmkfs\b`
+	builtinDrop  = `(?i)\bDROP\s+(?:DATABASE|TABLE|SCHEMA|INDEX|VIEW|TABLESPACE|USER|ROLE)\b`
+	builtinFlush = `(?i)\b(?:FLUSHALL|FLUSHDB)\b`
+	builtinV2    = `(?i)\b(rm\s+-rf\s+/|mkfs|dd\s+if=|:\(\)\s*\{|shutdown|reboot|>\s*/dev/sd)`
+)
+
+func testRule(target, matchType, pattern string) compiledRule {
+	r := compiledRule{InterceptRule: db.InterceptRule{
+		ID: 1, Name: "test", Enabled: true, Action: "deny",
+		MatchTarget: target, MatchType: matchType, Pattern: pattern,
+	}}
+	if matchType == "regex" {
+		r.re = regexp.MustCompile(pattern)
+	}
+	return r
+}
+
+// interceptorWith returns an Interceptor whose rule cache is pre-filled, so
+// Match runs without a database.
+func interceptorWith(rules ...compiledRule) *Interceptor {
+	return &Interceptor{cached: rules}
+}
+
+func toolInput(t *testing.T, v any) []byte {
+	t.Helper()
+	b, err := json.Marshal(v)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return b
+}
+
+func TestMatchToolInputSeesDecodedValues(t *testing.T) {
+	cases := []struct {
+		name    string
+		pattern string
+		tool    string
+		input   any
+	}{
+		{"rm -rf on its own line", builtinRmRf, "Bash", map[string]string{"command": "cd /tmp\nrm -rf /var/www"}},
+		{"rm -rf after a tab", builtinRmRf, "Bash", map[string]string{"command": "echo done\trm -rf /var/www"}},
+		{"mkfs on its own line", builtinMkfs, "Bash", map[string]string{"command": "lsblk\nmkfs.ext4 /dev/sdb"}},
+		{"DROP inside a heredoc", builtinDrop, "Bash", map[string]string{"command": "psql <<EOF\nDROP TABLE users;\nEOF"}},
+		{"FLUSHALL inside a heredoc", builtinFlush, "Bash", map[string]string{"command": "redis-cli <<EOF\nFLUSHALL\nEOF"}},
+		{"shutdown on its own line", builtinV2, "Bash", map[string]string{"command": "sync\nshutdown -h now"}},
+		// encoding/json writes > as >, so this only matches the decoded value.
+		{"redirect to a disk", `>\s*/dev/sd`, "Bash", map[string]string{"command": "cat img > /dev/sda"}},
+		{"interactive shell text", builtinRmRf, "shell_send", map[string]string{"text": "ls\nrm -rf /srv/data\n"}},
+		{"nested edit value", builtinDrop, "MultiEdit", map[string]any{
+			"file_path": "/tmp/m.sql",
+			"edits":     []map[string]string{{"old_string": "-- x", "new_string": "-- x\nDROP TABLE users;"}},
+		}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			ic := interceptorWith(testRule("tool_input", "regex", tc.pattern))
+			in := toolInput(t, tc.input)
+			if _, ok := ic.Match(tc.tool, in); !ok {
+				t.Fatalf("rule %q did not match input %s", tc.pattern, in)
+			}
+		})
+	}
+}
+
+func TestMatchToolInputStringRuleSeesDecodedValues(t *testing.T) {
+	// The raw JSON holds rm -rf \"/var/www\", so the quotes only line up once decoded.
+	ic := interceptorWith(testRule("tool_input", "string", `rm -rf "/var/www"`))
+	in := toolInput(t, map[string]string{"command": `rm -rf "/var/www"`})
+	if _, ok := ic.Match("Bash", in); !ok {
+		t.Fatalf("string rule did not match input %s", in)
+	}
+}
+
+// Rules written against the raw JSON form must keep working.
+func TestMatchToolInputStillMatchesRawJSON(t *testing.T) {
+	ic := interceptorWith(testRule("tool_input", "regex", `"command":"rm `))
+	in := toolInput(t, map[string]string{"command": "rm notes.txt"})
+	if _, ok := ic.Match("Bash", in); !ok {
+		t.Fatalf("raw JSON rule did not match input %s", in)
+	}
+}
+
+func TestMatchToolInputInvalidJSONFallsBackToRaw(t *testing.T) {
+	ic := interceptorWith(testRule("tool_input", "regex", builtinMkfs))
+	if _, ok := ic.Match("Bash", []byte(`not json: mkfs /dev/sdb`)); !ok {
+		t.Fatal("rule did not match non-JSON input")
+	}
+}
+
+func TestMatchToolInputNoFalsePositives(t *testing.T) {
+	ic := interceptorWith(
+		testRule("tool_input", "regex", builtinRmRf),
+		testRule("tool_input", "regex", builtinMkfs),
+		testRule("tool_input", "regex", builtinDrop),
+	)
+	for _, cmd := range []string{
+		"cd /tmp\nls -la",
+		"nmap -sV 10.0.0.5\ncurl -s http://10.0.0.5/",
+		"echo firm\nrmdir build", // "firm" and "rmdir" must not look like rm
+	} {
+		in := toolInput(t, map[string]string{"command": cmd})
+		if dec, ok := ic.Match("Bash", in); ok {
+			t.Fatalf("unexpected match %q for input %s", dec.RuleName, in)
+		}
+	}
+}
+
+// The match key ("command") must not be matched as a value.
+func TestMatchToolInputIgnoresObjectKeys(t *testing.T) {
+	ic := interceptorWith(testRule("tool_input", "regex", `^mkfs$`))
+	in := toolInput(t, map[string]string{"mkfs": "ls"})
+	if _, ok := ic.Match("Bash", in); ok {
+		t.Fatalf("object key matched as a value for input %s", in)
+	}
+}
+
+func TestMatchToolNameUnchanged(t *testing.T) {
+	ic := interceptorWith(testRule("tool_name", "string", "WebFetch"))
+	if _, ok := ic.Match("WebFetch", []byte(`{"url":"http://x"}`)); !ok {
+		t.Fatal("tool_name rule did not match")
+	}
+	if _, ok := ic.Match("Bash", []byte(`{"command":"WebFetch"}`)); ok {
+		t.Fatal("tool_name rule matched tool input")
+	}
+}

internal/httpidentity/useragent.go

@@ -0,0 +1,67 @@
+// Package httpidentity defines the explicit identity for target-facing HTTP requests.
+package httpidentity
+
+import (
+	"crypto/rand"
+	"io"
+	"math/big"
+	"net/http"
+	"os"
+	"strings"
+)
+
+// Keep these 20 standard desktop browser UA samples in sync with the standalone
+// Python templates. Reviewed 2026-11-11; see HTTP_USER_AGENT.md for versions and
+// vendor format references. Keep normal reduced platform/version fields and no
+// tool identifiers, device-specific details or custom suffixes.
+var userAgents = [...]string{
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0",
+	"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0",
+	"Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0",
+	"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15",
+}
+
+// UserAgent returns the configured printable ASCII value, or a random browser
+// sample when unset, empty or malformed. Call once per logical request and reuse
+// the selected value for redirects and retries.
+func UserAgent() string {
+	value := strings.Trim(os.Getenv("ARTEX_HTTP_USER_AGENT"), " ")
+	for _, c := range value {
+		if c < 32 || c > 126 {
+			return randomUserAgent(rand.Reader)
+		}
+	}
+	if value != "" {
+		return value
+	}
+	return randomUserAgent(rand.Reader)
+}
+
+func randomUserAgent(source io.Reader) string {
+	index, err := rand.Int(source, big.NewInt(int64(len(userAgents))))
+	if err != nil {
+		return userAgents[0] // Keep a valid nonempty UA if system entropy fails.
+	}
+	return userAgents[index.Int64()]
+}
+
+// Apply selects an identity for this request; net/http preserves it on redirects.
+func Apply(req *http.Request) {
+	req.Header.Set("User-Agent", UserAgent())
+}

internal/httpidentity/useragent_test.go

@@ -0,0 +1,65 @@
+package httpidentity
+
+import (
+	"bytes"
+	"net/http"
+	"net/http/httptest"
+	"slices"
+	"testing"
+)
+
+func TestApplyOnWire(t *testing.T) {
+	for _, tc := range []struct{ name, value, want string }{
+		{"empty", "", ""},
+		{"configured", " Captured-UA/1.0 ", "Captured-UA/1.0"},
+		{"whitespace", "   ", ""},
+		{"injection", "ok\r\nX-Injected: yes", ""},
+		{"non-ascii", "客户端", ""},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			t.Setenv("ARTEX_HTTP_USER_AGENT", tc.value)
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				got := r.UserAgent()
+				if tc.want == "" && !slices.Contains(userAgents[:], got) {
+					t.Errorf("default User-Agent is not a browser sample: %q", got)
+				} else if tc.want != "" && got != tc.want {
+					t.Errorf("User-Agent = %q, want %q", got, tc.want)
+				}
+				if r.Header.Get("X-Injected") != "" {
+					t.Error("header injection")
+				}
+			}))
+			defer server.Close()
+			req, err := http.NewRequest(http.MethodGet, server.URL, nil)
+			if err != nil {
+				t.Fatal(err)
+			}
+			Apply(req)
+			resp, err := server.Client().Do(req)
+			if err != nil {
+				t.Fatal(err)
+			}
+			resp.Body.Close()
+		})
+	}
+}
+
+func TestRandomUserAgentSelection(t *testing.T) {
+	if len(userAgents) != 20 {
+		t.Fatalf("pool size = %d, want 20", len(userAgents))
+	}
+	seen := make(map[string]bool, len(userAgents))
+	// Exercise every choice deterministically without a statistical/flaky test.
+	for index, want := range userAgents {
+		if seen[want] {
+			t.Errorf("duplicate sample: %q", want)
+		}
+		seen[want] = true
+		if got := randomUserAgent(bytes.NewReader([]byte{byte(index)})); got != want {
+			t.Errorf("choice %d = %q, want %q", index, got, want)
+		}
+	}
+	if got := randomUserAgent(bytes.NewReader(nil)); got != userAgents[0] {
+		t.Errorf("entropy failure fallback = %q", got)
+	}
+}

internal/webfetch/webfetch.go

@@ -0,0 +1,467 @@
+// Adapted from github.com/Autumn-27/norma/tool/webfetch.go at v0.4.3.
+// Keep the SDK tool metadata and fetch/render behavior; use ARTEX's explicit
+// HTTP identity for both the proxy attempt and its direct fallback. This local
+// adapter is needed until the SDK exposes request headers in actool.WebFetchConfig.
+package webfetch
+
+import (
+	"context"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net/http"
+	"net/url"
+	"os"
+	"regexp"
+	"strings"
+	"time"
+
+	"github.com/Autumn-27/artex/internal/httpidentity"
+	actool "github.com/Autumn-27/norma/tool"
+	"golang.org/x/net/html"
+)
+
+// New keeps the SDK's schema, permission checks and tool metadata.
+func New(cfg actool.WebFetchConfig) actool.CoreTool {
+	base := actool.NewWebFetch(cfg)
+	return actool.Build(actool.Spec{
+		Name: base.Name(), Description: base.Description(), Prompt: base.Prompt(),
+		Schema: base.InputSchema(), Permissions: base.CheckPermissions,
+		ReadOnly: base.IsReadOnly, Concurrent: base.IsConcurrencySafe,
+		Run: webFetchRunner(cfg),
+	})
+}
+
+const maxFetchBytes = 2 << 20 // 2 MiB
+
+// newFetchClient returns an HTTP client honoring the proxy / TLS config.
+func newFetchClient(cfg actool.WebFetchConfig) *http.Client {
+	tr := &http.Transport{}
+	if cfg.Proxy != "" {
+		if pu, err := url.Parse(cfg.Proxy); err == nil {
+			tr.Proxy = http.ProxyURL(pu)
+		}
+	}
+	switch {
+	case cfg.InsecureTLS:
+		tr.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+	case cfg.CACert != "":
+		// Trust the proxy's CA in addition to the system roots, so certs it
+		// re-signs verify while every other connection stays fully checked.
+		if pem, err := os.ReadFile(cfg.CACert); err == nil {
+			pool, _ := x509.SystemCertPool()
+			if pool == nil {
+				pool = x509.NewCertPool()
+			}
+			if pool.AppendCertsFromPEM(pem) {
+				tr.TLSClientConfig = &tls.Config{RootCAs: pool}
+			}
+		}
+	}
+	return &http.Client{Transport: tr}
+}
+
+// proxyLikelyErr reports whether a fetch error is the kind a recording/MITM proxy
+// introduces — a closed tunnel (EOF / connection reset), a failed CONNECT, or a
+// TLS/HTTP2 mismatch — so a direct (no-proxy) retry may succeed. Name-resolution
+// failures and caller cancellation/timeout are target- or caller-side problems a
+// direct retry won't fix, so those return false (no retry).
+func proxyLikelyErr(err error) bool {
+	if err == nil {
+		return false
+	}
+	s := strings.ToLower(err.Error())
+	for _, n := range []string{"no such host", "context canceled", "context deadline", "timeout"} {
+		if strings.Contains(s, n) {
+			return false
+		}
+	}
+	for _, p := range []string{
+		"eof", "connection reset", "connection refused", "proxyconnect",
+		"broken pipe", "forcibly closed", "malformed", "http2", "http/2",
+		"tls:", "protocol error",
+	} {
+		if strings.Contains(s, p) {
+			return true
+		}
+	}
+	return false
+}
+
+// staticAssetExts are file extensions WebFetch refuses: linked resources (scripts,
+// styles, source maps, images, fonts, media, wasm) that are not page content.
+var staticAssetExts = map[string]bool{
+	".js": true, ".mjs": true, ".cjs": true, ".css": true, ".map": true,
+	".png": true, ".jpg": true, ".jpeg": true, ".gif": true, ".svg": true,
+	".webp": true, ".avif": true, ".ico": true, ".bmp": true,
+	".woff": true, ".woff2": true, ".ttf": true, ".otf": true, ".eot": true,
+	".wasm": true, ".mp4": true, ".webm": true, ".mp3": true, ".wav": true, ".ogg": true,
+}
+
+// staticAssetExt returns the offending extension (with dot) if rawURL points at a
+// static asset, or "" otherwise. The query string and fragment are ignored, so
+// "/app.js?v=3" is still caught.
+func staticAssetExt(rawURL string) string {
+	path := rawURL
+	if u, err := url.Parse(rawURL); err == nil {
+		path = u.Path
+	}
+	if i := strings.LastIndexByte(path, '/'); i >= 0 {
+		path = path[i+1:] // last segment only
+	}
+	dot := strings.LastIndexByte(path, '.')
+	if dot < 0 {
+		return ""
+	}
+	if ext := strings.ToLower(path[dot:]); staticAssetExts[ext] {
+		return ext
+	}
+	return ""
+}
+
+func webFetchRunner(cfg actool.WebFetchConfig) func(context.Context, json.RawMessage, *actool.ToolContext) (actool.Result, error) {
+	client := newFetchClient(cfg)
+	// direct is the no-proxy fallback used when a proxied fetch fails for a
+	// proxy-caused reason (closed tunnel / failed CONNECT / TLS or HTTP2 mismatch):
+	// it bypasses the configured proxy so a recording/MITM hiccup doesn't fail an
+	// otherwise-reachable URL. nil when no proxy is configured (nothing to bypass).
+	var direct *http.Client
+	if cfg.Proxy != "" {
+		direct = newFetchClient(actool.WebFetchConfig{InsecureTLS: cfg.InsecureTLS})
+	}
+	return func(ctx context.Context, input json.RawMessage, _ *actool.ToolContext) (actool.Result, error) {
+		var in struct {
+			URL     string `json:"url"`
+			Extract bool   `json:"extract"`
+		}
+		if err := json.Unmarshal(input, &in); err != nil {
+			return actool.Result{}, err
+		}
+		if !strings.HasPrefix(in.URL, "http://") && !strings.HasPrefix(in.URL, "https://") {
+			return actool.Errorf("Error: url must start with http:// or https://"), nil
+		}
+		if ext := staticAssetExt(in.URL); ext != "" {
+			return actool.Errorf("Error: WebFetch does not fetch static assets (" + ext + " file). It reads HTML pages, documentation, and API responses — not linked resources like scripts, styles, images, or fonts. Fetch the page that references this asset instead."), nil
+		}
+		cctx, cancel := context.WithTimeout(ctx, 30*time.Second)
+		defer cancel()
+		req, err := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil)
+		if err != nil {
+			return actool.Errorf("Error: " + err.Error()), nil
+		}
+		userAgent := httpidentity.UserAgent()
+		req.Header.Set("User-Agent", userAgent)
+		resp, err := client.Do(req)
+		// One direct retry when the proxied attempt failed for a proxy-caused reason.
+		// A non-proxy error (bad host, caller timeout/cancel) is returned as-is — a
+		// direct retry wouldn't fix it. The direct response is NOT captured by the proxy.
+		if err != nil && direct != nil && proxyLikelyErr(err) {
+			if r2, e2 := http.NewRequestWithContext(cctx, http.MethodGet, in.URL, nil); e2 == nil {
+				r2.Header.Set("User-Agent", userAgent)
+				if resp2, err2 := direct.Do(r2); err2 == nil {
+					resp, err = resp2, nil
+				} else {
+					err = err2 // surface the direct failure (more representative of the target)
+				}
+			}
+		}
+		if err != nil {
+			return actool.Errorf("Error fetching URL: " + err.Error()), nil
+		}
+		defer resp.Body.Close()
+		body, _ := io.ReadAll(io.LimitReader(resp.Body, maxFetchBytes))
+
+		header := fmt.Sprintf("[%d %s] %s\n", resp.StatusCode, http.StatusText(resp.StatusCode), in.URL)
+		// Cheap fingerprint signal that HTML→Markdown would otherwise hide.
+		if sv := resp.Header.Get("Server"); sv != "" {
+			header += "Server: " + sv + "\n"
+		}
+		if xp := resp.Header.Get("X-Powered-By"); xp != "" {
+			header += "X-Powered-By: " + xp + "\n"
+		}
+
+		var out string
+		if strings.Contains(strings.ToLower(resp.Header.Get("content-type")), "html") {
+			doc, perr := html.Parse(strings.NewReader(string(body)))
+			if perr != nil { // html.Parse is lenient; this is effectively unreachable
+				out = string(body)
+			} else {
+				out = htmlToMarkdown(doc)
+				if in.Extract {
+					if recon := extractRecon(doc); recon != "" {
+						out += "\n\n" + recon
+					}
+				}
+			}
+		} else {
+			out = string(body)
+		}
+		return actool.Text(truncate(header+strings.TrimSpace(out), 50000)), nil
+	}
+}
+
+var (
+	reWS         = regexp.MustCompile(`[ \t]+`)
+	reLineTrim   = regexp.MustCompile(` *\n *`) // strip spaces hugging a newline
+	reBlankLines = regexp.MustCompile(`\n{2,}`) // collapse blank lines → one newline
+	// reEndpoint matches quoted absolute URLs and root-relative paths in inline JS.
+	reEndpoint = regexp.MustCompile("[\"'`](https?://[^\"'`\\s]+|/[A-Za-z0-9._~$&'()*+,;=:@%/?#\\[\\]-]{2,})[\"'`]")
+)
+
+// skipMarkdown are elements whose subtree carries no readable prose.
+var skipMarkdown = map[string]bool{"script": true, "style": true, "noscript": true, "svg": true, "template": true}
+
+// htmlToMarkdown renders the visible content of a parsed document as Markdown.
+// Text nodes are already entity-decoded by the parser, so no unescaping is needed.
+func htmlToMarkdown(doc *html.Node) string {
+	var b strings.Builder
+	renderNode(&b, doc)
+	s := reWS.ReplaceAllString(b.String(), " ")
+	s = reLineTrim.ReplaceAllString(s, "\n")   // drop indentation / trailing spaces per line
+	s = reBlankLines.ReplaceAllString(s, "\n") // no blank lines — compact, token-lean output
+	return strings.TrimSpace(s)
+}
+
+func renderNode(b *strings.Builder, n *html.Node) {
+	switch n.Type {
+	case html.TextNode:
+		b.WriteString(strings.ReplaceAll(n.Data, "\n", " "))
+		return
+	case html.ElementNode:
+		if skipMarkdown[n.Data] {
+			return
+		}
+	default:
+		if n.Type != html.DocumentNode {
+			return
+		}
+	}
+	tag := ""
+	if n.Type == html.ElementNode {
+		tag = n.Data
+	}
+	switch tag {
+	case "h1", "h2", "h3", "h4", "h5", "h6":
+		b.WriteString("\n\n" + strings.Repeat("#", int(tag[1]-'0')) + " ")
+		renderChildren(b, n)
+		b.WriteString("\n\n")
+	case "title":
+		b.WriteString("\n\n# ")
+		renderChildren(b, n)
+		b.WriteString("\n\n")
+	case "p", "div", "section", "article", "header", "footer", "main", "ul", "ol", "table", "tr", "blockquote":
+		b.WriteString("\n\n")
+		renderChildren(b, n)
+		b.WriteString("\n\n")
+	case "br":
+		b.WriteString("\n")
+	case "hr":
+		b.WriteString("\n\n---\n\n")
+	case "li":
+		b.WriteString("\n- ")
+		renderChildren(b, n)
+	case "a":
+		var inner strings.Builder
+		renderChildren(&inner, n)
+		text := strings.TrimSpace(inner.String())
+		if href := attr(n, "href"); href != "" && text != "" {
+			fmt.Fprintf(b, "[%s](%s)", text, href)
+		} else {
+			b.WriteString(text)
+		}
+	case "img":
+		if src := attr(n, "src"); src != "" {
+			fmt.Fprintf(b, "![%s](%s)", attr(n, "alt"), src)
+		}
+	case "strong", "b":
+		b.WriteString("**")
+		renderChildren(b, n)
+		b.WriteString("**")
+	case "em", "i":
+		b.WriteString("*")
+		renderChildren(b, n)
+		b.WriteString("*")
+	case "code":
+		b.WriteString("`")
+		renderChildren(b, n)
+		b.WriteString("`")
+	case "pre":
+		var inner strings.Builder
+		textOnly(&inner, n)
+		b.WriteString("\n\n```\n" + strings.TrimSpace(inner.String()) + "\n```\n\n")
+	default:
+		renderChildren(b, n)
+	}
+}
+
+func renderChildren(b *strings.Builder, n *html.Node) {
+	for c := n.FirstChild; c != nil; c = c.NextSibling {
+		renderNode(b, c)
+	}
+}
+
+// extractRecon walks the document for security-relevant signal that Markdown
+// rendering drops: comments, script sources, inline endpoints, forms, meta, links.
+func extractRecon(doc *html.Node) string {
+	var comments, scriptSrcs, inlineEndpoints, forms, metas, links []string
+	seenLink := map[string]bool{}
+
+	var walk func(n *html.Node)
+	walk = func(n *html.Node) {
+		switch n.Type {
+		case html.CommentNode:
+			if c := oneLine(n.Data, 300); c != "" {
+				comments = append(comments, c)
+			}
+		case html.ElementNode:
+			switch n.Data {
+			case "script":
+				if src := attr(n, "src"); src != "" {
+					scriptSrcs = append(scriptSrcs, src)
+				} else {
+					var body strings.Builder
+					textOnly(&body, n)
+					for _, m := range reEndpoint.FindAllStringSubmatch(body.String(), 40) {
+						inlineEndpoints = append(inlineEndpoints, m[1])
+					}
+				}
+			case "form":
+				forms = append(forms, describeForm(n))
+			case "meta":
+				name := attr(n, "name")
+				if name == "" {
+					name = attr(n, "property")
+				}
+				if content := attr(n, "content"); name != "" && content != "" {
+					metas = append(metas, name+"="+oneLine(content, 120))
+				}
+			case "a", "link":
+				if h := attr(n, "href"); h != "" && !seenLink[h] {
+					seenLink[h] = true
+					links = append(links, h)
+				}
+			}
+		}
+		for c := n.FirstChild; c != nil; c = c.NextSibling {
+			walk(c)
+		}
+	}
+	walk(doc)
+
+	var b strings.Builder
+	b.WriteString("=== RECON ===")
+	reconSection(&b, "COMMENTS", comments)
+	reconSection(&b, "SCRIPTS (src)", dedup(scriptSrcs))
+	reconSection(&b, "INLINE ENDPOINTS", dedup(inlineEndpoints))
+	reconSection(&b, "FORMS", forms)
+	reconSection(&b, "META", metas)
+	reconSection(&b, "LINKS", capSlice(links, 100))
+	if b.Len() == len("=== RECON ===") {
+		return "" // nothing worth reporting
+	}
+	return b.String()
+}
+
+// describeForm summarizes a <form> as "METHOD action → name(type[=value]) …".
+// Hidden inputs keep their value (CSRF tokens, parameter defaults matter in recon).
+func describeForm(form *html.Node) string {
+	method := strings.ToUpper(attr(form, "method"))
+	if method == "" {
+		method = "GET"
+	}
+	var inputs []string
+	var walk func(n *html.Node)
+	walk = func(n *html.Node) {
+		if n.Type == html.ElementNode {
+			switch n.Data {
+			case "input", "textarea", "select":
+				typ := attr(n, "type")
+				if typ == "" {
+					typ = n.Data
+				}
+				desc := attr(n, "name") + "(" + typ
+				if typ == "hidden" {
+					if v := attr(n, "value"); v != "" {
+						desc += "=" + oneLine(v, 60)
+					}
+				}
+				inputs = append(inputs, desc+")")
+			}
+		}
+		for c := n.FirstChild; c != nil; c = c.NextSibling {
+			walk(c)
+		}
+	}
+	walk(form)
+	return fmt.Sprintf("%s %s → %s", method, attr(form, "action"), strings.Join(inputs, " "))
+}
+
+// --- small helpers ---
+
+func attr(n *html.Node, key string) string {
+	for _, a := range n.Attr {
+		if a.Key == key {
+			return a.Val
+		}
+	}
+	return ""
+}
+
+func textOnly(b *strings.Builder, n *html.Node) {
+	if n.Type == html.TextNode {
+		b.WriteString(n.Data)
+		return
+	}
+	for c := n.FirstChild; c != nil; c = c.NextSibling {
+		textOnly(b, c)
+	}
+}
+
+func oneLine(s string, max int) string {
+	s = strings.Join(strings.Fields(s), " ")
+	if len(s) > max {
+		return s[:max] + "…"
+	}
+	return s
+}
+
+func reconSection(b *strings.Builder, title string, items []string) {
+	if len(items) == 0 {
+		return
+	}
+	b.WriteString("\n\n## " + title + "\n")
+	for _, it := range items {
+		b.WriteString("- " + it + "\n")
+	}
+}
+
+func dedup(in []string) []string {
+	seen := map[string]bool{}
+	out := in[:0:0]
+	for _, s := range in {
+		if s == "" || seen[s] {
+			continue
+		}
+		seen[s] = true
+		out = append(out, s)
+	}
+	return out
+}
+
+func capSlice(in []string, n int) []string {
+	if len(in) <= n {
+		return in
+	}
+	return append(in[:n:n], fmt.Sprintf("… (%d more)", len(in)-n))
+}
+
+// truncate retains the SDK's head/tail output limit.
+func truncate(s string, max int) string {
+	if max <= 0 || len(s) <= max {
+		return s
+	}
+	half := max / 2
+	return s[:half] + fmt.Sprintf("\n\n... [%d characters truncated] ...\n\n", len(s)-max) + s[len(s)-half:]
+}

internal/webfetch/webfetch_test.go

@@ -0,0 +1,175 @@
+package webfetch
+
+import (
+	"context"
+	"encoding/json"
+	"encoding/pem"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"testing"
+
+	actool "github.com/Autumn-27/norma/tool"
+)
+
+func callFetch(t *testing.T, tool actool.CoreTool, target string, extract bool) actool.Result {
+	t.Helper()
+	input, err := json.Marshal(map[string]any{"url": target, "extract": extract})
+	if err != nil {
+		t.Fatal(err)
+	}
+	result, err := tool.Call(context.Background(), input, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return result
+}
+
+func TestIdentityAcrossRoutes(t *testing.T) {
+	for _, ua := range []string{"", "Captured-UA/1.0"} {
+		for _, route := range []string{"direct", "proxy", "fallback"} {
+			t.Run(route+"/"+ua, func(t *testing.T) {
+				t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
+				var targetCalls, proxyCalls atomic.Int32
+				var mu sync.Mutex
+				var selected string
+				checkUA := func(r *http.Request) {
+					mu.Lock()
+					defer mu.Unlock()
+					got := r.UserAgent()
+					if ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
+						t.Errorf("random UA missing: %q", got)
+					} else if ua != "" && got != ua {
+						t.Errorf("User-Agent = %q, want %q", got, ua)
+					}
+					if selected != "" && selected != got {
+						t.Errorf("redirect/retry UA changed from %q to %q", selected, got)
+					}
+					selected = got
+				}
+				target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+					targetCalls.Add(1)
+					checkUA(r)
+					if r.URL.Path == "/start" {
+						http.Redirect(w, r, "/page", http.StatusFound)
+						return
+					}
+					w.Header().Set("Content-Type", "text/html")
+					fmt.Fprint(w, `<h1>Page</h1><!--keep--><script src="/app.js"></script><form action="/submit"><input name="csrf" type="hidden" value="token"></form>`)
+				}))
+				defer target.Close()
+				proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+					proxyCalls.Add(1)
+					checkUA(r)
+					if route == "fallback" {
+						conn, _, err := w.(http.Hijacker).Hijack()
+						if err != nil {
+							t.Error(err)
+							return
+						}
+						conn.Close() // Deterministic EOF, causing the SDK-compatible retry.
+						return
+					}
+					req := r.Clone(r.Context())
+					req.RequestURI = ""
+					// A forwarding proxy must preserve absence, like ARTEX's sink.
+					if _, present := req.Header["User-Agent"]; !present {
+						req.Header.Set("User-Agent", "")
+					}
+					resp, err := target.Client().Do(req)
+					if err != nil {
+						t.Error(err)
+						w.WriteHeader(http.StatusBadGateway)
+						return
+					}
+					defer resp.Body.Close()
+					for key, values := range resp.Header {
+						w.Header()[key] = values
+					}
+					w.WriteHeader(resp.StatusCode)
+					io.Copy(w, resp.Body)
+				}))
+				defer proxy.Close()
+				cfg := actool.WebFetchConfig{}
+				if route != "direct" {
+					cfg.Proxy = proxy.URL
+				}
+				result := callFetch(t, New(cfg), target.URL+"/start", true)
+				for _, want := range []string{"[200 OK]", "# Page", "=== RECON ===", "keep", "/app.js", "csrf(hidden=token)"} {
+					if result.IsError || !strings.Contains(result.Flatten(), want) {
+						t.Fatalf("missing %q in result: %+v", want, result)
+					}
+				}
+				if targetCalls.Load() != 2 {
+					t.Errorf("target calls = %d, want 2", targetCalls.Load())
+				}
+				if route != "direct" && proxyCalls.Load() == 0 {
+					t.Error("configured proxy was bypassed")
+				}
+			})
+		}
+	}
+}
+
+func TestTLSAndStaticAssetGuard(t *testing.T) {
+	t.Setenv("ARTEX_HTTP_USER_AGENT", "Captured-UA/1.0")
+	var calls atomic.Int32
+	server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		calls.Add(1)
+		if r.UserAgent() != "Captured-UA/1.0" {
+			t.Errorf("User-Agent = %q", r.UserAgent())
+		}
+		fmt.Fprint(w, "verified")
+	}))
+	defer server.Close()
+	certPath := filepath.Join(t.TempDir(), "ca.pem")
+	cert := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: server.Certificate().Raw})
+	if err := os.WriteFile(certPath, cert, 0600); err != nil {
+		t.Fatal(err)
+	}
+	tool := New(actool.WebFetchConfig{CACert: certPath})
+	if result := callFetch(t, tool, server.URL, false); result.IsError || !strings.Contains(result.Flatten(), "verified") {
+		t.Fatalf("CA trust lost: %+v", result)
+	}
+	if result := callFetch(t, tool, server.URL+"/app.js?v=1", false); !result.IsError {
+		t.Fatalf("static asset guard lost: %+v", result)
+	}
+	if calls.Load() != 1 {
+		t.Errorf("static asset triggered network request: calls = %d", calls.Load())
+	}
+}
+
+func TestIdentitySelectedForEachCall(t *testing.T) {
+	t.Setenv("ARTEX_HTTP_USER_AGENT", "Before-Tool-Creation")
+	tool := New(actool.WebFetchConfig{})
+	seen := make(chan string, 2)
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		seen <- r.UserAgent()
+		fmt.Fprint(w, "ok")
+	}))
+	defer server.Close()
+	for _, want := range []string{"First-Call", "Second-Call"} {
+		t.Setenv("ARTEX_HTTP_USER_AGENT", want)
+		if result := callFetch(t, tool, server.URL, false); result.IsError {
+			t.Fatal(result.Flatten())
+		}
+		if got := <-seen; got != want {
+			t.Fatalf("identity cached across calls: got %q, want %q", got, want)
+		}
+	}
+}
+
+func TestSDKMetadataPreserved(t *testing.T) {
+	base := actool.NewWebFetch(actool.WebFetchConfig{})
+	local := New(actool.WebFetchConfig{})
+	if local.Name() != base.Name() || local.Description() != base.Description() || !reflect.DeepEqual(local.InputSchema(), base.InputSchema()) {
+		t.Fatal("SDK tool contract changed")
+	}
+}

selfupdate/github.go

@@ -9,6 +9,8 @@ import (
 	"strconv"
 	"strings"
 	"time"
+
+	"github.com/Autumn-27/artex/internal/httpidentity"
 )
 
 // Repo 是发布源。写死而不是做成配置项：更新源可配等于给任何能改配置的人一条
@@ -125,7 +127,7 @@ func FetchLatest(ctx context.Context, c *http.Client) (*Release, error) {
 		return nil, err
 	}
 	req.Header.Set("Accept", "application/vnd.github+json")
-	req.Header.Set("User-Agent", "artex-selfupdate")
+	httpidentity.Apply(req)
 
 	resp, err := c.Do(req)
 	if err != nil {

selfupdate/github_test.go

@@ -56,3 +56,38 @@ func TestFetchLatestMissingRelease(t *testing.T) {
 		t.Fatalf("unexpected missing-release error: %v", err)
 	}
 }
+
+func TestUpdateUserAgent(t *testing.T) {
+	for _, tc := range []struct{ env, want string }{
+		{"", ""},
+		{"Captured-UA/1.0", "Captured-UA/1.0"},
+	} {
+		t.Run(tc.env, func(t *testing.T) {
+			t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
+			calls := 0
+			client := &http.Client{Transport: releaseTransport(func(req *http.Request) (*http.Response, error) {
+				calls++
+				if got := req.UserAgent(); tc.want == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
+					t.Errorf("random UA missing: %q", got)
+				} else if tc.want != "" && got != tc.want {
+					t.Errorf("User-Agent = %q, want %q", got, tc.want)
+				}
+				body := `{"tag_name":"v0.3.17"}`
+				return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(strings.NewReader(body))}, nil
+			})}
+			if _, err := FetchLatest(context.Background(), client); err != nil {
+				t.Fatal(err)
+			}
+			for _, name := range []string{"SHA256SUMS", "artex.zip"} {
+				body, err := get(context.Background(), client, "https://github.com/RuoJi6/ARTEX/releases/download/v0.3.17/"+name)
+				if err != nil {
+					t.Fatal(err)
+				}
+				body.Close()
+			}
+			if calls != 3 {
+				t.Fatalf("calls = %d, want 3", calls)
+			}
+		})
+	}
+}

selfupdate/stage.go

@@ -13,6 +13,8 @@ import (
 	"runtime"
 	"strings"
 	"time"
+
+	"github.com/Autumn-27/artex/internal/httpidentity"
 )
 
 // sumsAsset 是 release.yml 生成的校验和清单，覆盖 Release 里全部 zip。
@@ -229,7 +231,7 @@ func get(ctx context.Context, c *http.Client, rawURL string) (io.ReadCloser, err
 	if err := checkURL(req.URL); err != nil {
 		return nil, err
 	}
-	req.Header.Set("User-Agent", "artex-selfupdate")
+	httpidentity.Apply(req)
 	resp, err := c.Do(req)
 	if err != nil {
 		return nil, err

server/customtool.go

@@ -18,6 +18,7 @@ import (
 	"time"
 
 	"github.com/Autumn-27/artex/db"
+	"github.com/Autumn-27/artex/internal/httpidentity"
 	"github.com/Autumn-27/norma/permission"
 	actool "github.com/Autumn-27/norma/tool"
 )
@@ -438,6 +439,7 @@ func (s *Server) runHTTPTool(ctx context.Context, execRaw json.RawMessage, param
 	if err != nil {
 		return actool.Errorf(maskSecrets(err.Error())), nil
 	}
+	httpidentity.Apply(req)
 	for k, v := range spec.Headers {
 		req.Header.Set(k, renderTemplate(v, params, identity))
 	}

server/egress_probe.go

@@ -23,6 +23,7 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/Autumn-27/artex/internal/httpidentity"
 	"github.com/Autumn-27/artex/traffic"
 )
 
@@ -305,7 +306,7 @@ func probeOne(ctx context.Context, client *http.Client, target string) (string,
 	if err != nil {
 		return "", err
 	}
-	req.Header.Set("User-Agent", "artex-egress-probe/1.0")
+	httpidentity.Apply(req)
 
 	resp, err := client.Do(req)
 	if err != nil {

server/egress_probe_test.go

@@ -27,6 +27,27 @@ func fakeIPServer(ip string) *httptest.Server {
 	}))
 }
 
+func TestProbeOneUserAgent(t *testing.T) {
+	for _, ua := range []string{"", "Captured-UA/1.0"} {
+		t.Run(ua, func(t *testing.T) {
+			t.Setenv("ARTEX_HTTP_USER_AGENT", ua)
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				if got := r.UserAgent(); ua == "" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
+					t.Errorf("random UA missing: %q", got)
+				} else if ua != "" && got != ua {
+					t.Errorf("User-Agent = %q, want %q", got, ua)
+				}
+				fmt.Fprint(w, `{"ip":"203.0.113.7"}`)
+			}))
+			defer server.Close()
+			ip, err := probeOne(context.Background(), server.Client(), server.URL)
+			if err != nil || ip != "203.0.113.7" {
+				t.Fatalf("ip = %q, err = %v", ip, err)
+			}
+		})
+	}
+}
+
 // TestProbeEgress 覆盖 probeEgress 的输入/响应形态：成功、非法协议、死代理、坏响应、回落。
 // 每条用例自带服务端与断言，失败时 t.Run 的名字直接指出是哪一条。
 func TestProbeEgress(t *testing.T) {

server/http_identity_test.go

@@ -0,0 +1,54 @@
+package server
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+)
+
+func TestTargetHTTPIdentityOverrides(t *testing.T) {
+	t.Setenv(repeaterAllowLocalEnv, "1")
+	for _, route := range []string{"custom", "repeater"} {
+		for _, tc := range []struct {
+			name, env, want string
+			headers         map[string]string
+		}{
+			{name: "default"},
+			{name: "environment", env: "Captured-UA/1.0", want: "Captured-UA/1.0"},
+			{name: "override", env: "Environment-UA/1.0", want: "Explicit-UA/1.0", headers: map[string]string{"user-agent": "Explicit-UA/1.0"}},
+			{name: "explicit-empty", env: "Environment-UA/1.0", headers: map[string]string{"User-Agent": ""}},
+		} {
+			t.Run(route+"/"+tc.name, func(t *testing.T) {
+				t.Setenv("ARTEX_HTTP_USER_AGENT", tc.env)
+				server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+					if got := r.UserAgent(); tc.name == "default" && !strings.HasPrefix(got, "Mozilla/5.0 ") {
+						t.Errorf("random UA missing: %q", got)
+					} else if tc.name != "default" && got != tc.want {
+						t.Errorf("User-Agent = %q, want %q", got, tc.want)
+					}
+					if _, present := r.Header["User-Agent"]; tc.name == "explicit-empty" && present {
+						t.Error("empty User-Agent present on wire")
+					}
+					w.Write([]byte("ok"))
+				}))
+				defer server.Close()
+				payload := map[string]any{"url": server.URL, "headers": tc.headers, "use_recording_proxy": false}
+				if route == "custom" {
+					raw, _ := json.Marshal(payload)
+					result, err := (&Server{}).runHTTPTool(context.Background(), raw, nil, nil)
+					if err != nil || result.IsError {
+						t.Fatalf("custom tool: %v %+v", err, result)
+					}
+				} else {
+					status, result, body := repeaterTestSend(t, &Server{}, payload)
+					if status != http.StatusOK || result.Status != http.StatusOK {
+						t.Fatalf("repeater: %d %s", status, body)
+					}
+				}
+			})
+		}
+	}
+}

server/repeater.go

@@ -16,6 +16,8 @@ import (
 	"strings"
 	"syscall"
 	"time"
+
+	"github.com/Autumn-27/artex/internal/httpidentity"
 )
 
 const (
@@ -141,6 +143,7 @@ func (s *Server) sendRepeater(w http.ResponseWriter, r *http.Request) {
 		writeErr(w, http.StatusBadRequest, "构造请求失败")
 		return
 	}
+	httpidentity.Apply(req)
 	for key, value := range in.Headers {
 		name := strings.TrimSpace(key)
 		if name == "" {

skills/api-recon/scripts/harvest_static.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-harvest_static.py <BASE_URL> <OUTDIR>
+harvest_static.py <BASE_URL> <OUTDIR> [--user-agent VALUE]
 
 参考模板 — 非通用成品。执行前须按目标站点调整，常见改动：
   - extract_endpoints 正则（endpoint 方言）
@@ -16,16 +16,54 @@ Static SPA bundle harvester. Framework-agnostic; tuned for webpack + Vite.
 
 Stdlib only. TLS verification is disabled (recon against self-signed/internal hosts).
 """
-import sys, os, re, ssl, json, urllib.request, urllib.parse
+import sys, os, re, ssl, secrets, json, argparse, urllib.request, urllib.parse
 from concurrent.futures import ThreadPoolExecutor
 
 CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
-UA = "Mozilla/5.0 (spa-api-recon)"
+USER_AGENTS = (
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
+)
 
-def fetch(url, binary=False):
+
+def user_agent(value=None):
+    """Explicit value > environment > random sample; explicit empty omits UA."""
+    explicit = value is not None
+    if value is None:
+        value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
+    value = value.strip(" ")
+    if not all(32 <= ord(c) <= 126 for c in value):
+        return secrets.choice(USER_AGENTS)
+    if value or explicit:
+        return value
+    return secrets.choice(USER_AGENTS)
+
+
+def fetch(url, binary=False, user_agent_value=None):
     try:
-        req = urllib.request.Request(url, headers={"User-Agent": UA})
-        with urllib.request.urlopen(req, context=CTX, timeout=25) as r:
+        ua = user_agent(user_agent_value)
+        req = urllib.request.Request(url, headers={"User-Agent": ua} if ua else {})
+        opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
+        opener.addheaders = []  # Do not reintroduce Python-urllib's default UA.
+        with opener.open(req, timeout=25) as r:
             data = r.read()
             return (data if binary else data.decode("utf-8", "ignore")), r.status
     except Exception as e:
@@ -112,14 +150,17 @@ def extract_routes(js_text):
 
 # ---- main -------------------------------------------------------------------
 def main():
-    if len(sys.argv) < 3:
-        print("usage: harvest_static.py <BASE_URL> <OUTDIR>"); sys.exit(1)
-    base, outdir = sys.argv[1], sys.argv[2]
+    ap = argparse.ArgumentParser()
+    ap.add_argument("base"); ap.add_argument("outdir")
+    ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
+    args = ap.parse_args()
+    base, outdir = args.base, args.outdir
+    ua = user_agent(args.user_agent)
     if not base.startswith("http"): base = "https://" + base
     jsdir = os.path.join(outdir, "js"); os.makedirs(jsdir, exist_ok=True)
 
     print(f"[*] entry: {base}")
-    html, status = fetch(base)
+    html, status = fetch(base, user_agent_value=ua)
     open(os.path.join(outdir, "index.html"), "w").write(html)
     origin = origin_of(base)
 
@@ -137,7 +178,7 @@ def main():
         if not fn.endswith(".js"): return None
         lp = os.path.join(jsdir, fn)
         if url in have: return have[url]
-        data, st = fetch(url, binary=True)
+        data, st = fetch(url, binary=True, user_agent_value=ua)
         if st == 200 and data and not data[:15].lstrip().startswith(b"<"):
             open(lp, "wb").write(data); have[url] = lp; return lp
         return None

skills/api-recon/scripts/spider_mpa.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4]
+spider_mpa.py <BASE_URL> <OUTDIR> [--cookie "k=v; k2=v2"] [--max 200] [--depth 4] [--user-agent VALUE]
 
 参考模板 — 非通用成品。执行前须按目标调整 --exclude、cookie、depth/max 等同域策略。
 
@@ -14,20 +14,58 @@ the same origin and extracts:
 
 Stdlib only. Same-origin, bounded, polite. Provide --cookie for an authed crawl.
 """
-import sys, os, re, ssl, argparse, urllib.request, urllib.parse, time
+import sys, os, re, ssl, secrets, argparse, urllib.request, urllib.parse, time
 from html.parser import HTMLParser
 from collections import deque
 
 CTX = ssl.create_default_context(); CTX.check_hostname = False; CTX.verify_mode = ssl.CERT_NONE
-UA = "Mozilla/5.0 (spa-api-recon spider)"
+USER_AGENTS = (
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0.0.0 Safari/537.36',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36 Edg/157.0.0.0',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 Edg/156.0.0.0',
+    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (X11; Linux x86_64; rv:159.0) Gecko/20100101 Firefox/159.0',
+    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/27.1 Safari/605.1.15',
+)
+
+
+def user_agent(value=None):
+    """Explicit value > environment > random sample; explicit empty omits UA."""
+    explicit = value is not None
+    if value is None:
+        value = os.environ.get("ARTEX_HTTP_USER_AGENT", "")
+    value = value.strip(" ")
+    if not all(32 <= ord(c) <= 126 for c in value):
+        return secrets.choice(USER_AGENTS)
+    if value or explicit:
+        return value
+    return secrets.choice(USER_AGENTS)
+
 SKIP_EXT = re.compile(r'\.(css|png|jpe?g|gif|svg|ico|woff2?|ttf|pdf|zip|mp4|webp|js)(\?|$)', re.I)
 API_HINT = re.compile(r'/(?:api|rest|service|ajax|action|do|rpc|graphql|v\d|admin|backend)\b', re.I)
 
-def fetch(url, cookie):
-    h = {"User-Agent": UA}
+def fetch(url, cookie, user_agent_value=None):
+    ua = user_agent(user_agent_value)
+    h = {"User-Agent": ua} if ua else {}
     if cookie: h["Cookie"] = cookie
     try:
-        with urllib.request.urlopen(urllib.request.Request(url, headers=h), context=CTX, timeout=20) as r:
+        opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=CTX))
+        opener.addheaders = []  # Do not reintroduce Python-urllib's default UA.
+        with opener.open(urllib.request.Request(url, headers=h), timeout=20) as r:
             ct = r.headers.get("Content-Type", "")
             if "html" not in ct and "xml" not in ct: return "", ct
             return r.read().decode("utf-8", "ignore"), ct
@@ -70,7 +108,9 @@ def main():
     ap.add_argument("--cookie", default=""); ap.add_argument("--max", type=int, default=200)
     ap.add_argument("--depth", type=int, default=4); ap.add_argument("--delay", type=float, default=0.1)
     ap.add_argument("--include", default=""); ap.add_argument("--exclude", default="logout|signout|delete|remove|destroy")
+    ap.add_argument("--user-agent", default=None, help="User-Agent override; default: ARTEX_HTTP_USER_AGENT or random browser sample")
     args = ap.parse_args()
+    ua = user_agent(args.user_agent)
     base = args.base if args.base.startswith("http") else "https://" + args.base
     os.makedirs(args.outdir, exist_ok=True)
     origin = "{0.scheme}://{0.netloc}".format(urllib.parse.urlparse(base))
@@ -82,7 +122,7 @@ def main():
     n = 0
     while q and n < args.max:
         url, d = q.popleft(); n += 1
-        html, ct = fetch(url, args.cookie)
+        html, ct = fetch(url, args.cookie, user_agent_value=ua)
         if args.delay: time.sleep(args.delay)
         if not html: continue
         p = Page()

skills/api-recon/scripts/test_user_agent.py

@@ -0,0 +1,124 @@
+"""Offline regressions: python3 -m unittest discover -s skills/api-recon/scripts."""
+import contextlib
+import http.server
+import importlib.util
+import os
+from pathlib import Path
+import re
+import subprocess
+import sys
+import tempfile
+import threading
+import unittest
+from unittest.mock import patch
+
+
+SCRIPTS = Path(__file__).resolve().parent
+# Keep the offline fixtures independent of the runner's outbound/recording proxy.
+DIRECT_ENV = {"no_proxy": "*", "NO_PROXY": "*"}
+
+
+def load_script(name):
+    spec = importlib.util.spec_from_file_location(name, SCRIPTS / f"{name}.py")
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+@contextlib.contextmanager
+def capture_server():
+    seen = []
+
+    class Handler(http.server.BaseHTTPRequestHandler):
+        def do_GET(self):
+            seen.append((self.path, self.headers.get("User-Agent"), self.headers.get("Cookie")))
+            if self.path == "/start":
+                self.send_response(302)
+                self.send_header("Location", "/page")
+                self.end_headers()
+                return
+            self.send_response(200)
+            self.send_header("Content-Type", "text/html")
+            self.end_headers()
+            self.wfile.write(b'<h1>Page</h1><script src="/app.js"></script><form action="/submit"></form>')
+
+        def log_message(self, *args):
+            pass
+
+    with http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) as server:
+        thread = threading.Thread(target=server.serve_forever)
+        thread.start()
+        try:
+            yield f"http://127.0.0.1:{server.server_port}", seen
+        finally:
+            server.shutdown()
+            thread.join()
+
+
+class UserAgentTests(unittest.TestCase):
+    def assert_identity(self, seen, expected, samples):
+        self.assertTrue(seen)
+        if expected == "random":
+            self.assertIn(seen[0][1], samples)
+            expected = seen[0][1]
+        self.assertTrue(all(ua == expected for _, ua, _ in seen), seen)
+
+    def test_random_selection_and_explicit_override(self):
+        go_source = (SCRIPTS.parents[2] / "internal/httpidentity/useragent.go").read_text()
+        go_pool = tuple(re.findall(r'^\s*"(Mozilla/5\.0 [^"]+)",$', go_source, re.MULTILINE))
+        self.assertEqual(len(go_pool), 20)
+        self.assertEqual(len(set(go_pool)), 20)
+        for name in ("harvest_static", "spider_mpa"):
+            module = load_script(name)
+            self.assertEqual(module.USER_AGENTS, go_pool)
+            for sample in module.USER_AGENTS:
+                with self.subTest(script=name, sample=sample):
+                    with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": ""}), patch.object(module.secrets, "choice", return_value=sample) as choose:
+                        self.assertEqual(module.user_agent(), sample)
+                        choose.assert_called_once_with(module.USER_AGENTS)
+                    with patch.dict(os.environ, {"ARTEX_HTTP_USER_AGENT": "Configured-UA"}), patch.object(module.secrets, "choice") as choose:
+                        self.assertEqual(module.user_agent(), "Configured-UA")
+                        self.assertEqual(module.user_agent("Override-UA"), "Override-UA")
+                        self.assertEqual(module.user_agent(""), "")
+                        choose.assert_not_called()
+
+    def test_fetch_and_redirect(self):
+        for name in ("harvest_static", "spider_mpa"):
+            module = load_script(name)
+            for value, expected in (("", "random"), (" Captured-UA/1.0 ", "Captured-UA/1.0"),
+                                    ("ok\r\nX-Injected: yes", "random"), ("客户端", "random")):
+                with self.subTest(script=name, value=value), capture_server() as (base, seen):
+                    with patch.dict(os.environ, {**DIRECT_ENV, "ARTEX_HTTP_USER_AGENT": value}):
+                        if name == "harvest_static":
+                            body, status = module.fetch(base + "/start")
+                            self.assertEqual(status, 200)
+                        else:
+                            body, _ = module.fetch(base + "/start", "session=sample")
+                            self.assertTrue(all(cookie == "session=sample" for _, _, cookie in seen))
+                    self.assertIn("Page", body)
+                    self.assertEqual([path for path, _, _ in seen], ["/start", "/page"])
+                    self.assert_identity(seen, expected, module.USER_AGENTS)
+
+    def test_cli_precedence_and_outputs(self):
+        for name in ("harvest_static", "spider_mpa"):
+            module = load_script(name)
+            for env_ua, flags, expected in (("", [], "random"),
+                                           ("Environment-UA/1.0", [], "Environment-UA/1.0"),
+                                           ("Environment-UA/1.0", ["--user-agent", "Override-UA/1.0"], "Override-UA/1.0"),
+                                           ("Environment-UA/1.0", ["--user-agent", ""], None)):
+                with self.subTest(script=name, flags=flags), capture_server() as (base, seen):
+                    with tempfile.TemporaryDirectory() as outdir:
+                        env = dict(os.environ, **DIRECT_ENV, ARTEX_HTTP_USER_AGENT=env_ua)
+                        command = [sys.executable, str(SCRIPTS / f"{name}.py"), base + "/start", outdir] + flags
+                        if name == "spider_mpa":
+                            command += ["--max", "1", "--delay", "0"]
+                        result = subprocess.run(command, env=env, capture_output=True, text=True, timeout=15)
+                        self.assertEqual(result.returncode, 0, result.stderr)
+                        self.assertGreaterEqual(len(seen), 2)
+                        self.assert_identity(seen, expected, module.USER_AGENTS)
+                        artifact = "api_static.txt" if name == "harvest_static" else "forms.txt"
+                        self.assertTrue((Path(outdir) / artifact).exists())
+
+
+if __name__ == "__main__":
+    unittest.main()

traffic/traffic.go

@@ -412,6 +412,20 @@ type sink struct {
 	t *Traffic
 }
 
+// Preserve an absent User-Agent when go-mitmproxy creates its outbound Go
+// request. An explicit empty value suppresses net/http's automatic default.
+func (s *sink) Requestheaders(f *mproxy.Flow) {
+	if f.Request == nil {
+		return
+	}
+	if f.Request.Header == nil {
+		f.Request.Header = make(http.Header)
+	}
+	if _, present := f.Request.Header["User-Agent"]; !present {
+		f.Request.Header.Set("User-Agent", "")
+	}
+}
+
 func (s *sink) Response(f *mproxy.Flow) {
 	if f.Request == nil || f.Response == nil {
 		return
@@ -683,6 +697,9 @@ func headerLines(h map[string][]string) string {
 func requestHeaderLines(req *mproxy.Request) string {
 	headers := req.Header.Clone()
 	headers.Del("Host")
+	if headers.Get("User-Agent") == "" {
+		headers.Del("User-Agent") // Empty suppression value is not sent on the wire.
+	}
 	host := req.URL.Host
 	if raw := req.Raw(); raw != nil && strings.TrimSpace(raw.Host) != "" {
 		host = raw.Host

traffic/useragent_test.go

@@ -0,0 +1,48 @@
+package traffic
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"testing"
+
+	mproxy "github.com/lqqyt2423/go-mitmproxy/proxy"
+)
+
+func TestProxyPreservesUserAgent(t *testing.T) {
+	for _, ua := range []string{"", "Captured-UA/1.0"} {
+		t.Run(ua, func(t *testing.T) {
+			server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+				if got := r.UserAgent(); got != ua {
+					t.Errorf("forwarded User-Agent = %q, want %q", got, ua)
+				}
+				if _, present := r.Header["User-Agent"]; ua == "" && present {
+					t.Error("proxy reintroduced a User-Agent")
+				}
+			}))
+			defer server.Close()
+			u, _ := url.Parse(server.URL)
+			flow := &mproxy.Flow{Request: &mproxy.Request{URL: u, Header: make(http.Header)}}
+			if ua != "" {
+				flow.Request.Header.Set("User-Agent", ua)
+			}
+			(&sink{}).Requestheaders(flow)
+			// go-mitmproxy rebuilds the outbound request by adding every value.
+			req, _ := http.NewRequest(http.MethodGet, server.URL, nil)
+			for key, values := range flow.Request.Header {
+				for _, value := range values {
+					req.Header.Add(key, value)
+				}
+			}
+			resp, err := server.Client().Do(req)
+			if err != nil {
+				t.Fatal(err)
+			}
+			resp.Body.Close()
+			if ua == "" && strings.Contains(requestHeaderLines(flow.Request), "User-Agent:") {
+				t.Error("recording contains a synthetic header that was not sent")
+			}
+		})
+	}
+}

web/src/lib/mock/data.ts

@@ -2921,7 +2921,6 @@ export const trafficDetail: TrafficDetail = {
   req: `GET /v1/orders?id=1002 HTTP/1.1
 Host: api.acme.com
 Authorization: Bearer eyJhbGciOiJIUzI1NiJ9...
-User-Agent: ARTEX-worker/1.0
 Accept: application/json
 
 `,