feat(llm): LLM 配置轮询(故障转移),某个配置没额度/失效时自动切下一个

默认关闭。开启后「未指定模型」的 agent 在当前配置不可用(402 余额不足 /
401 key 失效 / 429 限流 / 5xx)时自动切到下一个配置;agent 绑定或任务 pin
了配置的默认不轮询(绑定即独占,可由 llm_pool_bind_fallback 打开兜底)。

链序 = 激活配置 + 其余按 priority DESC,排除 pool_exclude 的。只给
llm_profiles 加两列,不引入关联表,所以 agents/tasks/conversations 对
llm_profile_id 的四处引用一行未改。

核心约束:只在本次调用尚未 yield 出任何事件时切换成员。已经吐出
text_delta / tool_use_start 后中途断流一律原样上抛(交给 agentcore 的
resume),否则重发同一 request 会重复输出、污染会话历史。SDK 的 doStream
保证 4xx/5xx 在读 body 前返回,所以目标场景都落在这个安全窗口内。

context.Canceled / DeadlineExceeded 绝不触发切换——那是用户取消或任务
超时,误判会白烧备用 key 并污染 a6bd180 的终止原因诊断。400 也不切
(请求本身有问题,换模型结果一样)。

熔断:402/401/403/404 一次即熔断,429/5xx/网络错误连续 3 次;冷却
60s→5min→30min 退避,到期半开试探,成功清零。Registry 是进程级的,独立
于 provider 缓存,重建链路不丢状态,且全进程共享(A 任务撞见的 402,B 任务
立刻知道)。状态旁路落库,重启只恢复尚未到期的冷却窗口。

上下文窗口装不下当前请求的成员直接跳过(必然 400),不把主力模型的窗口
压到全链最小值。全员熔断时仍探链首,不让引擎停摆。

Pool 包在 llmrec 外面、每个成员自带 recorder,失败调用会以 status=error
+ 正确的 profile 名进 llm_records;切换/熔断走 log.Printf,自动出现在
系统日志页。

顺带修两个既有问题:
- DecomposeGoals 原本自己 NewProvider(),绕过限速器/录制/轮询,且每个
  任务新建一个实例——改为由 server 传入共享 provider
- saveLLMConfig 会把 priority/pool_exclude 重置为零值

前端:LLM 页新增轮询卡片(总开关/兜底开关/实际链路顺序/健康徽章/冷却
倒计时/立即恢复),配置编辑区加优先级与「不参与轮询」。

只有一个配置时链长度为 1,poolChain 返回 nil 走裸 provider,行为与开关
打开前完全一致。设计见 docs/LLM轮询设计.md。
This commit is contained in:
Autumn-27 committed 2026-08-19 11:07:53 -04:00
1 parent 89c5e9b26f
commit 0c653a2135
19 files changed
+1750 -50

No files matched your search

+261
View File
@@ -0,0 +1,261 @@
package llmpool
import (
"context"
"errors"
"fmt"
"iter"
"log"
"regexp"
"strconv"
"strings"
"sync/atomic"
"github.com/Autumn-27/norma/llm"
)
// Member is one LLM profile in the chain, already built into a provider (wrapped
// with the recorder by the caller, so a failed attempt is still recorded under
// its own profile name).
type Member struct {
ID int64 // llm_profiles.id
Name string // profile name, for logs / UI
Model string
Format string // "anthropic" | "openai"
Priority int // the profile's configured priority (display only)
Active bool // is_default (display only)
// Rank is the ordering key the caller assigned: higher goes first, and members
// sharing a Rank take turns leading (load-spreading across duplicate keys).
// The caller encodes "active profile heads the chain" as a Rank above every
// user-settable priority, so this type needs no policy of its own.
Rank int
// WindowTokens is the profile's context window in tokens. A member whose
// window can't hold the request is skipped rather than made to fail on it.
WindowTokens int
Prov llm.Provider
}
// RankActive is the Rank the caller gives the chain head (the active profile, or
// an explicitly bound one) so it always outranks any configured priority.
const RankActive = int(^uint(0)>>1) - 1
// ErrExhausted is returned when every member of the chain failed.
var ErrExhausted = errors.New("LLM 轮询:所有配置均不可用")
// Pool is an llm.Provider that fails over across an ordered chain of members.
// It is safe for concurrent use: members are immutable after construction and
// all mutable state lives in the shared Registry.
type Pool struct {
members []*Member // in chain order (active first, then priority DESC)
health *Registry
rr atomic.Uint64 // rotates the starting point within an equal-priority group
}
// New builds a Pool over members (already in chain order). Returns nil when the
// chain is empty. A single-member chain is still a valid Pool — it just behaves
// exactly like the bare provider.
func New(members []*Member, health *Registry) *Pool {
if len(members) == 0 {
return nil
}
if health == nil {
health = NewRegistry(nil, nil)
}
return &Pool{members: members, health: health}
}
// Members returns the chain in order (read-only).
func (p *Pool) Members() []*Member { return p.members }
// Head returns the first member of the chain.
func (p *Pool) Head() *Member { return p.members[0] }
// Stream implements llm.Provider with failover.
//
// The one hard rule: a member may only be abandoned BEFORE it has yielded any
// event. Once text or a tool_use has reached the caller, re-sending the same
// request to another model would duplicate output and corrupt the conversation
// history — so a mid-stream failure is surfaced as-is and left to the agent
// harness's resume logic. Fortunately the failures this exists for (402 no
// credit, 401 bad key, 429, 5xx) all surface during request establishment,
// before the body is read, so they always land in the safe window.
func (p *Pool) Stream(ctx context.Context, req llm.CompletionRequest) iter.Seq2[llm.StreamEvent, error] {
order := p.order(req)
return func(yield func(llm.StreamEvent, error) bool) {
var lastErr error
for i, m := range order {
emitted := false
var failed error
for ev, err := range m.Prov.Stream(ctx, req) {
if err != nil && !emitted && shouldFailover(ctx, err) {
failed = err
break // safe window: nothing reached the caller yet
}
emitted = true
if !yield(ev, err) {
return // caller stopped consuming (cancel / early exit)
}
if err != nil {
return // terminal error already handed to the caller
}
}
if failed == nil {
p.health.Pass(m.ID) // completed (or failed in a non-failover way)
return
}
lastErr = failed
hard := isHardFailure(failed)
if p.health.Trip(m.ID, trimErr(failed), hard) {
log.Printf("[llmpool] 配置 %q(%s) 已熔断:%s", m.Name, m.Model, trimErr(failed))
}
if i+1 < len(order) {
n := order[i+1]
log.Printf("[llmpool] LLM 故障转移:%q(%s) → %q(%s),原因:%s",
m.Name, m.Model, n.Name, n.Model, trimErr(failed))
}
}
if lastErr == nil {
lastErr = ErrExhausted
}
log.Printf("[llmpool] 轮询链已耗尽(%d 个配置全部失败),最后错误:%s", len(order), trimErr(lastErr))
yield(llm.StreamEvent{}, fmt.Errorf("%w:%v", ErrExhausted, lastErr))
}
}
// order picks the members to try, in order: skip those in a cooling-off window
// and those whose context window can't hold this request, then rotate within each
// equal-priority group so same-priority profiles share the load. Never returns an
// empty slice — if everything is filtered out, the head of the chain is tried
// anyway, since stalling the engine is worse than one more failed request.
func (p *Pool) order(req llm.CompletionRequest) []*Member {
est := estimateTokens(req)
var open []*Member
for _, m := range p.members {
if p.health.IsOpen(m.ID) {
continue
}
if m.WindowTokens > 0 && est > m.WindowTokens {
continue // would 400 on length — not a useful failover target
}
open = append(open, m)
}
if len(open) == 0 {
return p.members[:1] // last resort: probe the head rather than stall
}
return rotateGroups(open, p.rr.Add(1)-1)
}
// rotateGroups rotates each run of equal-ranked members by n, so profiles sharing
// a priority take turns going first (free load-spreading across duplicate keys).
// The chain head has its own Rank and always stays at the front.
func rotateGroups(in []*Member, n uint64) []*Member {
out := make([]*Member, 0, len(in))
for i := 0; i < len(in); {
j := i + 1
for j < len(in) && in[j].Rank == in[i].Rank {
j++
}
g := in[i:j]
if len(g) > 1 {
off := int(n % uint64(len(g)))
for k := range g {
out = append(out, g[(k+off)%len(g)])
}
} else {
out = append(out, g...)
}
i = j
}
return out
}
// statusRe pulls the HTTP status out of the SDK's error text, which is formatted
// as "<prefix>: status <code>: <body>" (norma/llm/retry.go). The SDK exposes no
// typed error, so the string is what we have.
var statusRe = regexp.MustCompile(`status (\d{3})`)
// statusOf returns the HTTP status carried by err, or 0 if it isn't one.
func statusOf(err error) int {
m := statusRe.FindStringSubmatch(err.Error())
if m == nil {
return 0
}
code, _ := strconv.Atoi(m[1])
return code
}
// shouldFailover reports whether err justifies trying the next profile.
//
// Never fails over on context cancellation — that's the user stopping a task or a
// task-level timeout, and burning a backup key on it would both waste credit and
// pollute the run's termination diagnosis. Never on 400 either: a malformed or
// over-long request fails identically everywhere.
func shouldFailover(ctx context.Context, err error) bool {
if err == nil {
return false
}
if ctx.Err() != nil || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
switch code := statusOf(err); {
case code == 0:
return true // no status → transport-level failure (reset / DNS / timeout)
case code == 400:
return false // bad or over-long request: identical everywhere
case code == 401, code == 402, code == 403, code == 404, code == 408, code == 429:
return true
case code >= 500:
return true
}
return false
}
// isHardFailure reports whether the failure is deterministic (the profile will
// keep failing until a human fixes it) rather than transient. Hard failures open
// the breaker on the first occurrence.
func isHardFailure(err error) bool {
switch statusOf(err) {
case 401, 402, 403, 404:
return true
}
return false
}
// trimErr shortens an error for logs/UI — provider bodies can be long.
func trimErr(err error) string {
s := strings.TrimSpace(strings.ReplaceAll(err.Error(), "\n", " "))
if len(s) > 300 {
s = s[:300] + "…"
}
return s
}
// estimateTokens roughly sizes a request so members whose context window clearly
// can't hold it are skipped. Deliberately crude (~3.5 chars/token) and biased to
// over-estimate slightly; it only needs to tell "fits" from "nowhere near".
func estimateTokens(req llm.CompletionRequest) int {
n := 0
for _, s := range req.System {
n += len(s)
}
for _, m := range req.Messages {
n += blocksLen(m.Content)
}
for _, t := range req.Tools {
// InputSchema is a decoded map, so its serialized size isn't available
// cheaply — charge a flat ~120 chars per top-level property instead.
n += len(t.Name) + len(t.Description) + len(t.InputSchema)*120
}
return n * 2 / 7 // ≈ len/3.5
}
func blocksLen(bs []llm.ContentBlock) int {
n := 0
for _, b := range bs {
n += len(b.Text) + len(b.Thinking) + len(b.Input) + len(b.Name)
if len(b.Content) > 0 {
n += blocksLen(b.Content)
}
}
return n
}