AI Coding Evaluation Topic Map

這是 AI coding agent 評測、trace、post-training 與軟體品質控制的主題導覽。

Evaluation as product spec

• eval-is-spec — eval dataset 作為產品規格
• replit-agent-eval-scale — Replit ViBench + Telescope + Eval as Stream
• test-time-compute-evaluation — 模型表現作為推論時算力函數
• bayesian-teaching-llm-reasoning — 用 Bayesian traces 檢查互動式信念更新與 domain transfer
• user-simulator-evaluation — 測量 synthetic users 與真人互動分布的 realism gap
• model-harness-fit — 評估 agent 時把模型與 harness 配對一起量測
• knowledge-profiling-factuality — 用 fact-level encoding／recall／recognition 拆解 factuality bottleneck

Trace and improvement

• agent-trace-observability — trace 觀察 agent 決策與錯誤
• coding-agent-as-optimizer — coding agent 作為外層優化器
• agent-post-training-workflow — 從 coding agent traces 到 SFT baseline

Engineering practices

• build-for-next-model — 為下一個模型而寫
• model-specific-system-prompts — 模型特定系統指令
• harness-engineering-for-ai-coding — AI coding agent 的驗證回饋面

Raw sources

• raw/articles/aihao-eval-is-spec-2026-05-20.md
• raw/articles/aihao-generic-ai-metrics-mirage-2026-02-17.md
• raw/articles/google-research-better-ai-benchmarks-raters-2026-03-31.md
• raw/articles/google-research-convapparel-user-simulators-2026-04-09.md
• raw/articles/google-research-dialoglab-human-ai-group-conversations-2026-02-10.md
• raw/articles/google-research-bayesian-teaching-llm-reasoning-2026-03-04.md
• raw/articles/aihao-traces-new-source-of-truth-2026-02-17.md
• raw/articles/aihao-replit-agent-eval-scale-2026-06-01.md
• raw/articles/aihao-harness-engineering-model-harness-fit-2026-06-26.md
• raw/articles/google-research-knowledge-profiling-parametric-factuality-2026-08-12.md
• raw/articles/bnext-gemini-3-7-flash-2026-08-14.md
• raw/transcripts/training-agents-coding-agent-continual-learning-2026-06-22.md
• raw/transcripts/hermes-model-specific-system-prompts-2026-06-18.md