Agent Harness

The boundaries here are harder than on the model side, and most of them are engineering and trust problems rather than algorithmic ones. No common ruler: this domain has almost no accepted leaderboard. Terminal-Bench, OSWorld and GAIA measure a model paired with one specific scaffold, and the score moves when the harness changes, so ranking two runtimes against each other is basically not possible - which is why this domain records checkable engineering facts here (persistence, sandbox boundaries, size of the skill ecosystem) instead of completion-rate tables. Self-improvement is unproven: projects such as hermes-agent sell the idea that the agent writes its own skills after a task and then refines them in use. The claim is strong, and falsifiable controlled experiments (same model, same tasks, loop on versus loop off) are almost absent from public data, so all of it is marked needs-reproduction. Permissions and safety are the weak spot: most open-source runtimes either have no permission system at all (pi's README says so itself) or only a confirmation prompt. An agent that can run a shell, edit files and reach the network crosses a line after one prompt injection, so sandboxing is a requirement, not an option. Stars do not equal usability: community heat at this layer tracks release cadence closely, and a project with tens of thousands of stars may still only support one model family on one operating system - you pay the evaluation cost yourself. Cost and latency are systematically underestimated: the runtime decides how many model calls a task needs and how much context each call carries. The same task can differ by several times in token bill between "one long context" and "many small steps plus retrieved memory", and tool-call retries inside long tasks amplify that again, so budget for the worst case.