OCTOPUS docsocto.ad

Docs / Comparisons / Questions to ask any harness vendor

Questions to ask any harness vendor

The questions to put to any vendor, one per dimension.

What to look for

Ask these questions. Does memory persist across sessions, agents and teams, or per run? Is context checked against a schema before it reaches the model? Are decisions recorded with options, reasoning and who approved? Is autonomy set per lane, with a kill switch? Does a task close on evidence or on the agent's report? Are approvals sorted by reversibility? Is tenant isolation structural? Can outside agents connect over a standard protocol?

One question per dimension

  1. Memory persistence. Show me what one agent learned last month, queried by another agent today.
  2. Context assembly. Show the context for one task and the schema it was checked against.
  3. Entity resolution. Show one person who appears in three sources, as one record.
  4. Decision record. Replay one consequential decision: the options, the reasoning, who approved it.
  5. Autonomy control. Show the autonomy setting for one lane and the kill switch for one new behavior.
  6. Task closure. Show a task the agent reported done that the platform refused to close.
  7. Approval model. How does your approval flow differ between a draft edit and an outbound payment?
  8. Failure handling. Show the record for your last agent failure and the fix that closed it.
  9. Silence and unknown states. What does your dashboard show when the monitor itself cannot read a signal?
  10. Schema contracts. Show a write your platform rejected for schema drift.
  11. Audit and provenance. Trace one answer back to its sources and to who created them.
  12. Tenancy. What stops a tenant credential from addressing another tenant's data, structurally?
  13. Integration. Can an outside AI client work with the platform over a standard protocol, scoped to my tenant?
  14. Grounding guards. Where in the system, not the prompt, is 'verify before reporting success' enforced?

These are requests to make in any evaluation. The first steps of early access are laid out at https://octo.ad/start/; timing and commercial terms are not published; ask the team.