[REQUEST] Cost Governance for Multi-Agent Workflows — How Are You Tracking LLM Spend Across Agents? #3700
Description
Hey Sim community 👋
Feature idea / integration question: Cost Governance for Multi-Agent Workflows
A recent paper — AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem — frames this well. It argues that as we move toward agent-native computing, the "Agent Kernel" needs to allocate token budgets and API rate limits across concurrent agent threads, treating LLM resource scheduling as a first-class OS-level concern. The paper identifies the need but doesn't propose an implementation.
This got me thinking about Sim specifically. When you're building workflows with multiple LLM-powered blocks — say an agent block calling GPT-5, feeding into another agent block using Claude, with a function block in between — a few questions come up:
- Per-block cost attribution — Which block in a workflow is the most expensive? If a workflow costs $2.40 per run, is 80% of that coming from one agent block that could use a cheaper model?
- Budget enforcement — Can you set a per-workflow or per-team spending cap so a misconfigured loop doesn't burn through your API credits overnight?
- Caching for repeated calls — In workflows triggered frequently via webhook, identical LLM calls with the same input waste money. Semantic caching at the gateway level can cut this significantly.
- Anomaly detection — A workflow that normally costs $0.50/run suddenly costs $5.00. How do you catch that before it scales?
I've been building AgentCost — an open-source cost governance platform — to solve exactly these problems. It provides per-agent cost attribution, budget enforcement with approval workflows, gateway-level caching, anomaly detection, and a pricing database covering 2,600+ models. It's designed to sit alongside orchestration platforms (not replace them) as the cost intelligence layer.
A few integration patterns I think could work well with Sim:
- Webhook receiver — Sim workflows could POST execution metadata (model used, tokens consumed, workflow ID) to an AgentCost endpoint after each run, building cost analytics over time without any changes to workflow logic itself.
- AgentCost as a Sim tool block — A custom "Cost Check" block that queries AgentCost's pre-call estimation API before an expensive LLM call. If estimated cost exceeds a threshold, the workflow routes to a cheaper model or pauses for approval.
- Cost center mapping — AgentCost's cost center concept maps naturally to Sim's workspace structure. Each workspace or workflow ID becomes a cost center with its own budget and alerts.
I'd love to hear from the community:
- How are you currently tracking costs across your Sim workflows?
- Would per-block cost breakdowns in the execution logs be useful?
- Any interest in exploring an integration proof-of-concept?
Happy to dig into any of these patterns. And if the Sim team has thoughts on where cost observability fits in the roadmap, I'd be really curious to hear.