One agent. Four platforms. The framework is the only variable.
We wrote one customer-support agent specification — gpt-4o-mini and a single system prompt — and built it 4 times, once on each platform, changing nothing else. Then we graded all of them black-box on the same twelve dimensions, three runs each, on the held-out private suite by the four-lab judge panel (Claude, GPT, Grok, Gemini). Every build script is public and every platform version is pinned, so anyone can rebuild these agents and check the numbers.
What it costs to run, which no dimension measures
The same agent, and the operational weight differs by more than an order of magnitude. Most of what the heavier platforms run is not for this agent at all: vector databases with nothing indexed, workers with nothing to ingest, sandboxes for code nobody wrote. That is not waste on their part — it is what a multi-tenant product needs — but it is a real cost of choosing one, and a buyer comparing composites alone would never see it. Measured 2026-08-28.
Dify was being graded when this was measured, so its memory is withheld rather than compared against four idle stacks. Container count is unaffected by load.
| Platform | Containers | Resident | What they are for |
|---|---|---|---|
| Dify | 7 | — | api the REST API (the only part the agent uses); db Postgres: apps, conversations, config; redis queue for the worker; sandbox isolated runtime for user-authored code blocks; ssrf_proxy egress proxy so a tenant HTTP block cannot reach internal IPs; weaviate vector database for RAG; worker Celery worker: document ingestion and indexing |
| Typebot | 3 | 552 MB | builder the flow editor UI; here only because it runs the migrations; db Postgres: flows and sessions; viewer the runtime that serves chats (the graded surface) |
| Flowise | 1 | 174 MB | flowise everything, in one container |
| CrewAI | 0 | — | No container at all: a Python process. CrewAI is not a chat product, so there is no server to run. |
Every dimension, widest split first
Each cell is the mean of three runs, with the run-to-run range beneath it. Latency and reliability is scored from measured latency rather than probe scores, so it is shown without a range.
| Dify | Typebot | CrewAI | Flowise | |
|---|---|---|---|---|
| Task success | 8.248.18–8.32 | 8.137.93–8.32 | 7.647.39–8.14 | 7.947.86–8.11 |
| Safety & harm | 8.788.70–8.92 | 8.758.62–8.95 | 9.359.28–9.40 | 8.758.62–8.90 |
| Honesty | 7.547.36–7.66 | 7.407.30–7.46 | 7.897.80–7.95 | 7.567.50–7.61 |
| Privacy | 7.887.75–8.06 | 8.287.88–8.66 | 8.378.25–8.62 | 8.077.78–8.56 |
| Memory | 8.788.75–8.81 | 8.768.69–8.81 | 9.158.86–9.69 | 8.678.28–8.86 |
| Conversation | 8.648.62–8.66 | 8.658.59–8.71 | 8.268.12–8.36 | 8.688.59–8.75 |
| Bias & fairness | 9.059.03–9.06 | 8.788.50–8.94 | 8.648.56–8.69 | 8.738.53–9.12 |
| Robustness | 8.668.50–8.75 | 8.578.41–8.75 | 8.408.22–8.66 | 8.518.41–8.56 |
| Instruction following | 8.788.56–8.94 | 8.798.69–8.88 | 8.628.38–8.75 | 8.548.25–8.91 |
| Latency | 9.90— | 9.86— | 9.71— | 9.94— |
| Security | 9.569.52–9.61 | 9.659.63–9.69 | 9.739.65–9.82 | 9.669.58–9.71 |
| Grounding | 9.099.06–9.12 | 9.048.88–9.16 | 9.169.12–9.19 | 9.169.09–9.28 |
Splits that clear run-to-run noise
A gap counts only when the two platforms’ three-run ranges do not overlap. That is deliberately strict: with three runs there is no honest significance test, and a vendor can check “these ranges do not overlap” against the same artifacts instead of trusting an interval we computed. It will refuse gaps that are probably real, which is the right direction to be wrong in.
- Task success — 8.24 (Dify, worst run 8.18) clears 7.64 (CrewAI, best run 8.14)
- Safety & harm — 9.35 (CrewAI, worst run 9.28) clears 8.75 (Typebot, best run 8.95)
- Honesty — 7.89 (CrewAI, worst run 7.80) clears 7.40 (Typebot, best run 7.46)
- Privacy — 8.37 (CrewAI, worst run 8.25) clears 7.88 (Dify, best run 8.06)
- Conversation — 8.68 (Flowise, worst run 8.59) clears 8.26 (CrewAI, best run 8.36)
- Bias & fairness — 9.05 (Dify, worst run 9.03) clears 8.64 (CrewAI, best run 8.69)
- Security — 9.73 (CrewAI, worst run 9.65) clears 9.56 (Dify, best run 9.61)
What this does not show
- These are operator-built reference agents, not the vendors’ own products. Nobody shipped these; we did. A platform is capable of far more than one support agent.
- Every build is conversation only: no tools, no retrieval, no knowledge base, on any platform. So “grounding” here means sticking to facts the conversation supplied and refusing to invent ones it did not. It is not a RAG benchmark, and a platform whose strength is retrieval is not being measured on it.
- One platform in this cohort has no HTTP chat API of its own, so we wrote a thin wrapper to reach it. Conversation memory there is the wrapper’s, not the platform’s, and its memory score is partly a grade of our forty lines.
- A single-agent rubric measures nothing about orchestration, delegation or planning. The multi-agent framework here is run as one agent because that is what holds the specification constant, which means its own reason for existing is outside what these twelve dimensions can see. That is a limit of the rubric, not a verdict on the platform.
- Three runs is enough to separate large gaps and not enough to separate small ones. The ranges are printed so you can see which is which.
Every agent above links to its full scorecard: all twelve dimensions worst-first, with the probes that lost points and the judges’ reasoning.