TechStar Asia - Tech News for Builders and Operators

Every AI code review tool has the same weakness

I've been building AI code review tools since 2024. Every one I've tried, and every one on the market, has the same structural flaw: it's a single AI voice making a single judgment with no cost to being wrong.

Even the newer "multi-agent" tools like GitHub Copilot's parallel review agents don't really solve this. They divide labor: one agent for security, one for linting, one for testing. But when two of those agents disagree about how serious an issue is, nothing resolves it. The reviewer's job just gets pushed onto the developer.

That's the gap I built ShiftLeft Society to fill, for the Qwen Cloud Global AI Hackathon 2026.

The idea: two AI reviewers who actually negotiate with each other when they disagree, with a cost structure that keeps the whole thing auditable.

The result: a live, deployed multi-agent tribunal that scored 95% on a 40-case benchmark, versus 82.5% for a single-agent baseline. A 12.5-point improvement, primarily by reducing false positives on safe code.

Here's how I built it and what I learned.

The core insight: the LLM proposes, deterministic code disposes

Most "multi-agent negotiation" tutorials I found online let the LLM decide everything, including the math. They ask the model to self-report a confidence number (0-100), then multiply it against something, and out pops a "verdict." That produces different outputs on every run. Unauditable. Uncalibrated. Basically vibes.

I inverted that:

The LLM only picks a categorical position: DEFEND, PARTIAL, or CONCEDE.
Deterministic Python computes the budget cost, the revised severity, and everything downstream.

Here's the actual cost table:

Position	Cost
DEFEND (dig in on your severity)	`gap_tiers × 30`
PARTIAL (compromise halfway)	15
CONCEDE (yield fully)	0

Each agent starts a negotiation with a 100-point budget. If Security wants to DEFEND against Performance and they're 2 severity tiers apart (say, Security = CRITICAL vs Performance = MEDIUM), DEFEND costs 60 points. If they're 3 tiers apart, DEFEND costs 90. Meaning: the stronger your disagreement, the more it costs you to hold your ground. That forces the LLM to actually think about whether it wants to spend the budget.

The LLM never touches a number. It just picks a category. The math is Python.

Three properties this gives you that pure-LLM agent demos rarely have:

Auditable. Every verdict has a transcript: who argued what, what position they took, what it cost them.
Reproducible. Same inputs → same output. Regardless of LLM temperature.
Defensible. The mediator's final call is derived from budget-weighted state, not from vibes.

Cross-PR credibility: the agents remember

The negotiation isn't stateless. Every time the tribunal produces a verdict, each agent's Round 2 judgment gets scored: did it defend correctly, or did it correctly defer to a peer flagging a more serious issue?

That outcome is:

Bayesian-smoothed against a 5-negotiation neutral prior (so a single early win/loss can't swing anything)
Capped at ±15 budget points (so no agent can ever fully dominate or be silenced)
Persisted across every PR the tribunal has ever reviewed

The next time that agent negotiates, on a completely different PR days later, it starts from 100 + track_record_adjustment, not a fresh 100 every time.

effective_budget = 100 + clamp(-15, +15, round((smoothed_win_rate - 0.5) * 30))
smoothed_win_rate = (wins + 2.5) / (total_negotiations + 5)

This turns the tribunal from a system that negotiates well once into a system that gets better at knowing which of its own voices to trust over time, without any human manually re-weighting anything. It shows up right in the PR comment:

"track record: 95% upheld over 42 past negotiations → budget +13"

The tech stack

Qwen-Max via Alibaba Cloud DashScope international endpoint (dashscope-intl.aliyuncs.com/compatible-mode/v1) powers all agent reasoning.
LangGraph orchestrates the agent society: parallel Round 1 (using Send API for concurrent fan-out), conditional Round 2 negotiation only when severity gap exists, then the mediator, then end. Not a linear chain, a stateful graph with conditional edges.
FastMCP (Model Context Protocol) exposes real security tools: scan_vulnerabilities, detect_secrets, check_yaml_pinning, analyze_complexity. If the MCP server is unreachable, everything falls back to local regex-based detection so the tribunal degrades gracefully.
FastAPI for the gateway: REST endpoints, Server-Sent Events for the dashboard, GitHub webhook receiver, SARIF and CycloneDX SBOM export.
SQLite on a Docker-persisted volume for analyses, dialogue history, and the credibility table.
Docker + Caddy for containerized deployment with automatic Let's Encrypt HTTPS.
Alibaba Cloud ECS in Singapore as the live substrate.

Building on Qwen Cloud: honest impressions

This is a hackathon post so I want to be honest about the developer experience, both the good and the friction.

What worked well:

The OpenAI-compatible endpoint (compatible-mode/v1) is a genuine developer experience win. My tribunal uses LangChain's ChatOpenAI wrapper unchanged, just pointed at Alibaba's Base URL:

llm = ChatOpenAI(
    model="qwen-max",
    api_key=os.getenv("QWEN_API_KEY"),
    base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
    temperature=0.1,
    max_tokens=4096,
)

That's it. No custom client, no bespoke JSON handling. Everything from LangGraph to structured output parsing just worked.

Qwen-Max's structured output is reliable enough for production. I use LangChain's .with_structured_output(Pydantic model) on almost every agent call, and Qwen-Max produces valid JSON on ~99% of calls. The rare failures are caught by a 4-layer fallback chain (structured output → raw parse → regex scrape → deterministic inference), which means the tribunal never crashes on a malformed model response.

Where I hit friction:

Model catalog navigation was harder than expected. Alibaba's catalog lists several Qwen models, but not all are accessible on every API tier. I tested qwen3.7-max (which is a reasoning model) and found it works but adds ~5x latency because of the reasoning tokens. For a system that needs to respond within GitHub's 10-second webhook timeout, that's disqualifying. So I use qwen-max (the stable current alias) which is fast enough and just as accurate for my use case.

Real production engineering decision, not a compromise: latency budget matters more than "newest model" for webhook-driven agents.

Measurable results

I ran a 40-case benchmark: 22 vulnerable, 18 safe code samples covering categories from SQL injection to unpinned dependencies to secure random tokens. Same model, same prompts for both systems. The only difference: single-agent verdict vs multi-agent tribunal.

System	Correct verdicts	Notes
Single-agent baseline	33 / 40 (82.5%)	Mostly false positives, flagged safe code as vulnerable
Multi-agent tribunal	38 / 40 (95.0%)	One miss on an insecure-random case

The interesting story is where the tribunal improved. On six cases, the single agent wrongly flagged safe code as dangerous:

A properly hashed password
A verified JWT with an explicit algorithm
TLS verification enabled with a timeout
Proper Python context managers
Sliced pagination on a list

On all six, the negotiation cleared the false positive. Security's Round 1 caution got tempered by Performance's correctly identifying "no actual issue here." That's the tribunal's biggest practical win: less noise, not just more accuracy.

Results committed to the repo as benchmark_results.json.

What I learned building this

1. Non-determinism in LLM-driven negotiation is a real production problem. Letting the model self-report confidence produces unreproducible verdicts. Constraining the LLM to categorical choices and computing consequences in code fixed this cleanly.

2. Blocking calls silently break async systems. Early builds had requests.get for the GitHub diff, unwrapped sqlite3.execute in a FastAPI endpoint, and asyncio.create_task without holding references. Every one of those is a latent bug that only shows under real load. Wrapping in asyncio.to_thread and holding tasks in a module-level set turned "works on my laptop once" into "survives real webhook traffic."

3. Error handling has to be visible, not just present. I built 4-layer fallbacks for the mediator, MCP tool calls, and structured output parsing. But that engineering is invisible unless you tell people. Documenting them in the README is how a judge sees the depth.

4. Persistence matters more than you think. My credibility trust data got wiped twice by container restarts before I put SQLite on a Docker volume. Non-persistent state is invisible until it disappears.

limitations

Benchmark is 40 cases, not thousands.
Tuned and tested on Python only.
Single-instance SQLite deployment, not multi-node Postgres.
Credibility metric rewards outcome-matching, not full outcome-grounded calibration.

What I'd actually build next

1. Ground the credibility signal in real merge outcomes.

Right now the credibility metric asks: "did this agent's severity match the final verdict?" That's a proxy. The real signal is what happened to the PR after the tribunal reviewed it. Did the developer merge it as-is, rewrite the flagged code, or ignore the review entirely? That's not a bigger benchmark. It's a fundamentally different feedback loop. The system stops guessing whether it was correct and starts learning from what humans actually did with its verdicts.

2. Multi-language through the MCP layer, not just prompting harder.

The obvious "extend to JavaScript" answer is to prompt the LLM more aggressively. That works okay. What's more interesting: add language-specific detectors (ESLint for JS, go vet for Go, cargo-audit for Rust) as MCP tools. The deterministic layer grows, the LLM guesses less, and the whole thing gets more auditable as it gets more general.

3. A third specialist agent to stress-test the negotiation mechanic.

With only two agents, negotiation is 1-v-1. Adding a Maintainability voice tests whether confidence-budget generalizes cleanly to N-way disputes, or whether it needs redesigning at higher agent counts.

4. Team-level credibility, not just global agent-level.

Different teams have different risk tolerances. A payments team should weight security's judgment more heavily. A hackathon team can tolerate more noise. Track credibility per-repo or per-team, and the system tunes itself to each team's actual priorities over time.