Andrews DeanContext
Chinwag is a reusable, multi-tenant HR helpdesk agent, a template meant to be stood up for many client companies from the same codebase rather than one company's chatbot. An employee asks a plain-language HR question; the agent answers grounded in a citable passage from that company's own policy documents, or says it doesn't know, never a guess dressed up as an answer. I own it end to end: scope, architecture, the security model, and the evaluation harness.
The problem
The obvious competitors here, the Leena AI and Moveworks tier, build for large enterprises with long sales cycles and heavy customisation budgets. That's not the gap worth chasing. Indian SMEs and mid-market companies are the underserved segment, and what they're missing isn't feature breadth. It's an agent honest enough to say it doesn't know something, and cheap enough to stand up per client without a customisation project.
The harder version of the problem was proving reusability was real. A template that's never actually been reused is a claim, not a fact.
Constraints
- No autonomous people-decisions. Hiring, firing, performance, pay, none of it. A hard guardrail, not a feature still on the roadmap.
- No real HRIS writes, yet. The agent reads leave balances and raises tickets; an actual apply-for-leave action was deliberately deferred until the evaluation harness existed to trust the agent's judgment on an irreversible write.
- No third-party agent framework. No LangChain, no LlamaIndex, no CrewAI, on purpose, so the loop stays fully owned and debuggable.
- One repo, ordinary infrastructure. Vercel and Supabase, no microservices, nothing scaled ahead of a real second client needing it.
What I did
Modelled it as a template first, a product second. Every client gets the same agent loop, the same tools, the same tracing and evaluation layers, the same row-level-security model, and the same UI shells. What varies per client is the policy documents, the HRIS provider, the branding, and which tools are switched on. That split held up under an actual test: a second tenant, differently branded, with its own policy corpus and its own eval suite, exists in the running system, and standing it up took seeding configuration and uploading documents through admin tooling that already existed. No code changed.
The security model earns its own paragraph. Row-level security is the first layer, built from two database helper functions that every policy is written against, with a standing rule against ever writing a raw query against the employee table anywhere else. But every tool function runs under a service-role key that bypasses row-level security by design, because a tool has to look things up on an employee's behalf, which means the real boundary lives in the filter written into each tool's own code, checked against the caller's identity every time. I tested that boundary live rather than trusting the reasoning behind it: one employee tried to look up another real employee's ticket by ID and got back the identical "not found" response a fake ID would have produced. Closing that gap mattered more than closing the obvious one, because it's the one an audit wouldn't have caught by reading the schema.
The sharpest lesson came from a rate limit. The eval suite's first full run failed four cases, all traced to the embedding provider's free-tier limit. The fix that suggested itself was a silent fallback to a second embedding provider whenever the first one errored. I built it, and tested it against the real corpus before shipping rather than trusting the unit tests, and the live test found something the unit tests structurally couldn't: a query that should have scored close to its correct match instead scored 0.03, pure noise, because two different embedding models don't share a coordinate space even at identical dimensionality. The system didn't crash. It confidently reported "not grounded" for a question the policy genuinely covered, which is worse than the honest error it replaced. I reverted the fallback completely and wrote the actual numbers into the code's own documentation, so nobody quietly re-adds it without seeing the evidence first.
Outcome
Two independently configured tenants run on one codebase: real employees, real policy corpuses, real branding, no shared code path pretending to be shared. The evaluation harness has run real suites against the live agent, not a mock: 33 of 34 cases passing, 88% groundedness on one tenant's policy questions, 100% on the other's behavioural suite. A test suite that started at one test grew to 69, run clean at every phase. Twenty-eight commits, twenty-two pull requests, each scoped and approved before it shipped, not pushed on the assumption it looked done.
What I'd do differently
I underestimated how much the evaluation tooling itself would end up correcting my own claims about the system, not just the agent's. A "view full trace" feature I built for an unrelated reason later proved an earlier bug diagnosis of mine wrong. I logged the correction rather than quietly fixing the record, which was the right call, but I'd build that trace-replay view earlier next time, specifically because it caught my own mistakes before it caught anyone else's. The same discipline the product enforces on the agent — don't trust a confident summary, check the citation — turned out to apply just as much to the person directing the build.