AI you can run, audit, and explain.

Most software vendors hand over a working build and disappear. That’s fine for a marketing site. It’s a disaster for the dispatch board your freight company runs on, the inspection platform 200 organizations submit AMPP reports through, or the intake system that drives signed cases at a law firm. The day-two question — “is this thing still working, who’s watching, and what happens when it isn’t?” — is where most operational software actually fails.

AI compounds the problem. An LLM that drafts an insurance demand letter, an agent that posts loads to a board, a vision model classifying coating defects — each adds a new failure surface. Hallucinations, prompt injection, model drift, vendor outages, silent degradation. The frontier labs ship models; nobody ships the operating substrate that makes those models safe in your business.

This page is that substrate. The practices below aren’t aspirational — they’re the default for every Sytepoint engagement, instrumented from day one, visible on our own production. The marketing site you’re reading right now runs on the same backbone.

Human-in-the-loop on every destructive or irreversible path. Agents draft, propose, summarize, route — they don’t commit. Sending a payment, deleting a record, posting an external message, accepting a contract: a human approves the agent’s proposal before it fires. The pattern adds milliseconds to most flows and zero risk of an autonomous LLM doing something you can’t undo.

Eval harnesses before production, regression suites after. Every prompt-driven workflow ships with an eval set — ~30–100 representative inputs with expected outputs (or expected shape, where free-form). The suite runs on every change. New prompt? Re-run evals. Model version bump? Re-run evals. Regression in pass rate gates the deploy.

Prompt-injection scanning + output validation. Inbound user content runs through an injection detector before it reaches an LLM with privileged context. Outbound LLM responses are validated against the expected schema before any tool call or downstream action fires. Belt and suspenders; both are cheap.

Vendor diversity, model versioning, kill switches. Critical AI workflows are written against an abstraction, not a vendor. Model versions are pinned per workflow and bumped explicitly, not silently. Feature flags can disable any AI path in production in under 30 seconds if the vendor degrades or pricing shifts.

No direct production writes. Even an emergency hotfix flows through the full pipeline below — with an expedited canary window, not a bypass. Audit-logged at every gate, reversible at every step.

Blue-green & canary deploys on every production release. The canary catches what staging missed; promotion is gated by error rate + latency stability on the canary slice. Feature flags decouple deploy from release — code reaches production before users see it, which gives us a kill switch independent of the deploy system itself.

Rollback drillsrun quarterly in staging — actual rollbacks, actual recovery, actual stopwatch. The marketing site you’re reading rolls back in under 90 seconds; the larger platforms in under five minutes. A documented number means it’s a number we’ve actually measured.

Backups + RPO/RTO commitments. Postgres point-in-time recovery to the last 7 days, full snapshots daily, snapshots tested monthly by restoring to a fresh instance and running the smoke suite. RPO 5 minutes, RTO 30 minutes for platform workloads — calibrated to the actual business cost of data loss for each workflow.

Change management. Every change has a PR, a required reviewer, an automated check suite, and a paper trail. Direct production access is provisioned per-engagement, not per-employee, and audited quarterly. Secret rotation runs automatically; revocation on engineer departure is same-day.

Severity classification. Sev-1: production down or data at risk. Sev-2: significant degradation. Sev-3: minor issue, no user impact. The classification drives paging, response SLA, and post-mortem depth.

Paging & response. Sev-1 pages the on-call engineer within 5 minutes via the monitoring stack. Acknowledgment SLA: 15 minutes. Communication SLA: status page update within 30 minutes, then every 30 minutes until resolved. The client is in the loop from the first acknowledgment, not after the fix.

Blameless post-mortems.Sev-1 and Sev-2 incidents get a written post-mortem within 5 business days. Root cause, contributing factors, timeline, what fixed it, what we’re changing so it doesn’t recur. Shared with the affected client. Action items tracked to completion — the post-mortem isn’t closed until they ship.

Durability over speed.The first fix is often a quick script; the second fix is the system change that makes the same incident class impossible. We track recurrence rate as a KPI — same root cause twice is the indicator we’re fixing symptoms instead of causes.