This Week in AI Production: What You Can Do Right Now

This Week in AI Production: What You Can Do Right Now

TMLS Newsletter article May 4, 2026

TMLS Insights | Week of May 4, 2026 By Graham Toppin (Graham is a Co-founder and Analyst at Peerlabs.ai - a subscriber funded Intelligence firm, focused on primary research on emerging technology)

This is the first in a series of practitioner-focused posts. No news roundups. No vendor hype. We want to give you concrete advice you can test this week, grounded in what we’re hearing from practitioners and seeing in our own work.

This week we’re going to be covering how to run open-weight models at 1/40th the cost; locking down your AI stack before the next supply chain attack; and managing context rot before it manages you. It’s going to be Operational AI as the focus though there will be cross over to folks directly offering AI in their Product.

1. A Quick Guide to your local/open-weight model setup: a concrete starting point

Six months ago, a lot of us were asking: “should I run local models?” The question has shifted. It’s no longer about ideology or capability parity with frontier models. It’s about operational resilience, sovereign data, and control. What happens when your primary provider rate-limits you, changes its pricing, or goes down during a deadline. In one of my recent deep dives with a technical team, their lead said “Remember how ‘Github is down’ used to be the reason for no work getting done? Now it’s ‘Claude is down’.”

Folks talk about Claude 4.5 Opus as an inflection point. There has been a similar inflection point for Open Weight, Open Source models.

The pricing picture is fluid, but here’s the pricing reality as of this week:

Frontier (proprietary)

Open-weight heavyweight (frontier-competitive)

Open-weight lightweight (cost-optimized, local-capable)

(sourced from each provider’s publicly listed pricing pages and OpenRouter, verified May 4, 2026. DeepSeek V4-Pro promo surfaced in an earlier version of this article, was to expire May 5, but has been extended to May 31. Prices change frequently; this represents the latest as of this writing)

The spread tells the story. At the lightweight end, Qwen3 Coder at $0.07/$0.07 is 40x cheaper than Sonnet 4.6 on input and 214x on output. The heavyweight open-weight models sit in between: DeepSeek V4-Pro at $0.44/$0.87 (promo) delivers frontier-competitive reasoning at roughly 1/7th the cost of GPT-5.5. MiniMax M2.7 at $0.30/$1.20 scores 94% of GLM-5.1’s SWE-Bench Pro at a fraction of the cost. In March and April, I started paying closer attention to open-weight models after folks in my network told me they were saving 2-3 orders of magnitude relative to the frontier labs on specific workflows. We’re seeing similar results in our own pipeline at Peerlabs, where we use Novita.ai as a fallback during Anthropic rate throttling.

Caveat emptor:

A stack worth trying this week:

This is a recipe I’ve used with great success:

For coding and implementation — Novita.ai (hosted open-weight models, OpenAI-compatible API, no rate limits, pay-as-you-go) + one of:

For planning and reasoning — use the heavyweights via their native APIs:

We tend to use bigger models for planning and reasoning. A winning combination for example, is using a frontier model, or one of the larger open weight models for partnering on planning, creating PROJECT.md, HANDOFF.md or your context du jour, and having cheaper agents implement the design.

2. Your AI stack’s attack surface: what to lock down now

Two incidents from the last five weeks illustrate the same point: the security risk in your AI system is not (just) the model. It’s everything around it. The sophistication of these attacks, and the orthogonal nature of them - they didn’t attack your PROMPTs - signals a sophistication beyond the more tangible Day Zero bugs we’ve seen with the Claude Mythos hype-cycle.

Incident 1: Trivy → LiteLLM supply chain cascade (March 19-24)

A threat actor (TeamPCP) compromised Trivy, an open-source vulnerability scanner, by rewriting Git tags in its GitHub Action repository. Multiple security firms (Snyk, Wiz, Kaspersky) have reconstructed a plausible chain linking this to the subsequent compromise of LiteLLM: the compromised Trivy action exfiltrated PyPI publishing credentials from LiteLLM’s CI/CD runner, and malicious LiteLLM versions were published directly to PyPI containing a multi-stage credential stealer.

LiteLLM’s own disclosure hedges: ”we believe this incident may be linked”. The connection should be treated as strongly indicated but not definitively confirmed.

LiteLLM is present in 36% of cloud environments as a dependency of CrewAI, DSPy, Browser-Use, Mem0, Instructor, and Guardrails. A single compromised installation exposes API credentials for every model provider it connects to.

Incident 2: Cursor agent deletes production database (April 25)

A Cursor agent running Claude Opus 4.6 deleted a production database and all volume-level backups via a single Railway API call. The agent later produced a written explanation of the safety rules it had violated. The model didn’t hallucinate or jailbreak. It executed a legitimate API call through a legitimate tool with legitimate credentials that had no scope restrictions.

What you can do this week:

Dependency management (30 minutes):

Agent credential scoping (1 hour):

Broader assessment (ongoing): This doesn’t mean you should ignore prompt injections:

3. Context rot is real. Here’s how practitioners are managing it.

Meta’s Justin Jeffress gave the failure mode a name at All Things AI (April 4, sourced via The Register, which we’re going to treat as conference colour, not survey data): “context rot.”

As agent interactions accumulate across a session, more context competes for the model’s attention window and output quality degrades. You’ve probably seen this: the agent is sharp for the first 20 minutes, then starts contradicting itself, forgetting constraints, or repeating work. This happens in interactive sessions, and in autonomous workflows.

This is not a model problem you can solve by upgrading to a bigger context window. It’s an information management problem. Research is formalizing it (MIT’s Recursive Language Models, LinkedIn’s Cognitive Memory Agent, ICLR 2026’s MemAgents workshop), but the practical solutions available today are simpler.

What you can do now:

Externalize state between sessions. Don’t rely on the context window to be your agent’s memory. Have agents emit structured artifacts -- markdown files, JSON state objects, HTML reports -- at checkpoints. These become the input for the next session rather than the entire conversation history.

For operational AI: A concrete pattern from Jerry Liu (LlamaIndex): agents write .md and .html artifacts to preserve context. Obsidian serves as a local viewer and search interface across the accumulated artifacts. This separates “what the agent knows” from “what’s in the current context window.”

For product AI: Create formalized, versioned repositories your agents can persist, share and modify. Expressly define workflows for these tasks, with Pre- and Post- commit hooks to manage the flow. This is the agent taxonomy / ontology allowing you to capture the important contexts for your agents.

Compress, don’t concatenate. When you need to carry context forward, summarize rather than append. If your agent has been working for 30 minutes and you’re about to hit a new subtask, ask it to produce a structured summary of decisions made, constraints discovered, and current state - then start a fresh session with that summary as input. This is manual today; some harnesses (e.g. Claude Code’s compaction, Opencode, and others) are starting to automate it. Harnesses like pi allow you to roll your own, and have an extensive plugin community.

Context is the aspirational moat for Frontier Labs. It’s a concrete moat for practitioners. Frontier model providers, like Anthropic, have fairly opaque and changeable contexts which will not be externalized to you without explicitly planning for them. This is a form of vendor lock in you should be aware of. Your entire project context is in ~/.claude, and the structure of the Claude Code context isn’t an agreed upon format or API for you. Providers like Opencode have transparent logging and context sharing; but their codebase can be volatile. Pi allows you to create your own context - logging, transparency, etc. - and provides a basis for not just operational control, but for production workflows.

Monitor context usage. LangChain shipped a Claude Code → LangSmith tracing plugin logging subagents, tool calls, compaction events, and token usage. If you’re running agents at any scale, you need observability into how context is being consumed. Without it, you’re debugging quality degradation blind. There are a lot of (better) alternatives to LangChain’s LangSmith:

Scope sessions tightly. The practitioners reporting the best results with parallel agents (2-4 concurrent sessions seems to be the practical ceiling for most people) are running each agent on a tightly scoped subtask, not on an open-ended mandate. “Implement the authentication module per this spec” works. “Build the backend” doesn’t.

What’s coming (but isn’t ready yet):

Google’s Gemini Enterprise Agent Platform (April 22) includes Agent Gateway, Agent Observability, and Agent Identity as governance primitives -- the first major vendor attempt to address context and state management top-down. Databricks published “Memory Scaling for AI Agents” identifying retrieval quality, not storage, as the main bottleneck. Mem0’s “State of AI Agent Memory 2026” report describes memory as a “first-class architectural component” with its own benchmark suite. Whether any of these mature into production-ready tooling this quarter is an open question.

For now, the advice is simple: treat context as a resource you actively manage, not a bucket you fill until it overflows.

What we’re working on next

Next week we plan to cover: agents as a distributed systems problem (checkpoint/resume, retry depth, cold start latency), what the Microsoft-OpenAI deal means for your vendor strategy, and why eval culture is still the hardest problem in production AI.

If you’re running open-weight models in production, or if you’ve built internal tooling around agent context management, we’d like to hear from you. We’re collecting practitioner accounts (attributed or anonymous, your choice) for a more systematic write-up. Reach out at info@torontomachinelearning.com

TMLS Insights is produced by the TMLS Steering Committee and Peerlabs. We produce practitioner-focused analysis for the Toronto Machine Learning Society and MLOps World communities. Aspects of our research pipeline use AI; all claims are human-reviewed and sourced.