Your Confidence Signal Is Going Away: A Practical Field Guide

TMLS Insights | Week of June 18, 2026

By Graham Toppin (Graham is a TMLS Chair, and Co-founder and Analyst at Peerlabs.ai - a subscriber funded Intelligence firm, focused on primary research on emerging technology)

What’s happening

The tools we use to evaluate and customize closed models are quietly being removed. If you build against a vendor's API, you've been depending on instruments that the newest frontier models no longer offer.

This week, we're going to talk about what that means for how you manage your models.

Specifically, four material things have happened:

1. Token log probabilities (logprobs) are unavailable on OpenAI’s reasoning models, from o1 through GPT-5.5.
2. The logit-bias and logprobs combination was restricted in 2024, explicitly because the combination was shown to allow theft of internal model parameters.
3. Self-serve fine-tuning (SFT) is being removed. OpenAI is winding down self-serve access by January 2027, and Anthropic never offered a first-party path (only a narrow one via Bedrock).
4. Raw chain-of-thought (CoT) output is being hidden behind summaries on the latest models.

This essay takes up the first three: logprobs and customization. CoT is its own story, and we’ll tackle it separately.

BLUF (Bottom Line Up Front): What you should do

To sum up:

1. The signals you use to evaluate and customize models on the big closed APIs are being withdrawn, and unevenly.
2. If you depend on a cheap confidence read (logprobs) or on fine-tuning a hosted model, the latest, closed-weight, frontier models are the ones taking those away.
3. Our recommendation is to match each workload to a provider by what it actually needs, and to keep an open-weight option for the work that needs the instrument back.
4. In most cases, you should use a router or analogous API gateway to manage API independence between your products and operations, and the providers. Additionally, you should use a router when your production workloads may move across boundaries (providers, regions, etc.). They are also a strong architectural component to build availability and scalability.
5. When describing these customization and management techniques to upper management, avoid describing them as a correctness guarantee or something to eliminate hallucinations.
6. Build a reliable set of internal evals and guardrails specific to your domain to provide stronger, more reliable guarantees about system behaviour.

Introduction

When we build our AI-enabled products and operations against a vendor’s API, the API represents a few things:

• Our means of access to a specific model or capability
• The instruments we use to evaluate, customize and manage the model
• A means for measuring, copying and distilling the model

To be a little more specific, we’re talking about logprobs, fine-tuning and ways to understand and introspect on the internals of a model’s behaviour. We’ll discuss each of these in more detail in this article.

This has meant the same signals we use to make a model useful to us are the same signals allowing the model to “leak”. What we’ve seen in the last couple of years is the removal of these capabilities, for a few different reasons:

• Security and competitive protection - preventing models from being hacked, extracted, or distilled
• Safety - preventing misuse of the model (e.g. criminal activity)
• Cost - model inference is expensive (both for providers and for customers) and needs to be managed

This has meant how we take LLMs to production has to be re-imagined.

This is related to the thesis of model capabilities plateauing and capability becoming a commodity. If they are, then as a practitioner your focus should be on creating strategies to protect your business and thrive.

Let’s dig in.

What this means

• A logprob is the model’s own probability for a token it generated, the log of how much weight it placed on that choice.
• The logit-bias and logprobs combination is a pair of request parameters that, used together, let a caller probe the model’s output layer.
• Self-serve fine-tuning (SFT) is the hosted capability to train a base model on your own data.

Reading a class token’s probability and routing the low-confidence cases to a human or a larger model is four lines of code:

resp = client.chat.completions.create(model=MODEL, messages=msgs, logprobs=True, max_completion_tokens=1)
top = resp.choices[0].logprobs.content[0]                 # the chosen label token
surprise = -top.logprob                                   # nats; small = confident
action = “trust” if surprise < 1e-3 else “escalate to a human or larger model”

The catch isn’t the plumbing, it’s choosing what to gate on: on modern instruction-tuned models the chosen-token probability saturates near 1, so you gate on surprise, the magnitude of the logprob, not on the probability or the margin.

Perplexity needs a bit of explanation - intuitively, think of it this way: a lower perplexity means the text produced by the model seemed predictable. Higher perplexity means the opposite. For single-token classification, you should use the negative logprob directly, calibrate thresholds on held-out examples, and treat the log scale as more numerically useful than raw probability near saturation. For multi-token labels, normalize by token count or score the full label alternatives explicitly.

You can’t do this on a reasoning model that returns no probabilities, and there isn’t an equivalent, cheap way to measure confidence in its place.

Customization changes too: where before you could fine-tune a hosted model to specialize it, now you need alternatives. There are workarounds, and each one costs something. Let’s take it step by step.

What logprobs are good for, and where they break

• Given the context we provide to a model, logprobs give us an idea of the likelihood of a given token or tokens.
• Logprobs are not measures of factual correctness; they are closer to a measure of how confident a model is in a particular answer. This matters, because a model can be maximally confident and maximally wrong at the same time.
• OpenAI removed logprobs from its frontier lineup as of GPT-5.5 and is expected to cease support as its older models are removed; Anthropic never offered them; Google is the only major provider still maintaining support; local and open-weight models and inference servers still provide them.
• Logprobs can be a great cheap signal for constrained classification, but they degrade sharply for open-ended generation and RAG grounding.
• We don’t have a single good, general method for evaluating uncertainty across all dimensions; production systems increasingly need composite approaches. The loss of logprobs leaves a void that is not easily filled.
• The most common organizational anti-pattern we’ve seen is treating logprobs as a grounding mechanism rather than a triage signal. The misinterpretation is subtle: good practitioners understand it intuitively, but often struggle to communicate it to stakeholders.

Evaluating without the cheap signal

When the cheap read is gone, what you reach for depends on the question you’re actually asking. Decide the question first, then pick the method.

• Classification and triage. Logprobs still work where you can get them: standard tiers and open weights. You should gate on surprise (the magnitude of the chosen token’s logprob) rather than on the probability or the margin. On modern instruction-tuned models those saturate near 1 and stop discriminating, while the magnitude of the logprob still spans orders of magnitude. You should use logprob-derived scores only after task-specific calibration.
• Open-ended generation and RAG. Don’t trust logprobs here. A confident answer is not a grounded one. Check grounding at the system level: tie the answer to a source and verify the tie, rather than reading it off the model’s confidence.
• Meaning-level uncertainty (is the model unsure about the answer, or only about the wording?). Semantic entropy, by sampling several answers and clustering them by meaning. It works on any model but costs you several calls. The cheap one-pass version reads the model’s internals, so it is an open-weight move.
• A formal guarantee. Conformal prediction gives a coverage guarantee on any model, at the cost of larger, less precise answer sets.

The table below is an analyst synthesis, not a benchmark; reasonable readers could shift a rating, and the relative ordering is more defensible than any single cell.

Customization: when the hosted lever closes

Fine-tuning a hosted model is winding down on closed frontier models. The need for customization hasn’t disappeared. It has moved, and the order you try things in matters. Start with the cheapest, most reversible lever and stop as soon as one works.

Our top recommendations are to:

1. Use open-weight models whenever you need customization / SFT
2. Use a router or an API gateway not just for architecture flexibility, but also to limit vendor lock-in

Our more general advice is to consider:

1. Prompting and few-shot. Reversible and immediate. Exhaust this first.
2. Retrieval (RAG) when you need knowledge. Fine-tuning teaches form, not facts, so knowledge belongs in retrieval, where it also stays current.
3. Fine-tune for form. Use it for consistent format, structure, or tone, or to specialize a smaller, cheaper model on a narrow, stable task.
4. Distill into an open-weight model you control. When you need a frontier model’s behaviour at lower cost, lower latency, or with full control, distill it into a model you run. Mind the terms of service: distilling a competitor’s model through its API likely breaches them.

Recommendations

We’ve covered a lot of ground, let’s sum up, topic by topic.

Confidence and evaluation:

• Decide which question you are answering (confidence, grounding, or meaning-uncertainty) before you pick a method. Don’t overload one number.
• If you need a readable confidence signal, design for it now: a standard tier that still returns logprobs, Gemini’s top-k, or open weights. Gate on surprise (the magnitude of the logprob), not on probability or margin, which saturate on modern instruction-tuned models.
• More specifically, when working with single-token labels, use negative logprobs. For multi-token labels, normalize or else score each full label alternative.
• Never present a logprob to stakeholders as a grounding or correctness guarantee. Check grounding at the system level.

Customization:

• Exhaust prompting and retrieval before you train anything.
• Fine-tune for form; use retrieval for knowledge.
• For cost, latency, or control, distill into an open-weight model you run, and watch the terms-of-service line.

Architecture:

• Put an abstraction layer or router between your application and any one provider, so the choice stays reversible as postures keep shifting.
• Keep an open-weight escape hatch for any workload that must measure, adapt, or inspect the model.

Pick the provider per workload, design the confidence and customization paths in on purpose, and keep the open-weight door open for the work that needs the instruments back.