All features
LLM API monitoring

LLM API monitoring for the models your product depends on

A provider's status page speaks for their whole platform, not for your key, your model or your quota. An LLM API monitor sends a short prompt to the model you use, with your own key, and checks that a real answer comes back in time. When the key is revoked, the credits run out, the model is retired or replies get slow, you hear about it before your users do.

Test OpenAI, Claude and Gemini latency now, free

Your key, your model

Checks call the exact model your product uses, so a revoked key, an empty balance or a retired model fails the check like it fails your app.

Checks the answer

Require text in the reply, like OK, and set a time limit. A blank, cut-off or slow reply counts as down, not just an HTTP error.

Readable reasons

Alerts say what went wrong in plain words: key rejected, rate limited or out of credits, provider overloaded, or the reply that came back instead.

See it in Spot Downtime

The real app, with a demo team's monitors. Included on every plan, Free too.

An LLM API monitor for OpenAI with reply times
A model API checked with a tiny prompt, with how long each reply took.

What is LLM API monitoring?

When a product calls a language model, the model becomes a dependency with its own ways to fail. The key can be revoked or leaked and rotated, the account can run out of credits, the model can be retired, rate limits can tighten, and replies can slow to the point where users give up. Your site stays up through all of it while the feature people pay for is broken.

An LLM API monitor sends a short prompt to the model on a schedule, the same way your app does, and checks the answer. If the request fails, the reply is missing the text you expect, or it takes longer than your limit, an incident opens and your team is alerted.

Check the answer, not just the status code

A 200 response isn't always a working model. A reasoning model can spend its whole token budget thinking and return an empty reply, and a safety filter can block a prompt. By default the monitor asks the model to reply with OK and requires OK in the answer, so those count as failures too, with a reason that says what happened.

A time limit catches slowdowns before they become outages: set one a little above your normal response time, and a check that replies slower counts as down. The response time of every check is charted, so you can see a provider getting slower over days.

Keep the cost and the key under control

Every check is a real request billed to your key, so the monitor keeps them small: a one-line prompt and a 64-token reply cap by default, with an estimate of requests per month shown as you pick the interval. A failed check is retried once rather than repeated from every region.

Use a dedicated key with a low spending limit. Once saved, the key is never shown again, not even to your teammates, and it is only ever sent to the endpoint it was saved for.

How to set it up

  1. 1Click Add monitor and choose LLM API.
  2. 2Pick the provider and enter the model ID, or choose Custom for any OpenAI-compatible endpoint.
  3. 3Paste an API key. A separate key with a low spending limit is best.
  4. 4Optionally change the prompt, the text the reply must contain and the time limit.

The details

Providers
OpenAI, Anthropic, Google Gemini, Mistral, Groq, DeepSeek, xAI, OpenRouter, Together AI, and any OpenAI-compatible URL (Azure OpenAI, vLLM, LiteLLM, Ollama…)
Counts as down
An error status, no reply within 30 seconds, a reply missing your expected text, or one slower than your limit
Cost
Each check is one short request on your key; a 64-token reply cap by default. A failure is retried once, not repeated from every region
API key
Write-only: after saving, only its last 4 characters are shown, to you and your team
Plan
All plans, including Free. Each model counts as one monitor

Questions

How much will the checks cost me?
Very little with the defaults. The prompt asks for one word and replies are capped at 64 tokens, so a check costs a tiny fraction of a cent on most models. Every 5 minutes is about 8,640 requests a month; the form shows the number for the interval you pick.
Why not just watch the provider's status page?
Do both. Third-party monitoring tells you when the provider declares an outage. An LLM API monitor catches what their status page never will: your key, your quota, your model, and slowdowns that aren't bad enough for them to post about.
Does it work with reasoning models?
Yes, but they think before they answer, and the thinking counts toward the reply limit. Raise max reply tokens to 500 or more, or the reply can be cut off before any text and the check fails with a message saying so.
Can I monitor my own model server?
Yes, if it's reachable from the internet and speaks OpenAI's chat completions API, as vLLM, LiteLLM, Ollama and most gateways do. Choose Custom and enter its base URL; the API key is optional.

Set up LLM API monitoring in under a minute

Included on the Free plan. No card needed.

Create a free account