How to monitor an LLM API (OpenAI, Anthropic, Gemini) in Spot Downtime

A step-by-step guide to LLM API monitoring: use a separate key, pick the model, set the prompt and time limit, and keep the cost of checks to a fraction of a cent.

· 5 min read · By the Spot Downtime team

A provider's status page speaks for its whole platform, not for your key, your model or your quota. An LLM API monitor sends a short prompt to the exact model your product uses, with your own key, and checks that a real answer comes back in time. A revoked key, an empty balance, a retired model or slow replies all fail the check, just as they would fail your app.

A short prompt, a one-word answer: a real reply in time means your key, quota and model all work.
LLM API monitors are on every plan, including Free. Each model counts as one monitor.

Step 1: create a separate API key

In your provider's console, create a key just for monitoring, with a low spending limit. If it ever leaks, it can't run up a bill, and you can revoke it without touching production.

Step 2: create the monitor

  1. Open Monitors, click Add monitor and choose LLM API.
  2. Pick the Provider: OpenAI, Anthropic, Google Gemini, Mistral, Groq, DeepSeek, xAI, OpenRouter or Together AI. For Azure OpenAI, vLLM, LiteLLM, Ollama or any OpenAI-compatible server, choose Custom and enter its Base URL.
  3. Paste the API key. Once saved, only its last 4 characters are ever shown.
  4. Pick the Model from the list, or type its ID. Use the same model your product calls.
An LLM API monitor with the default prompt.

Step 3: set what counts as a good answer

FieldDefaultWhat to change
PromptReply with the single word OK.Usually nothing. A short prompt keeps checks cheap.
Reply must containOKNot case-sensitive. A blank or different reply counts as down.
Max reply tokens64Raise to 500 or more for reasoning models (see below).
Down if slower thanNo limitSet it if slow replies hurt your users, such as 5 seconds.

Reasoning models

They think before they answer, and the thinking counts toward the reply limit. With 64 tokens the reply can be cut off before any text, and the check fails with a message saying so. Give them 500 or more.

Step 4: choose how often

Pick Check every and click Add monitor. With the defaults each check costs a tiny fraction of a cent on most models. Every 5 minutes is about 8,640 requests a month, and the form shows the number for the interval you pick. A failure is retried once, not repeated from every region.

Step 5: read an alert

Alerts say what went wrong in plain words, for example:

  • Key rejected: the key was revoked or mistyped.
  • Rate limited or out of credits (HTTP 429): top up, or raise your limits.
  • Provider overloaded: often temporary. Check the provider's status too.
  • Reply missing your text: the alert shows what came back instead.

Pair it with a third-party service monitor on the same provider: one tells you about your key and model, the other about outages the provider declares.

LLM API latency testCompare OpenAI, Claude and Gemini response times now, free.

More in this series

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading