What is LLM API monitoring?
When a product calls a language model, the model becomes a dependency with its own ways to fail. The key can be revoked or leaked and rotated, the account can run out of credits, the model can be retired, rate limits can tighten, and replies can slow to the point where users give up. Your site stays up through all of it while the feature people pay for is broken.
An LLM API monitor sends a short prompt to the model on a schedule, the same way your app does, and checks the answer. If the request fails, the reply is missing the text you expect, or it takes longer than your limit, an incident opens and your team is alerted.
Check the answer, not just the status code
A 200 response isn't always a working model. A reasoning model can spend its whole token budget thinking and return an empty reply, and a safety filter can block a prompt. By default the monitor asks the model to reply with OK and requires OK in the answer, so those count as failures too, with a reason that says what happened.
A time limit catches slowdowns before they become outages: set one a little above your normal response time, and a check that replies slower counts as down. The response time of every check is charted, so you can see a provider getting slower over days.
Keep the cost and the key under control
Every check is a real request billed to your key, so the monitor keeps them small: a one-line prompt and a 64-token reply cap by default, with an estimate of requests per month shown as you pick the interval. A failed check is retried once rather than repeated from every region.
Use a dedicated key with a low spending limit. Once saved, the key is never shown again, not even to your teammates, and it is only ever sent to the endpoint it was saved for.

