TTFT p95 per model — the latency your users actually feel — as a Prometheus histogram (latenzy_ttft_seconds).
End-to-end request duration and streaming output tokens/sec, so you can compare speed as well as first-response.
Outcome counts (ok / rate_limited / timeout / error) surface 429 pressure before your users do.
An endpoint label compares the same model over direct API vs Bedrock vs Vertex — a real routing and procurement decision.
Small requests on an interval measure each configured model. ≤16 tokens — cost is negligible.
A fail-closed Prometheus exporter: loopback by default, auth required on any routable bind.
Import dashboard 25642: a model-comparison view, p50/p95/p99 recording rules, and alerts.
Wrap real LLM calls to chart your own p95 under source="live" — one dashboard shows both.
No existing stack? A docker compose up bundle runs the prober, Prometheus, and Grafana pre-provisioned.