Watchdog auto-heal: 4 models, zero user impact

What happened

On 27 July 2026, several models on the Plugsky API lost their primary upstream when NVIDIA NIM retired the google/gemma-3n-e2b-it model. Users saw no interruption: the automated watchdog detected the failures, selected healthy same-profile peers (same context window, same tier), and swapped the upstreams automatically. The affected models — including plugsky-vision-fast, plugsky-qwen-vl, and plugsky-gemma-4 — now run on meta/llama-3.2-11b-vision-instruct.

How the watchdog works

1. Every 5 minutes, each chat model receives a real 80-token probe.
2. Three consecutive failures mark a model broken.
3. The watchdog finds a healthy model with the same context window and tier.
4. The broken model's upstream is swapped to that peer's — no manual intervention.
5. Requests also cascade at runtime: if a model's upstream fails mid-request, the call routes to a healthy upstream and still returns.

Proof it works

In a controlled test we pointed a model at a nonexistent upstream. The API still returned 200 with content in under a second, tagged with the fallback model. A 10-message conversation survived the failover intact. See the latency report for the full data.

Why this matters

AI outages cost businesses billions; single-provider lock-in turns an upstream change into an outage. Automatic peer-healing is what makes an independent AI cloud operationally boring — in the good way.