---
name: plugsky-models
slug: plugsky--plugsky-models
description: "Use when asked about Plugsky AI models, which model to use, model capabilities, speed, coding performance, outage fallback, or chain reliability. Includes the canonical staged outage chain drill (main to backup to outage fallback 1/2/3/4), user-flow testing, and user-log error triage. Reference for all Plugsky models via api.plugsky.com/v1."
collection: Plugsky
category: Platform Ops
license: Plugsky
source: https://plugsky.com/skills
install: "Download the Plugsky skills pack (see below)"
---

# plugsky-models

Use when asked about Plugsky AI models, which model to use, model capabilities, speed, coding performance, outage fallback, or chain reliability. Includes the canonical staged outage chain drill (main to backup to outage fallback 1/2/3/4), user-flow testing, and user-log error triage. Reference for all Plugsky models via api.plugsky.com/v1.

## Install

```bash
Download the Plugsky skills pack (see below)
```

## Skill

# Plugsky AI Models — Capability & Speed Reference

## Staged outage chain drill — canonical user-flow test (verified 2026-10-05)

Run this whenever the owner asks "test the chain / shut down X / does failover work / check user errors".
It proves every link **in order** as a real API user, then triages user logs.

### Stage map (each stage = one unique request; expected served lane)

| Stage | Simulate | Expected served |
|---|---|---|
| S0 | normal | NVIDIA primary (`nvidia/nemotron-3-*`) |
| S1 | stop primary | NVIDIA backup / same-class peer (e.g. micro → `openai/gpt-oss-20b`) |
| S2 | stop all NVIDIA + stop Go | outage fallback #1 = **Go** free first (`longcat-2.5-preview-free` / `space-bunny-free`) |
| S3 | + stop Zen | outage fallback #2 = **OpenRouter** free first (`…:free` lane) |
| S4 | + stop OpenRouter | outage fallback #3 = **Groq** (`openai/gpt-oss-120b`) |
| S5 | + stop Groq | clean retryable **503** (OpenAI error shape, no 500/hang) |
| S6 | restore all | NVIDIA primary again + embeddings 200 |

### Mechanics (exact, production-safe)

1. **QA user**: `bash /tmp/prep_qa.sh` (qa.audit → admin+enterprise) → login → temp API key.
2. **Unique body per stage** (`"Say exactly OK [s3]"`) — primary responses cache 5 min; **rescue responses are never cached**, so without unique bodies later stages replay stage 0.
3. **Stop the primary only** (S1): SQL upsert into `model_upstream_overrides` (`upstream='nvidia/broken-qa-primary'`), **wait ≥32 s** (`refreshOverridesIfStale(30)`), then delete the row.
4. **Stop all NVIDIA** (S2+): `sed NVIDIA_BASE_URL=http://127.0.0.1:9/v1` + FPM reload. Must be restored by a `trap ... EXIT`.
5. **Stop a rescue provider**: `redis-cli -p 6390 setex "plugsky:cooldown:rescue-provider/<BASE_URL>" 180 1` — BASE_URL exactly as in `.env` (scheme included; client checks `rtrim(base,'/')`). Go base = `https://opencode.ai/zen/go/v1`, Zen = `https://opencode.ai/zen/v1`, OpenRouter/Groq from their env vars.
6. **Always trap-restore**: cooldown keys, `NVIDIA_BASE_URL`, override row, FPM reload; then revoke the temp key and `bash /tmp/restore_qa.sh`.
7. Assert the served lane per stage via `X-Plugsky-Served-Model` (display name; `:free` stripped).

### Pitfalls (learned the hard way)

- **Redis is port 6390** — the default 6379 has none of our keys.
- Override refresh is 30 s; a faster re-request still uses the old primary.
- Watchdog cron (`*/5`) has first-strike tolerance: one failed run = `slow`, only the 2nd flips `broken` — keep the drill under ~5 min and it won't write overrides/emails.
- Alert rule pages on **paid 503s > 5 / 15 min** — the drill should produce at most one 503 (S5).
- Go/Zen catalogs: `GET https://opencode.ai/zen/go/v1/models` and `/zen/v1/models` (public). Use curl — Python urllib's UA gets Cloudflare 403.
- **Zen free tier is CLI-only** (`403 FreeTierError: can only be used from within OpenCode`) except `space-bunny-free`; use the **Go** endpoint for the other free models.
- **Groq returns the answer in `reasoning`** (not `reasoning_content`) — lanes must promote it.
- **JSON-mode**: reasoning-first lanes must not advertise `json-mode` (promoted reasoning is never valid JSON); granite / lfm / north-mini-code are the verified free JSON lanes.
- `Empty content from upstream completion` = reasoning burned the token budget; check the lane's `promote_reasoning` flag.

### User-log triage loop (mandatory after the drill)

```sql
-- cluster errors by user/model/plan (2h)
SELECT ue.user_id, u.plan, ue.model, COUNT(*) n, MAX(ue.created_at) last
FROM usage_events ue JOIN users u ON u.id=ue.user_id
WHERE ue.status='error' AND ue.created_at >= NOW() - INTERVAL '2 hours'
GROUP BY 1,2,3 ORDER BY n DESC;
```
- Trace a request: `grep <request_id> storage/logs/*.log` → `chat.fallback.failed`, `chat.second_provider.failed`, `error_middleware.caught`.
- Classify: free-plan capacity (accepted) vs real bug. If a code bug → fix in `RescuePolicy`/`OpencodeClient`/`ChatService`, run `php vendor/phpunit/phpunit/phpunit --no-coverage tests/Unit` (must stay green), deploy, re-run the failing stage, confirm 0 new errors.
- Last verified drill (2026-10-05): S0–S6 all PASS; 21 errors/2h across platform were 16 free-plan micro waves + drill noise; the JSON subset was fixed (strict JSON instruction + capability flags).

## Current routing contract — 2026-10-03 reliability revamp (READ FIRST)

The owner-authorized reliability audit + follow-up (`Plugsky/docs/reliability/reliability-audit-2026-10-03.md`,
`reliability-followup-2026-10-03.md`) replaced the old catalogue plumbing. Everything below this section dated
2026-09-29 is historical — dead engines/tails/caps in it are superseded by this contract.

**Live engines (8, probed on the account):** `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` (262k, vision+tools+json),
`openai/gpt-oss-20b` (131k), `meta/muse-glimmer-30b` (131k, tools+json), `nvidia/nemotron-3-ultra-550b-a55b` (262k, tools+json),
`meta/llama-3.2-11b-vision-instruct` / `meta/llama-3.2-90b-vision-instruct` (131k, vision),
`nvidia/nemotron-3.5-lightning-30b-a3b` (262k), + 3 embedding aliases.
**Retired by NVIDIA — never re-add:** super-120b, nano-9b, mini-4b, step-3.7-flash, inkling, gemma-3n, mistral-nemotron
(all 410), laguna / z-ai glm (hang). Unknown engines are class −1 → never eligible.

**Alias classes and failover ranges (ModelLadder):**
| Class | Aliases | Range |
|---|---|---|
| 0 entry | micro, lite, tiny, nano, nano-omni, gpt-oss-20b, gemma3-*, phi, mistral-small, minimax, qwen-next, lightning, coder-fast, deepseek-flash | `[0,0]` — free must never consume premium |
| 1 vision/light | plus, gpt-oss, vision-fast, qwen-vl, gemma-4 | `[0,1]` |
| 2 professional | pro, reasoning, thinker, kimi, coder, coder-ultra, deepseek-pro, longctx, llama4, mistral-medium | `[2,3]` — never the entry engine, may upgrade |
| 3 frontier | max, frontier, ultra | `[3,3]` |

- Eligibility = class range **+ engine capability profile + serving context** (`candidateMatches`, `engineContext`).
  `qwen-vl` accepts only llama-90b; unknown engines rejected; watchdog peer-heal obeys the same rules.
- Every blocking/streaming response carries **`X-Plugsky-Served-Model`** (exact engine); body field `x_plugsky_served_model`.
- Paid plans keep all-model access. **Frontier may receive a professional engine under the published failover policy —
  exact 550B delivery is not promised.** Shared NVIDIA capacity + rescue budgets are the remaining availability risks.
- Rescue (second provider) is a **reviewed list** (`RescuePolicy`): opencode go / zen / openrouter / groq / selfhost,
  interleaved round-robin, class-distance ordered, per-provider + per-model cooldowns (funds 300s, retired 3600s,
  transient 15s). `bin/refresh-fallback-model.php` is discovery-only (writes nothing); discovery never changes routing.
- Rescue context = hard check on actual estimated tokens; primary context floor is only an ordering preference
  (requiring it dead-ended micro/vision — fixed 2026-10-03.004). Vision rescue: openrouter nano-omni:free +
  llama-3.2-11b-vision:free. Tiny caps promote reasoning text rather than returning empty.
- Usage truth: `UsageOutcomes` dedupes one outcome per user+endpoint+request_id (ok > error > fallback);
  `fallback` rows are attempts ("unresolved"), not customer failures.

**Second-provider fallback (OpenCode Zen) verified 2026-09-29 (morning):** account funded — all four
candidates return 200 (`space-bunny-free` ~1.0s, `glm-5.3-flash` ~1.0s, `deepseek-v4.1-flash` ~1.3s,
`qwen3.8-flash` ~1.9s). The chain runs free-first, then paid; end-to-end proven through
`OpencodeClient::complete()` → `OC-E2E-OK` (served by space-bunny-free). This tier is strictly
last-resort (paid plans only, reached when NVIDIA is broadly failing). Daily caps protect spend:
$3/day global, $1/day per user, 3,000 calls/day.

**2026-09-29 evening update — NVIDIA wave + credit drain:**
- OpenCode wallet exhausted during an NVIDIA-wide congestion wave (`Insufficient account funds` on
  every attempt; 332 failures/30 min at peak, 1,755 cascade attempts in the day). Paid hourly success
  dipped to 69–84% (24h still 96.89%). Funds circuit-breaker shipped: on funds-exhausted the provider
  is skipped for 6h (fail fast instead of ~30s stalls). **OpenCode Go (opencode.ai/go) top-up/subscribe
  = owner action**; chain runs cheapest/free model first.
- NVIDIA lane now rotates **3 API keys** (`NVIDIA_API_KEY`, `NVIDIA_API_KEY_2`, `NVIDIA_API_KEY_3` →
  Build key "NVIDIABuild-Autogen-96", exp 2027-03-29; independent quota pools). Key-pool 503s
  ("Worker local total request limit") are per-pool; rotation spreads load across pools.
- **Planned first fallback: self-hosted NIM `TobiasLogic/Qwen2.5-Coder-32B-abliterated`** (vLLM/OpenAI-
  compatible, port 8000). Not yet wired: requires a GPU host reachable from the VPS — the VPS itself has
  no NVIDIA GPU (`--gpus all` impossible there). Pending owner: run NIM on GPU box / cloud GPU, expose
  URL (public or tunnel), then wire as tertiary provider in `ChatService::trySecondProvider` chain. Also new:
**embed aliases return a clean 400 on /v1/chat/completions** (use /v1/embeddings) — they no longer
cascade to a chat model. Watchdog auto-heal verified (mistral-medium healed to nano-omni peer; restores
original after 3/3 passes). Remote MCP (`https://plugsky.com/mcp`) stress-verified: 30/30 chats
under 10-way concurrency, zero 5xx.

## Fastest models (live latency)

| Model | Speed | Upstream | Best For |
|-------|-------|----------|----------|
| `plugsky-vision-fast` | **~0.3s** | Llama 3.2 11B Vision | Image + text, fast OCR-style checks |
| `plugsky-qwen-vl` | **~0.3s** | Llama 3.2 11B Vision | Vision-language, tools |
| `plugsky-gemma-4` | **~0.3s** | Llama 3.2 11B Vision | Compact multimodal |
| `plugsky-micro` (free) | **~1.0s** | Nemotron 3 Nano Omni 30B | Fast free chat, tools, vision |
| `plugsky-lite` (free) | **~1.0s** | Nemotron 3 Nano Omni 30B | Fast free chat, tools, vision, JSON |
| `plugsky-gpt-oss` | ~1.5s | Nemotron 3 Nano Omni (GPT-OSS 20B backup) | GPT-OSS profile — upstream NIM worker degraded 24 Sep; served by omni, GPT-OSS kept as backup |
| `plugsky-gpt-oss-20b` | ~1.5s | Nemotron 3 Nano Omni (GPT-OSS 20B backup) | Same routing as above |
| `plugsky-max` | ~1.2s | Nemotron 3 Ultra 550B MoE | Largest model, deep reasoning |
| `plugsky-deepseek-pro` | ~0.6–2.8s | Nemotron 3 Ultra 550B MoE | Reasoning + code |
| `plugsky-llama4` | ~1.0s | Nemotron 3 Ultra 550B MoE | MoE-class quality |
| `plugsky-frontier` | ~1.2–4s | Nemotron 3 Ultra 550B MoE | Top-tier capability |
| `plugsky-ultra` | ~1.1–2.2s | Nemotron 3 Ultra 550B MoE | Deep reasoning, long tasks |

## Everyday workhorses (~1–2.5s, Nemotron 3 Nano Omni 30B: tools + vision + reasoning)

`plugsky-plus`, `plugsky-pro`, `plugsky-reasoning`, `plugsky-thinker`, `plugsky-kimi`,
`plugsky-nano`, `plugsky-minimax`, `plugsky-nano-omni`, `plugsky-lightning`,
`plugsky-mistral-small`, `plugsky-coder-fast`, `plugsky-deepseek-flash`,
`plugsky-qwen-next`, `plugsky-phi`, `plugsky-tiny`, `plugsky-gemma3-nano-2b`

- 128K context, streaming, JSON mode, tool calling.
- `plugsky-nano-omni` / `plugsky-pro` are the best default for general chat + vision + tools.

## Specialists

| Model | Upstream | Notes |
|-------|----------|-------|
| `plugsky-coder` | Nemotron 3 Ultra 550B MoE | Coding + reasoning |
| `plugsky-coder-ultra` | Poolside Laguna XS 2.1 | Fast specialised code (~0.4–5s; occasionally 90s-timeouts on the poolside worker, then recovers — watchdog swaps after 3 consecutive fails) |
| `plugsky-mistral-medium` | Mistral Nemotron → omni cascade | Fast-fails the Mistral upstream at 8s, then cascades to Nano Omni (~9s total). Mistral NIM returned 500s on 24 Sep |
| `plugsky-mistral-small` | Nemotron 3 Nano Omni | Fast small general |
| `plugsky-embed`, `-embed-nim`, `-embed-multilingual` | Nemotron 3 Embed 1B | Embeddings, 2048-dim, ~0.25s. **Chat endpoint returns 400 by design — use POST /v1/embeddings** |

## Reliability behaviour

- GPT-OSS 20B (`openai/gpt-oss-20b`) is currently **degraded on NVIDIA NIM** (times out >90s; direct probe). Both gpt-oss aliases now route to Nano Omni primary with GPT-OSS as backup; the client fast-fails GPT-OSS at 9s instead of 20s.
- Each model has a **same-profile backup** plus a peer `fallback_chain`; if the primary fails, the request cascades automatically (verified: dead primary → backup still returns 200).
- Generic last-resort: `plugsky-micro` → `plugsky-minimax` → `plugsky-lite`, plus a distinct-bucket tail `nvidia/nemotron-3-super-120b-a12b` (own NIM worker, ~0.2–0.6s) so cascades never stack on the shared nano-omni bucket.
- NVIDIA 503 “Service temporarily overloaded” now cascades (was skipping the fallback chain). Transport errors (cURL 28 timeouts) no longer surface as 500 — they cascade, and if every attempt fails the API returns a retryable 503 (OpenAI error shape).
- Ultra-primary models (`max`, `frontier`, `ultra`, `longctx`, `coder`, `deepseek-pro`, `llama4`) back up to super-120b first (separate bucket), then nano-omni.
- Model health-check cron now warms every model every **5 minutes** (was 15) to reduce NVIDIA cold starts.
- NVIDIA NIM caps ~16 concurrent requests per model; Plugsky guards at 12 with an 8s wait, then cascades.
- `nvidia/nemotron-3-super-120b-a12b` is only a backup now (intermittent 429).
- Full-NVIDIA outages: primary + ~4 cascade attempts, then the OpenCode second provider (paid plans only,
  free model first), then a retryable 503 with OpenAI error shape. Worst-case wait is bounded by a global
  deadline (~240s for large max_tokens, ~125s for small) — 2026-09-29 live incident showed 208s→503 during
  an NVIDIA-wide window with all buckets cooling.
- Usage is **unlimited on paid plans under fair-use RPM** (no daily/weekly token caps; token counts are
  observability only). Free = 500 req/day, Trial = 2,000 req/day; RPM by tier (60/300/600/1,500/5,000).

## ⛔ Do not use / removed (EOL on NVIDIA NIM)

`plugsky-qwen-next`-style upstreams `qwen/qwen3-next-80b`, `stepfun-ai/step-3.7-flash`,
`thinkingmachines/inkling`, `openai/gpt-oss-120b`, `minimaxai/minimax-m3`,
`meta/llama-3.2-3b-instruct`, `meta/llama-4-maverick`, `microsoft/phi-4-multimodal`,
`nvidia/nv-embed-v1`, `baai/bge-m3` are **gone** (HTTP 410). Do not target them.
Newest catalog models `z-ai/glm-5.3`, `moonshotai/kimi-k3`, `google/gemma-4-31b-it`
time out on this account and are **not enabled** (re-probed 24 Sep, 90s timeout).
Also probed and **rejected 24 Sep**: `deepseek-ai/deepseek-v4.1-flash` (timeout),
`nvidia/nemotron-3.5-lightning-30b-a3b` (timeout), `google/diffusiongemma-26b-a4b-it` (timeout),
`meta/muse-glimmer-30b` (200 but empty content), `z-ai/glm-5-3` (404).
Healthy extras already wired as backups: `nvidia/nemotron-3-super-120b-a12b` (~0.2–0.6s),
`nvidia/nemotron-3.5-content-safety` (~0.7s, classifier only), `poolside/laguna-xs-2.1` (~0.27s warm, coder-ultra primary).

## Recommendations by task

| Task | Model |
|------|-------|
| Fast free chat / classification | `plugsky-lite` |
| General chat + tools + vision | `plugsky-pro` |
| Hard reasoning / long analysis | `plugsky-max` or `plugsky-reasoning` |
| Coding | `plugsky-frontier` or `plugsky-coder` |
| Vision / images | `plugsky-nano-omni` or `plugsky-vision-fast` |
| Translation / simple text | `plugsky-lite` |
| Embeddings / RAG | `plugsky-embed` |

## Tiers

- **Free**: `plugsky-micro`, `plugsky-lite` (both Nemotron 3 Nano Omni) — tools + vision + JSON enabled.
- **Paid**: all other chat models.
- **Special**: `plugsky-embed*` (embeddings endpoint only, not chat).

## 2026-09-29 Evening Session Log — outage resilience fixes (READ THIS FIRST)

**Timeline / what happened:**
1. NVIDIA-wide congestion wave → paid-user hourly success dipped to 69–84% (24h 96.47%); many NVIDIA
   pools hit "Worker local total request limit (16/16)".
2. All cascade traffic fell to OpenCode Zen → its wallet drained → `Insufficient account funds` on all
   attempts (332 failures/30min peak). Owner topped up funds; Zen funded again.
3. Root cause of "free pick stopped working": `.env OPENCODE_FALLBACK_MODELS` had been overridden with a
   PAID-only list, silently replacing the free-first pick; then the funds circuit-breaker (too broad)
   suppressed the whole lane including the working free model. Both fixed.

**OpenCode Zen/Go facts learned (live-tested, decisive):**
- **GO API EXISTS: `https://opencode.ai/zen/go/v1`** (docs: opencode.ai/v2/docs/console/go) — endpoints
  `/chat/completions`, `/responses`, `/messages`; `GET /zen/go/v1/models` returns **30 Go models**.
  Our OpencodeClient now uses this base URL (both keys work). NOT `opencode.ai/go/v1` (that 404s).
- Go free (unlimited, limited-time): **`space-bunny-free`, `longcat-2.5-preview-free`** — both verified 200.
- Go cheap gems (chat/completions-compatible, verified 200): `mimo-v2.6-flash` ($0.14/$0.28, 4.9s),
  `glm-5.3-flash` ($0.15/$0.50, 0.87s), `deepseek-v4.1-flash` ($0.15/$0.60 off-peak), `hy3` ($0.14/$0.58),
  `longcat-2.0` ($0.30/$1.20). `muse-spark-*`/`grok`/`gpt-luna` use `/responses`; `qwen3.8-*`/`minimax-*`
  use `/messages` (Anthropic protocol) — not usable via our chat/completions client without changes.
- Free models on the plain Zen API other than space-bunny-free are CLI-only ("free tier can only be used
  from within OpenCode"); Go endpoint is the correct way to reach the rest.
- Go-plan models `mimo-v2.6-flash`, `mimo-v2.5`, `longcat-2.0` → "Model is unavailable";
  `gpt-5.4-nano` → "Model access is disabled"; `deepseek-v4-flash-vision-exp`/`deepseek-v4-flash` → Go-only.
- API-callable FREE: **`space-bunny-free` only** (~1.5s).
- API-callable PAID measured: `glm-5.3-flash` 1.08s, `minimax-m2.5` 1.18s, `deepseek-v4.1-flash` 1.52s,
  `qwen3.8-flash` 2.26s (qwen3.5/3.6-plus legacy, keep as tail).
- Test with proper headers (`User-Agent` + `X-Client: plugsky-fallback/1.0`) — bare Python urllib gets
  Cloudflare `403 error code 1010`.

**Live fallback chain (ordering strategy: CHEAPEST first, then FASTEST within tier):**
`space-bunny-free → deepseek-v4.1-flash ($60/$120 tier, fastest of cheapest tier) → glm-5.3-flash (fastest
paid overall) → minimax-m2.5 → qwen3.8-flash → qwen3.5-plus → qwen3.6-plus`

**Self-maintaining (since OpenCode changes catalog/pricing daily):**
- `/usr/local/bin/opencode-model-test.py` (cron 04:25) — calls all free+cheap candidates, writes
  `/etc/opencode-free-pref` (working free models, fastest first). Log `/var/log/opencode-model-test.log`.
- `/usr/local/bin/opencode-chain-refresh.sh` (cron 04:30) — rebuilds `OPENCODE_FALLBACK_MODELS` =
  working-free + price-tiered paid present in catalog; reloads FPM only on change.
  Log `/var/log/opencode-chain.log`. Never re-add CLI-only frees to the API chain.

**NVIDIA lane:** 3 rotating keys (`NVIDIA_API_KEY`, `_2`, `_3`=Build "NVIDIABuild-Autogen-96" exp 2027-03-29)
via `dependencies.php` → `NvidiaClient` per-request rotation (independent quota pools).
**Rules still active (unchanged):** Zen calls 3,000/day global, 1,000/day/user; $3/day global, $1/day/user;
funds breaker (gated: only when chain has NO free model, 1h window); MIN_COOLING=3; timeout 60s.
**Planned first fallback:** NIM `TobiasLogic/Qwen2.5-Coder-32B-abliterated` — needs GPU host reachable from
VPS (VPS has no NVIDIA GPU) → wire into `ChatService::trySecondProvider` chain when URL exists.
**Snapshots:** `Plugsky/deploy/2026-09-29-*` (surge-fixes, nvidia-3key, opencode-lane-revive,
opencode-auto-chain, opencode-model-tests).

## 2026-09-29 Late — Final 7-tier stack + OpenRouter + alert semantics (LATEST)

**Tier map (each tier knows its role; responses are tagged `x_plugsky_model`, suffix `(zen)` / `(or)`):**
1. NVIDIA primary (nemotron family) — 3 rotating keys, independent pools
2. NVIDIA backup/fallback — same-profile backup + cascade tails (super-120b, muse-glimmer, nano-omni)
3. OpenCode Go free — `space-bunny-free`, `longcat-2.5-preview-free` (unlimited free, Go endpoint)
4. OpenCode Go cheapest — mimo → hy3 → glm-flash → deepseek (Go endpoint, subscription usage)
5. Zen free → 6. Zen cheapest (pay-as-you-go, fund-capped $1/user/day, $3/day global)
7. **OpenRouter free-first, then cheapest paid** (`/api/v1`, own key + HTTP-Referer/X-Title headers):
   free: nemotron-3-super-120b:free (0.54s) → space-bunny-alpha → nemotron-nano-omni:free → gemma-4-26b:free
   → cohere/north-mini-code:free → inkling:free → nemotron-ultra-550b:free → openrouter/free
   paid tail (tested): ~deepseek/deepseek-v4-flash-latest ($0.01/$1.25) → ibm-granite-4.0-h-micro ($0.02/$0.11)
   → openai/gpt-oss-20b ($0.02/$0.09) → deepseek/deepseek-v4-flash-0731 ($0.02/$0.32). Key limit $100.
Final: clean retryable 503.

**Alert semantics (bin/alert-paid-errors.php, cron */15 → mustafa@plugsky.com):**
- funds_exhausted > 0 → funds alert; second_provider(rescue) failures > 10 → rescue alert;
  fallback.failed > 800/15m → SEVERE wave alert. 1h throttle per alert type.
- Individual `fallback.failed` counts during a wave are NORMAL (system absorbs them) — threshold is 800
  for this reason. Do not lower it back to 50 (alert spam, 2026-09-29 lesson).

**Known-good self-test:** `OPENCODE_FALLBACK_MODELS=none OPENCODE_ZEN_MODELS=none php /tmp/lane_test.php`
forces the OpenRouter tier; expect `served by: nvidia/nemotron-3-super-120b-a12b:free (or)`.

**Open outage audit (18:51 UTC):** NVIDIA pools saturated (503 16/16, 429) but paid users 100%/100%/85.6%
last hour (top talker), zero platform 503s — the 7-tier stack was coping; the earlier alert was a threshold
artifact, not a user-facing outage.

## 2026-09-29 FINAL FIX — free-only rescue bypasses budget caps (root cause of "structure failing")

**Gap:** during waves, heavy paid users hit the Zen spend caps (`second_provider.capped`, e.g. 441 events/2h for
user 1521) and `trySecondProvider` returned null WITHOUT trying the FREE rescue tiers → user received 503 after
35–96s waits. Budget caps must never block $0 models.

**Fix:** `OpencodeClient::complete(..., bool $freeOnly = false)` + ChatService capped branches (all 4: spend_global,
spend_user, calls_global, calls_user) now attempt a free-only pass (Go free → Zen free → OpenRouter free) before
giving up. Verified: free-only call served by `space-bunny-free`; 8× concurrent stress all served.

**OpenRouter free inventory (20):** 8 chat models in chain (nemotron-super-120b:free 0.54s, space-bunny-alpha,
nano-omni:free, gemma-4-26b:free, cohere-north-mini-code:free, inkling:free, nemotron-ultra-550b:free,
openrouter/free) + cheap paid tail (deepseek-v4-flash-latest $0.01/M, granite-micro, gpt-oss-20b,
deepseek-v4-flash-0731). Excluded: google/lyria-* (media gen), content-safety (classifier). Leftovers available if
needed: ling-3.0-flash-sante:free, dots-3-note-preview:free, laguna-s-2.1:free, lfm-2.5-2.6b:free.

## 2026-09-29 — FINAL verified state (post-rescue fixes 77ae600a6, 15b6dce03, c84360838)

Two follow-up bugs found in retest loop and fixed:
1. Free-only rescue referenced `$client` before instantiation (undefined var, caught silently -> 503).
   Fix: instantiate OpencodeClient inside each of the 4 capped rescue blocks; 45s bounded; logs
   `chat.free_rescue.ok/miss/failed`.
2. Rescue unreachable during heavy waves (MIN_COOLING=3 + CASCADE_ATTEMPT_TIMEOUT=30 exhausted the
   deadline inside NVIDIA cascade first). Tuned: MIN_COOLING=2, CASCADE_ATTEMPT_TIMEOUT=20 (.env).

Verified live: 503s 41/25m -> ~1/2m; 275+ free rescues OK / 0 failed; stress 8/8; paid users 953/admin 100%.
Alert semantics: rescue-failures>10 or severe-wave>800 per 15m (email mustafa@plugsky.com). Silent = healthy.

Full session history: Plugsky/docs/SESSION_2026-09-29_AI_MODEL_RESILIENCE.md
Local backup: Plugsky/backups/plugsky-2026-09-29-ai-resilience.tar.gz

## 2026-10-12 — outage drill ladder re-verified + entry free-lane rescue fix (LATEST)

- **Drill ladder (3 passes, all restored clean):** NVIDIA dead → OR free nvidia lane; +zen/OR cooled → **Go free
  (`opencode/longcat-2.5-preview`)**; +Go cooled → clean 503 (entry aliases never use paid rescue); class-2 pro →
  **`groq/openai/gpt-oss-120b`** → **`byteplus/deepseek-v4-flash-260425`** → clean 503; restore →
  `nvidia/nemotron-3-ultra-550b-a55b` + embeddings 200.
- **RescuePolicy fix:** `allows()` free-lane class bypass now applies to entry/light aliases (class ≤1) for ALL plans
  (was `$freeOnly` only). Symptom before: paid users' class-0 requests (micro/lite) dead-ended 503 while healthy free
  lanes sat idle (same-class OR free lanes were the only class-0 rescue candidates). Unit suite neutral.
- **Provider balances (2026-10-12):** OpenRouter $15 credits fully spent (paid lanes dead; `:free` lanes still serve —
  no credits needed). OpenCode Zen 402 insufficient funds (Go subscription fine). Funds breaker +
  `plugsky:cooldown:rescue-provider/<base>` cooldowns handle it; no active drain.
- **Drill mechanics:** cool a whole provider via
  `redis-cli -p 6390 setex "plugsky:cooldown:rescue-provider/<BASE_URL rtrimmed>" 300 1`; after any root `sed -i .env`
  ALWAYS `chown www-data:www-data .env && chmod 640 .env`; create the drill key BEFORE and revoke it AFTER the
  restore-verification request (revoking early → 401 on S6).
- **NVIDIA catalog:** 80 live, 0 new/removed; deepseek-v4.1-flash / gemma-4-31b / kimi-k3 still >40s timeouts on
  this account.

### 2026-10-13 — BytePlus fallback verified + free packs exhausted (operational fact)

- BytePlus IS the active 4th rescue tier: BytePlus-side records show **4,137 reqs / 120.3M tokens (Oct 8–13)** on
  seed-2-0-mini / deepseek-v4-flash / deepseek-v4-pro / seed-2-0-lite (≈90% input cache hit). Console "Package usage 0%"
  = packs for models we don't call; our packs are consumed.
- **Free-token packs EXHAUSTED (500K/500K)** for seed-2-0-mini, deepseek-v4-flash, deepseek-v4-pro → all further
  byteplus lane calls bill **pay-as-you-go**, bounded by `RESCUE_TIER_USD_BYTEPLUS` ($5/day default; today hit $4.25).
  To restore the free allowance the owner would need new packs/subscription in the BytePlus console.
- Verify anytime: `arkcli usage stats --start <date> --by model` (root login: omit `--mine`; SSO via
  `arkcli auth login --no-browser` two-phase) and `arkcli usage balance --type free-quota`.

### 2026-10-13 — BytePlus Coding-Plan subscription lane (`byteplus-coding`) added

- **New rescue lane `byteplus-coding`** — base `https://ark.ap-southeast.bytepluses.com/api/coding/v3`
  (`BYTEPLUS_CODING_BASE_URL`), models `BYTEPLUS_CODING_MODELS` = deepseek-v4-flash-260425, deepseek-v4-pro-260425,
  dola-seed-2-0-pro, glm-5-3-flash-260828, deepseek-v4-1-flash-260910, seed-2-0-lite-260428. Calls bill the
  **prepaid Coding Plan quota** (check: `arkcli usage plan --product coding-plan`), NOT pay-as-you-go.
  `seed-2-0-mini` is NOT covered by the plan (404 UnsupportedModel).
- Ordering: profiles for the new models use ranking price $0.001 → tried **before any metered lane**; free lanes
  still come first by policy. Lane order in `OpencodeClient::complete()`: go → zen → openrouter →
  **byteplus-coding** → byteplus (PAYG) → groq → selfhost. Served rows record as `byteplus-coding/<model>`.
- Admin visibility: `/admin#health` "Outage rescue lanes" includes the `byteplus-coding` probe row.
- Verified live 2026-10-13: NVIDIA blackholed + go/zen/OR cooled → pro served by `byteplus-coding/glm-5-3-flash-260828`,
  coder by `byteplus-coding/dola-seed-2-0-pro`; plan percent ticked (session 0.0000015% → 0.096%).

### 2026-10-13 (later) — daily cap for the subscription lane + full lane verification

**Daily cap (subscription lane):**
- `RESCUE_TIER_USD_CODING` (default **$4/day**) bounds `byteplus-coding`; spend tracked in
  `plugsky:rescue:spend:byteplus-coding:<utc-day>` (micro-USD, computed from the PAYG-equivalent profile prices of
  the coding models). When exhausted the lane is skipped until the next UTC day → falls through to metered lanes.
- **Ordering is now 3 tiers** in `OpencodeClient::complete()`: `0 free` → `1 byteplus-coding (prepaid)` → `2 metered`
  (healthy first, then up to 6 half-open cooling lanes across tiers). The prepaid lane is ALWAYS tried before any
  metered lane while its daily budget remains.
- Check: `/admin#health` → "Outage rescue lanes" spend row; `arkcli usage plan --product coding-plan` for the
  plan's own session/weekly/monthly windows (reset: session daily ~00:54 Beijing, weekly Mon, monthly on the 8th).

**Full lane verification (direct call to every model, 2026-10-13):**
- **go: 11/11 working** (glm-5.3-flash 0.8s, glm-5.3 1.0s, deepseek-v4.x 1.4–1.7s, longcat 1.8–2.0s, kimi-k2.7-code 1.2s, hy3 2.1s, mimo 6.9–8.9s)
- **byteplus (PAYG): 4/4 working** (seed-mini 1.0s, deepseek-flash 1.8s, deepseek-pro 2.1s, seed-lite 3.1s)
- **byteplus-coding (subscription): 6/6 working** (deepseek-v4-flash 1.6s, deepseek-v4-1-flash 1.5s, deepseek-pro 2.8s, glm-5.3-flash 3.8s, seed-lite 4.0s, dola-seed-2-0-pro 8.1s)
- **groq: 1/1 working** (gpt-oss-120b 0.1s)
- **zen: 0/9 — DOWN** ("Insufficient account funds"; space-bunny-free is CLI-only) → needs top-up
- **openrouter: 0/23 — DOWN** (paid: "Insufficient credits"; free: "free-models-per-day-high-balance" — OR applies a
  free daily cap when the account balance is $0) → needs top-up for any OR lane
- **Probe note:** reasoning-first models return EMPTY content at `max_tokens=8` (reasoning burns the budget) —
  probe with ≥64 tokens or accept `reasoning`/`reasoning_content`; do not call these lane failures.

### 2026-10-13 (evening) — Go subscription usage API + chain re-verification

- **`GET {OPENCODE_BASE_URL}/usage`** returns the Go subscription windows (same numbers as the console):
  `{"usage":{"rolling":{"percent":N,"resetsAt":…},"weekly":{…},"monthly":{…}}}` — both keys see the same plan.
  Quick check: **`php bin/opencode-usage.php`**. The plain Zen API has NO usage endpoint (404). Observed
  2026-10-13: rolling 1% · weekly 10% · monthly 22% (monthly renews Nov 1) — our Go lane traffic IS billing the plan.
- **NVIDIA engines direct check (2026-10-13):** nano-omni 1.0s · gpt-oss-20b 0.8s · lightning 5.7s · super-120b 0.7s ·
  ultra-550b 6.4s · muse-glimmer 8.8s · llama-11b-vision 0.25s — all OK. **llama-3.2-90b-vision still times out**
  (known broken; the watchdog override routes `plugsky-qwen-vl` → nano-omni — keep that override).
- **Go subscription models:** 11/11 working (mimo-v2.6-flash is flaky — retry passes).
- **Chain sweep (16 aliases + embeddings, all 200):** during the muse-glimmer/super-120b wave, class-2/3 aliases
  (pro/max/ultra/kimi/coder/nano-omni/deepseek-flash) served via the **free Go lane (`opencode/longcat-2.5-preview`)** —
  free-first rescue working as designed; micro/reasoning/deepseek-pro/lightning stayed on NVIDIA; vision aliases on
  llama-11b; qwen-vl on nano-omni via override.
- **Backup-override proof (S1-style):** primary overridden to a broken upstream → requests still 200
  (micro → nano-omni cascade; pro → ultra-550b cascade). Overrides cleaned after; only the qwen-vl watchdog override remains.

### 2026-10-13 — NVIDIA catalog research + entitled specialist models (report: docs/nvidia-catalog-opportunities-2026-10-13.md)

- Our NVIDIA key: `/v1/models` lists 80 entries but **most are not entitled** (404 "Function not found for
  account"). Verified WORKING beyond the 8 core engines: **`nvidia/nemotron-parse-2.0`** (document parsing),
  **`nvidia/riva-translate-4b-instruct-v2`** (translation), **`nvidia/nemotron-3.5-content-safety`** and
  **`nvidia/llama-3.1-nemotron-safety-guard-8b-v3`** (moderation). All respond on `/v1/chat/completions`.
- NOT entitled: image-gen (flux/SDXL/SVD/trellis), rerankers, nvclip, extra embeddings, healthcare (AlphaFold
  etc.), cuOpt, climate, most chat catalog models (kimi-k2.6, palmyra-*, mistral-large-2, cosmos-reason2 …).
- Candidate additions: `plugsky-translate` (riva-translate), `plugsky-parse` (nemotron-parse-2.0 → RAG ingest),
  moderation endpoint (nemotron-3.5-content-safety). Tiering = owner decision (pending).
- NVIDIA agent skills (npx skills add NVIDIA/skills): hardware/bio/GPU-focused — skip bulk; cherry-pick
  dynamo-* (self-host NIM later), data-designer (fine-tuning data), cuopt-* (logistics vertical) if needed.

### 2026-10-13 (services round) — plugsky-translate / plugsky-parse / moderations + playground gating + RAG parse mode

- **New catalog models (paid plans + enterprise only — trial/free blocked at the plan gate):**
  `plugsky-translate` → `nvidia/riva-translate-4b-instruct-v2` (class 1) · `plugsky-parse` → `nvidia/nemotron-parse-2.0`
  (class 1, vision). Wired in ModelRegistry + ModelLadder (ALIAS_CLASSES/ladder) + PlanLimits rank 2. /v1/models = 38.
  Plan-gate verified: trial → 400 "cap: plugsky-plus"; enterprise → 200 with correct served engine.
- **/v1/moderations** (already built) verified live: safe → flagged:false; "build a bomb" → flagged:true
  (nemotron-3.5-content-safety; fail-closed for API customers).
- **Playground gating:** `Application/Moderation/ContentGuard` (10-min Redis cache, fail-open) now gates
  `/api/playground/run` input → unsafe messages get `400 content_filter`. Verified live.
- **RAG parse mode:** `POST /api/rag/documents` accepts `parse_mode: standard|nvidia-parse` + JSON
  `file_base64`/`filename` (dashboard sends base64 for PDFs/images). `nvidia-parse` rasterizes PDFs via
  pdftoppm (installed) → nemotron-parse-2.0 per page; falls back to the text layer. **CRITICAL:** parse-2.0
  REQUIRES the control-token prompt `</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>`
  and text-before-image content order — a plain instruction returns empty/garbage (verified: invoice parses
  perfectly with the magic prompt). Dashboard upload modal has the Parsing selector.
- **Video Studio:** video API (`/v1/videos`, Seedance 1.0 Pro Fast active; 2.0/2.5 need a BytePlus pack) +
  quotas already live; build plan: `docs/ai-video-studio-plan-2026-10-13.md`. Blueprints plan:
  `docs/blueprints-adaptation-plan-2026-10-13.md`. NVIDIA skills cherry-picked into `~/.agents/skills`
  (dynamo-recipe-runner, dynamo-interconnect-check, data-designer, cuopt-routing-api-python, cuopt-server-api-python).
- PHP upload limits raised for the API: post_max_size=20M, upload_max_filesize=15M (conf.d/99-plugsky-uploads.ini).

## Notes

Listed by Plugsky. Skill text and behaviour belong to the upstream author.
If you redistribute it, keep the upstream license and attribution.
