---
name: plugsky-audit
slug: plugsky--plugsky-audit
description: "Full Plugsky AI platform audit and operations runbook \u2014 model testing, cascade/backup/fallback verification, NVIDIA-outage failover to the opencode second provider, Redis acceleration checks, dashboard/client-panel QA, log/error triage, and the Ployer end-to-end user flow. WHEN: audit plugsky, plugsky models not working, check all AI models, outage failover, fallback failing, redis speed, check dashboard, plugsky errors, model health, stress test models, verify AI agents, google login error, client panel audit."
collection: Plugsky
category: Platform Ops
license: Plugsky
source: https://plugsky.com/skills
install: "Download the Plugsky skills pack (see below)"
---

# plugsky-audit

Full Plugsky AI platform audit and operations runbook — model testing, cascade/backup/fallback verification, NVIDIA-outage failover to the opencode second provider, Redis acceleration checks, dashboard/client-panel QA, log/error triage, and the Ployer end-to-end user flow. WHEN: audit plugsky, plugsky models not working, check all AI models, outage failover, fallback failing, redis speed, check dashboard, plugsky errors, model health, stress test models, verify AI agents, google login error, client panel audit.

## Install

```bash
Download the Plugsky skills pack (see below)
```

## Skill

# Plugsky AI — full audit & operations runbook

Last full run: **2026-10-19** — **38/38 OK, 0 fails, stress 100%** (catalogue now 38 models; 0 cascades observed).
Findings & fixes that run: `mistralai/mistral-nemotron` is **410 Gone** at NVIDIA — plugsky-mistral-medium re-pointed to
nano-omni (primary) + super-120b (backup), now ~1.5s (was 8s); **opencode zen is OUT OF FUNDS again** (outage fallback
returns 503 until topped up — health-check alerts); cascade fallback attempts are now capped
(`CASCADE_ATTEMPT_TIMEOUT=45`s) so one slow pool can't eat the whole budget; `OPENCODE_DAILY_USER_CAP` raised to 1000
(USD caps still guard). Prior: **2026-09-27** — **36/36 OK, 0 fails, stress 100%** (two consecutive clean audits; catalog extras show isolated 500/503s on non-ladder models). Prior: **2026-09-26 (evening)** — **36/36 OK, 0 fails, stress 100%** (best result; new tail models lightning/laguna/muse-glimmer serving; previous run 31/36 was capacity-only). Prior: **2026-09-26** — **31/36 OK, stress 100%** (p50 1.2s, p95 2.3s); the 5 failures were all
transient NVIDIA 429s (capacity), zero code bugs. Run 1 was polluted because the QA user was on `trial`
(cap plugsky-plus → 14 models returned an expected 400). For a full-model audit, set the QA user to
`scale` first:
`UPDATE users SET plan='scale', trial_ends_at=NULL WHERE email='qa.audit@plugsky.com';`
(and revert to `trial` + revoke the temp key afterwards).
New: `meta/muse-glimmer-30b` added to the cascade tail (fast, ~0.7s when healthy). `openai/gpt-oss-20b`
recovering (2/3 probes) — watchdog override kept until 3/3. Audit harness still marks some 200-status
catalog entries FAIL on content assertions (empty/reasoning-only) — check the body, not just status.

## 0. Access & ground rules

| Thing | Value |
|---|---|
| VPS | `ssh -i ~/.ssh/plugsky_deploy root@72.61.81.91` |
| Backend | `/var/www/plugsky/backend` (PHP 8.4 + Slim, `bin/`, `storage/logs/`) |
| Frontend | `/var/www/plugsky/frontend` (`dashboard.html` = client panel SPA, `app.html`, landing) |
| API | `https://api.plugsky.com` (`/v1/*`), dashboard `https://plugsky.com` (`/api/*`) |
| Model ladder | `src/Application/Inference/ModelLadder.php` (+ `ModelDetails.php`) |
| Chat path | `ChatService.php` → `NvidiaClient.php`; second provider `OpencodeClient.php` |
| Redis (dedicated) | docker container `plugsky-redis` → `127.0.0.1:6390`, cache-only (LRU 256MB, no persistence) |
| Env | `backend/.env` — **MUST be `www-data:www-data 640`** (see 8.1) |
| Secrets | `NVIDIA_API_KEY`, `PLUGSKY_API_KEY` (customer key, use the one in Ployer's `app/.env`), `OPENCODE_API_KEY` |
| Backups | `/var/www/plugsky/backend/backups/<audit-date>/` before every change |

Rules:
- **Reload PHP after code changes**: `systemctl reload php8.4-fpm` (hard `restart` if behaviour looks stale).
- **Never leave the NVIDIA key out of `.env`**; verify with `curl api.plugsky.com/health` → `{"status":"ok"}`.
- Pacing: the audited key has a plan rpm cap (30/min). Pace audit calls ≥2.2 s apart or the run will be polluted with 429s.
- opencode `-free` models are **client-locked** (`403 FreeTierError` outside the OpenCode client) — never wire them into the server; use the cheapest *paid* model for the outage tier (auto-selected, see 6).

## 1. Smoke (30 s)

```bash
curl -s https://api.plugsky.com/health                 # {"status":"ok",...}
K=$(ssh ... ployer env PLUGSKY_API_KEY)
curl -s https://api.plugsky.com/v1/models -H "Authorization: Bearer $K" | jq '.data|length'   # 38
curl -s https://api.plugsky.com/v1/chat/completions -H "Authorization: Bearer $K" \
  -H 'Content-Type: application/json' \
  -d '{"model":"plugsky-lite","max_tokens":8,"messages":[{"role":"user","content":"Reply: OK"}]}'
```
Dashboard: `https://plugsky.com/login` (QA account: `qa.audit@plugsky.com` / `QaAuditPass123!`; recreate with `php bin/qa_user.php create`, remove with `delete`).

## 2. Full model audit (all 36 + capabilities + stress)

```bash
cd Plugsky/tools/audit
PLUGSKY_API_KEY=… AUDIT_PACE_MS=2400 node plugsky_audit.mjs --stress=6 --concurrency=1
```
Pass criteria: `models_total 36`, `chat_fail 0`, `stress_success 100%`.
For a deeper run (extras + bigger stress) stay under the rpm cap: `--stress=14 --concurrency=1`.
Capabilities checked by the harness: JSON mode, tools, streaming, vision, embeddings (2048-dim).
Fallback detection compares `response.model` against the **declared upstream** (`/v1/models`), not the alias.

## 3. Cascade / backup / fallback integrity

- Order per model: primary → `backup_upstream` (different NIM bucket) → `fallback_chain` peers (capability-filtered) → distinct-bucket tail `nvidia/nemotron-3-super-120b-a12b` → generic (`micro/minimax/lite`).
- Fast-fail overrides: `openai/gpt-oss-20b` 9 s, `mistralai/mistral-nemotron` 8 s (avoid 20 s stalls).
- Client retries fast 429/503 with backoff 1/2/4/8 s before cascading.
- Transport failures (DNS/refused/reset) are classed transient (`ConnectException`/`RequestException`) so they always cascade.
- All-cascade-failure → **503 retryable** (never a raw 500).
- Watchdog: `bin/health-check.php` every 5 min; 3 fails → auto-swap upstream; restores the original after 3/3 healthy probes.

## 4. Redis acceleration verification

```bash
ssh … 'redis-cli -p 6390 ping'                                  # PONG
ssh … 'redis-cli -p 6390 --scan --pattern "plugsky:*"'          # rl:/rlt: slot: keytouch: ws: models ladder:resp
```
Checks:
1. Catalog cache: two `GET /v1/models` → second has lower latency (log `X-Plugsky-Cache: hit` on the read path).
2. Response cache: two identical `temperature:0` chat calls → second returns `x_plugsky_cache:"hit"` (body) and is ~10× faster.
3. Rate limit: >30 req/min → `429` with `Retry-After` (Redis counters `plugsky:rl:*`).
4. Slots: `plugsky:slot:<upstream>` tracks live traffic and drains; TTL 180 s self-heals leaked slots.
5. Fail-open: `docker stop plugsky-redis` → requests still 200 (PostgreSQL fallback); `docker start plugsky-redis` → cache resumes.
6. Speed: rate-limit op Redis ≈0.2 ms vs PostgreSQL ≈3.5 ms (~18×).

## 5. NVIDIA-outage failover drill (second provider)

> **Canonical drill:** the full staged chain test (main → backup → Go → Zen → OpenRouter → Groq → clean 503)
> with exact commands, trap-restore, response-cache/Redis-port pitfalls and the user-log triage loop now lives in
> the **`plugsky-models` skill → "Staged outage chain drill"**. Run that procedure; the notes below are the
> historical single-stage version kept for reference.

```bash
# blackhole NVIDIA (fast connection-refused) then run the offline suite
ssh … 'cd /var/www/plugsky/backend && cp .env .env.bak && sed -i "s|^NVIDIA_BASE_URL=.*|NVIDIA_BASE_URL=http://127.0.0.1:9/v1|" .env'
# tests: chat / memory continuity / JSON / tools  → expect 200, provider=opencode, fallback=second-provider
#        vision → clean 503 "temporarily unavailable" (no vision fallback available)
ssh … 'cd /var/www/plugsky/backend && mv .env.bak .env && chown www-data:www-data .env && chmod 640 .env'
curl -s api.plugsky.com/v1/chat/completions … # back on NVIDIA
```
Memory/continuity: the fallback receives the **same body** (messages/tools/response_format/system), so the task history continues — verified by a two-turn recall test (code `APOLLO-4471` recalled through the fallback).
**Always restore ownership/permissions on `.env` after this drill** (see 8.1).

**2026-09-26 drill: PASSED.** NVIDIA blackholed → request served by `x_plugsky_provider: opencode`,
`x_plugsky_fallback: second-provider` ("FALLBACK-OK"); restore → NVIDIA serves again.
Two fixes made during this drill:
- Hard-down gate: transport failures (ConnectException / refused / timeout) now allow opencode even when
  fewer than `OPENCODE_MIN_COOLING` buckets are cooling (a real outage doesn't mark 429 cooldowns).
- Cap-counter bug: call caps were incremented on *denied* attempts, inflating the counter and blocking
  legitimate fallbacks (4,271/1,000). Now check-then-increment only for attempts that proceed.
- Spend caps (primary): `OPENCODE_DAILY_USD_CAP=3` global, `OPENCODE_DAILY_USER_USD_CAP=1` per user
  (micro-USD from response tokens). Call caps (secondary): 3000/day global, 200/day/user.

## 6. Outage fallback model refresh (pricing changes)

```bash
ssh … 'cd /var/www/plugsky/backend && php bin/refresh-fallback-model.php'
```
- Pulls `models.dev` pricing ∩ opencode accessible models, probes cheapest-first, writes `OPENCODE_FALLBACK_MODELS` (3 cheapest working) and `OPENCODE_VISION_MODELS` (only if a real image probe passes).
- Cron: daily 05:15 (`fallback-model.log`).
- 2026-09-24 results: `qwen3.8-flash` ($0.15/M in, $0.47/M out) → `qwen3.6-plus` → backups; **no** server-usable vision model (free tier locked, zen paid vision endpoints unavailable for this token, go-tier requires session header) → vision intentionally has no second provider.

## 7. Client dashboard / panel QA

- `/dashboard` is a **SPA with hash sections**; each sidebar item routes via `handleRoute()` → `PAGES_RENDER`.
- Registered pages (22): overview, keys, usage, models, agents, tools, rag, fusion, marketplace, team, billing, integrations, support, affiliates, enterprise, gpu, settings, security, status, legal, audit (+ `downloads` opens a **modal**, by design).
- Playwright check: login via `POST /api/auth/login`, load `/dashboard`, click `#nav a[href="#<page>"]`, assert hash + `#pageTitle` + content length, zero console/network errors.
- Known-benign: fresh accounts show empty states (`No agents yet`, `No events yet`, `No usage data`); Settings contains a “Coming soon” subsection; Support may show a transient “Failed to load” under rapid successive navigation (not reproducible standalone).
- Re-run: `node plugsky-dashboard-test.js` / `plugsky-dash-final.js` in `Ployer/tools/…` (Playwright harness lives with the audit scripts).

## 8. Failure playbook

1. **All endpoints 500 `Secret APP_SECRET is still set to a placeholder value`**
   → `.env` unreadable by FPM (usually after root edits / cp / mv). Fix:
   `chown www-data:www-data .env && chmod 640 .env`; verify `sudo -u www-data test -r .env`.
2. **Raw `Guzzle ConnectException` 500 on chat** → transient matcher regressed; ensure
   `$isTransient = $e instanceof ConnectException || $e instanceof RequestException || preg_match(…overload|throttl|refused|failed to connect…)`.
3. **`ResourceExhausted 16/16`** → NIM worker saturating; check slot counters, backups rebalanced to `super-120b`; never raise `MAX_INFLIGHT_PER_MODEL` above 12.
4. **Empty/slow cascade answer** → a request needing tools/vision/JSON cascaded to an incompatible peer; check `entrySupports()` filtering and `OPENCODE_VISION_MODELS` empty behaviour.
5. **gpt-oss / mistral aliases slow** → upstream NIM degradation; fast-fail overrides + re-mapped primaries (gpt-oss aliases now serve nano-omni, GPT-OSS as backup).
6. **Google login 500** → same `.env` readability cause; verify `GET /api/auth/google?next=/dashboard` returns 302 to `accounts.google.com` with `redirect_uri=https://plugsky.com/api/auth/google/callback`.

## 9. Logs & alerts

```bash
tail -200 /var/www/plugsky/backend/storage/logs/app.log      # error_middleware.caught entries
tail -50  /var/www/plugsky/backend/storage/logs/error.log
tail -20  /var/www/plugsky/backend/storage/logs/health.log   # watchdog per-model status
tail -20  /var/www/plugsky/backend/storage/logs/fallback-model.log
```
Filter out expected noise: `AuthException` 401s (bots) and `RateLimitException` 429s.
Alerts: `MODEL_HEALTH_WEBHOOK` (preferred) or email via `MODEL_HEALTH_EMAIL` → PHPMailer/SMTP; empty = logs only.

## 10. Ployer end-to-end (consumer of the API)

```bash
curl -c /tmp/c -X POST https://ployer.app/api/auth/login -H 'Content-Type: application/json' \
  -d '{"email":"demo.employer@ployer.app","password":"StrongPass123!"}'
curl -b /tmp/c -X POST https://ployer.app/api/ai/run -H 'Content-Type: application/json' \
  -d '{"task":"match","input":"Job: Nanny in Bahrain…","model":"plugsky-lite","max_tokens":300}'
```
Pass: `status=completed`, JSON result (score/gaps/…), ≤ ~30 s. Ployer's poller caps at 3 attempts and shows a friendly retry message on `upstream_timeout`.

## 11. Change log (condensed)

- **2026-09-24** — cascade hygiene (gpt-oss off backups, super-120b tail), fast-fail timeouts, transient matcher class-based + overload keywords, 503 instead of 500, watchdog every 5 min, second provider (opencode) with pricing-aware refresh + memory continuity, Redis across rate-limit/slots/auth/catalog/ladder/response-cache with fail-open, `.env` permission fix, dashboard QA (22/22), Google OAuth verified.

## 12. Playground Beta (`/playground-beta`) QA + privacy

**Isolation model (verified 2026-09-24)**
- Conversations: `GET/PATCH/DELETE /api/playground/conversations[/{id}]` all scope `WHERE user_id = :uid`; QA could not read/patch/delete another user's conversation (404). List returns only own rows.
- AI Drive: storage moved to `backend/storage/ai-drive/u<id>` **outside the webroot**; files served only via `GET /api/playground/drive/file?u&n&t` where `t = HMAC-SHA256(APP_SECRET, "u<id>|<name>")` (32 hex). Old `/ai-drive/...` URLs are `deny all` in nginx. Verified: signed URL 200; tampered token / wrong user / traversal → 403; old URLs → 403.
- Local worker: `POST /api/playground/local` only reaches workers whose registry `user_id` matches a prefix of the caller's own API keys.
- Shares: payload contains only title/model/messages (no user ids or files). Resolve is public by code — now IP rate-limited (60/min, Redis) against brute-force; verified 60×200 then 429.

**UI QA (Playwright, 39 visible controls, full flow 15/16 + 1 verified manually)**
- Panel toggles (left/right), new chat, model picker (38 models), theme toggle, activity bar (Explorer/Chat history/Plugins/Agents/Fusion/RAG/Team/Marketplace/Settings), attach file, send/reply (2 s), share link (resolves anonymously), download export (md/json/txt), local-worker guidance, reload persistence — all pass.
- Chat history panel shows the conversation with title/model/message count (selector-timing artefact in one automated check; verified manually).
- Fixes shipped: clipboard `writeText` rejection no longer throws (feedback only on success); CSP extended with `blob: https://esm.sh 'wasm-unsafe-eval' worker-src blob: https://*.hf.co` so the JS sandbox worker + in-browser WASM/transformers features run.
- Expected cosmetic: `/api/keys` 403 (plan key limit) is handled by the playground revoking/recreating its own "Playground Beta" key; browser logs the response but functionality is unaffected.

**Playground QA command**: `node pb-flow.js` (Login-as-QA harness; injects JWT into localStorage, sweeps controls, runs the user flow, screenshots to `/tmp/ployer-ui/pb-flow-*.png`).

## 13. Client dashboard deep-audit (all sections)

Runbook: `node dash-sections.js` (load every section, count controls/icons/broken images, screenshot) and `node dash-deep.js` (open every modal/filter/toggle, verify with 0 console/network errors).

Sections verified 2026-09-24 (0 errors, 0 broken images):
| Section | What works |
|---|---|
| `#models` | 38/38 cards (= live `/v1/models`); ⓘ hover popovers show released/updated/params/architecture/context/max-output/speed/cost/origin/cutoff/modality/capabilities/best-for/use-cases/benchmarks/id |
| `#agents` | Create-agent dialog (name, description, model, system prompt, temperature, max tokens) |
| `#tools` | Create Tool wizard (6 steps, HTTP/Webhook/MCP types), Browse Templates |
| `#rag` | New-collection dialog (name, description, embedding model) |
| `#fusion` | New fusion chain (name, strategy, model ordering), install/test chain |
| `#marketplace` | 64 templates, Details modal (installs, rating, description), filters |
| `#integrations` | 61 integrations, category filters (IDE/Desktop/CLI/SDK/…), copy + test actions |
| `#team` | Members table, Invite modal (email, role, message) |
| `#billing` | Plan cards, Monthly/Annual toggle (prices change), coupon field (invalid → error), invoices |

**Metadata drift check (fixed 2026-09-24)**: dashboard popovers must match `ModelDetails` ↔ `ModelLadder` upstreams. Drift found & fixed for micro/lite/plus/pro/reasoning/deepseek-flash/qwen-next/phi/gemma3-nano-2b (9B/4B/550B/120B → `MoE 30B (activated ~3B)`), mistral-medium (`dense 12B`), gemma-4 (`dense 11B`), coder-ultra (`dense 2.1B`), nano context (1M → 131,072), gpt-oss aliases (`underlying_model` → nano-omni primary with GPT-OSS backup), and a missing `plugsky-gpt-oss-20b` entry. Verify:
```bash
php -r 'require "vendor/autoload.php"; Dotenv\Dotenv::createImmutable(__DIR__)->safeLoad();
foreach (["plugsky-plus","plugsky-mistral-medium","plugsky-coder-ultra"] as $id)
  { $m = Plugsky\Application\Inference\ModelDetails::get($id, Plugsky\Application\Inference\ModelLadder::ladder()[$id]); echo "$id {$m["architecture"]} {$m["parameters"]}\n"; }'
```

## 14. Docs sync (`https://plugsky.com/docs`)

Single-page app (`frontend/docs.html`, anchored sections). Audit method: extract all factual claims and compare against live sources, then fix drift.

**Fixed 2026-09-24**
- Plans (both tables): Free trial **7 → 14 days**, API keys 1 → 2, cap noted as `up to plugsky-plus`; **added Hobby $8/mo (60 RPM, 3 keys)**; Starter **60 → 120 RPM**; annual note `20% (Hobby: 50%)`; enterprise annual wording.
- Model count: removed all `31 models` claims → "every model in the catalog".
- Embeddings example: `plugsky-embed-2` → `plugsky-embed`; output dimension `1536` → **2048**.
- Image generation: kept **coming soon** but corrected to planned aliases `dall-e-3`/`dall-e-2` and documented the temporary `501`.
- Migration guides: `/v1/messages` and `/v1/bedrock/converse` claims replaced with **Roadmap** notes (endpoints do not exist; use OpenAI-compatible mode).
- Smart routing: `model="auto"` clarified — only valid on `POST /v1/plugsky/route`, not `/v1/chat/completions` (which accepts `model="plugsky-fusion"`).
- Releases: heading → **Plugsky CLI v0.1.2 (2026-07-05)**; CLI card tag updated; dead `v0.1.0-desktop`/`v0.1.0-web` GitHub tag links retargeted to the releases page.
- Footer anchors: `#agents/#rag/#marketplace` → dashboard sections; tier links retargeted to existing anchors; all 63 internal anchors now resolve.

**Code bugs found via docs validation (fixed)**
- `POST /v1/moderations` returned **500**: the guard model `meta/llama-guard-4-12b` hangs on NIM. Switched to `nvidia/nemotron-3.5-content-safety` with keyword category mapping (safe → flagged false; toxic → flagged true + categories).
- `POST /v1/images/generations` returned a confusing 502 parse error (no image model in the upstream catalog) → now a clean **501 "not available on this deployment yet."**

Verify:
```bash
curl -s https://plugsky.com/docs | grep -c "31 models"      # 0
curl -si https://plugsky.com/docs | head -1                  # 200
curl -s -X POST https://api.plugsky.com/v1/moderations -H "Authorization: Bearer $K" -H 'Content-Type: application/json' -d '{"input":"hi"}'   # 200
```


## Admin panel audit — 2026-09-28

- **All 22 admin sections render** (SPA 130–1200ms, cold reload 140–1900ms), 0 console errors.
  QA account is NOT admin by default — promote temporarily:
  `UPDATE users SET role='admin' WHERE email='qa.audit@plugsky.com';` (revert to `user` after).
  Admin walk harness: `node admin-walk3.mjs` (local /tmp/tools-audit; needs token in localStorage `plugsky_token`).
- **Overview was slow (2–4.5s)**: `usage_events` (551K rows / 190MB) had no index on `created_at`.
  Added `idx_usage_events_created_at`, `(workspace_id, created_at)`, `(created_at, model)` → KPI query 127ms.
  Added Redis cache to `OverviewController` (60s TTL, `plugsky:admin:overview:v2`, `X-Plugsky-Cache: hit|miss`,
  `?fresh=1` bypass) → warm 83ms, cold 188ms.
- **Revenue was incomplete**: `revenue_records` had only enterprise contracts; Stripe subscription
  payments were never recorded (`onInvoicePaid` only wrote affiliate commissions). Fixed: webhook now
  records `type='subscription'` (idempotent by invoice id) + `bin/backfill-revenue.php` (read-only Stripe,
  31 invoices = $1,248 backfilled). Intervals normalised (`month→monthly`, `year→annual`) in webhook + DB.
- **Moderation page**: it is an error-triage queue (`status IN ('error','rate_limited','blocked')`), copy fixed.
  Data healthy: 247 failures/24h (0.84% of ~29K requests), 0 HTTP 5xx (Sep 25–26 had 22–26K/day / ~30% — pre-fix).
- **Model audit 2026-09-28: 36/36 OK, 0 fails, stress 100%** (3rd consecutive clean run); 42 cascades observed.
- **Outage drill PASSED**: NVIDIA blackholed → served by `opencode/qwen3.8-flash`
  (`x_plugsky_provider: opencode`, `x_plugsky_fallback: second-provider`) in 2s; restore → NVIDIA serves again.
  opencode spend today $0.012 of $3 cap (micro-dollar keys confirmed).
- Backups: `src/Interface/Http/Controller/Admin/OverviewController.php.bak-*`, `.../Billing/StripeWebhookController.php.bak-*`.

## 2026-09-28 — Revenue chart fix

- Symptom: "Daily (30d)" chart on `#revenue` rendered as a flat line at the bottom.
- Root cause: in the inline canvas script, `(by[r.d]=by[r.d]||0)+(+r.amount||0)` is an
  assignment expression whose sum is discarded → every daily value stayed 0 → flat line.
- Fix: `by[r.d]=(by[r.d]||0)+(+r.amount||0)` + empty-data guard ("No revenue data…").
- Verified: painted pixels 9,216 → 99,297; screenshot `/tmp/journey/revenue-page.png`.
- Backup: `frontend/admin.html.bak-20260928-*` (or the pre-fix copy in backups/).

## 2026-09-28 — Support tickets revamp (admin)

- **Admin access fix**: only `admin@plugsky.com` (id 1) had role=admin; owner's `mmarhama@gmail.com`
  (id 17) was `user` → admin panel silently redirected (`/dashboard?error=not_admin`), which broke
  "open support tickets". Fixed: id 17 promoted to admin. AuthMiddleware re-reads users from DB by id,
  so existing sessions pick up the new role immediately (no re-login needed).
- **New backend endpoints** (`TicketsController`):
  - `GET /api/admin/tickets/assignees` → admins for the transfer dropdown
  - `PATCH /api/admin/tickets/{id}` → `{status?, assigned_to?, priority?}` (validates status ∈ open/awaiting/closed,
    assignee must be a role=admin user, priority ∈ low/normal/high/urgent; sets/clears `resolved_at`)
  - `reply()` fixed: real statuses are **open/awaiting/closed** (old code offered pending/resolved which
    don't exist); replying with a message and no explicit status auto-sets **awaiting**; staff replies now
    **e-mail the ticket opener** (Mailer, non-fatal on failure).
- **New UI** (`admin.html`): tab-based list — **Open / Awaiting / Closed / All** with live counts, search
  (subject/email/ref/assignee), per-row Open + Close/Reopen, detail panel with Status select, **Transfer to**
  admin select + Apply, message thread, reply box with "mark as awaiting customer" + e-mail note.
  Helpers: `renderTicketsPage/renderTicketsTable/wireTicketTabs/wireTicketRows/ticketSetStatus/refreshTickets`.
- **Test data cleanup**: 83 test tickets deleted (ids ≤83 + the "Testing" id 99: E2E/QA/comprehensive runs);
  15 real tickets remain (12 open / 1 awaiting / 2 closed). 150 test users still exist (7 with QA-granted
  plans) — left in place, removable on request.
- Backups: `backend/src/.../TicketsController.php.bak-*`, `backend/src/Interface/Http/routes.php.bak-*`,
  `frontend/admin.html.bak-*`.

## Support operations — plan-sync, comps, billing intervals (2026-09-28)

- **Plan-sync audit query** (paying-but-free): see the error-triage skill §1. Root cause of PLG-87/89/90 was
  the August-era webhook that never applied `workspaces.plan`; the reconcile cron (*/15) now heals active subs.
- **Goodwill comps** (`plan_overrides`): ≤1 month of the paid plan for a proven paid-without-access case;
  `PlanOverrides::set($pdo, ws, plan, reason, expires)` + set `workspaces.plan`/`users.plan`; overrides win
  over Stripe during reconcile. Refunds → owner approval.
- **Billing interval check**: `STRIPE_PRICE_{HOBBY,STARTER,BUILDER,SCALE}[_ANNUAL]` must resolve via Stripe
  `Price::retrieve` (Builder $60/mo vs $720/yr); the dashboard Monthly/Annual toggle drives the interval sent
  to `/api/billing/subscribe` (was inverted once — verify the annual-button active check).
- **API keys**: unlimited per account (policy 2026-09-28); scopes filtered by plan; masked display
  `sk-live-****last4`; revoke instant; `plan_limit` 403s mean the cap code regressed.
- **Tickets module**: `/admin#tickets` has Open/Awaiting/Closed/All tabs, search, transfer (assignee=admin),
  close/reopen, same-line reply dropdown; replies e-mail the customer and auto-set `awaiting`.
  Full procedure: the `plugsky-support-tickets` skill.

## 2026-09-28 (evening) — audit + NVIDIA catalog review

- **NVIDIA live catalog: 81 models.** Probed the new ones: `mistral-large-2`, `kimi-k3`/`k2.6`, `nemotron-nano-3-30b`,
  `gemma-4-31b-it`, `glm-5.3` are either **404 (catalog-only, not deployed)** or **>40s (unusable)**. Only two new
  models actually work: **`nvidia/nemotron-parse-2.0`** (document parsing, 120ms) and
  **`nvidia/nemotron-3.5-content-safety`** (safety classifier, 250ms) — candidates for new aliases (product decision).
- **All Mistral upstreams are 404/410** → no Mistral is served by NVIDIA any more; don't re-add any `mistralai/*`.
- **Cascade budget**: `$perAttemptCap = min(150, 45 + max_tokens/100)`; global deadline `min(240, perAttemptCap+60)`.
  Fallback attempts now use `min(perAttemptCap, remaining, CASCADE_ATTEMPT_TIMEOUT=45)`.
- **Paid-user failure pattern** (2026-09-28): ed@forgetfulfarmer (scale) 44% errors / 120s per error during NVIDIA
  storms — cascade exhaustion; fixed by the attempt cap. chrollolucilfer2702 (starter) 1.6% (capacity noise);
  hy@n0s.tech (builder) 5%.
- **opencode zen**: 431 calls/24h last-resort usage; funds ran out → `chat.second_provider.failed Insufficient account funds`
  → 503 on hard NVIDIA outages. Owner must top up; the per-user call cap (now 1000) and USD caps stay.

## 2026-09-28 (late) — outage tier rebuilt + second NVIDIA key

- **opencode zen outage tier: WORKING again (free).** Findings:
  - zen paid models return `402 Insufficient account funds`; the `-free` models are client-locked
    (`403 OpenCode's free tier can only be used from within OpenCode`) — **except `space-bunny-free`**,
    which works server-side (1.3-1.8s, content + JSON mode ✅; limited-time unlimited).
  - `OPENCODE_FALLBACK_MODELS=space-bunny-free,glm-5.3-flash,deepseek-v4.1-flash,qwen3.8-flash`
    (client tries them in order; Go-plan models become usable once subscribed).
  - **opencode.ai/go = $10/mo** unlocks: MiMo-V2.6-Flash (~30k req/5h), DeepSeek V4.1 Flash (~26k),
    Muse Spark 1.3 Contributor (~45k, limited regions), GLM-5.3-Flash (~6.3k), Kimi K3 (110/5h) … subscribe
    with the same account → the same API key serves them (no code change).
  - Note: `space-bunny-free` is reasoning-heavy — tiny `max_tokens` (≤16) can yield empty `content`.
- **Second NVIDIA key added**: `NVIDIA_API_KEY_2` (NVIDIABuild-Autogen-83, exp 2027-03-28).
  `NvidiaClient` now takes `string|array` and **round-robins** keys per request (`authHeader()` full line /
  `authToken()` for Guzzle key=>value headers — do not mix those two up, it produces `Authorization: Authorization:` → 502).
  `dependencies.php` builds the filtered key array. Verified: requests 200, health-check 36 checked / 0 broken.
- **New NVIDIA models probed**: `nemotron-parse-2.0` (document-input parser — empty for chat, NOT a chat alias;
  use for document/RAG ingestion), `nemotron-3.5-content-safety` (classifier → "User Safety: safe";
  internal moderation only). Everything else new is 404 (catalog-only) or >40s.

## 2026-09-28 — RAG document ingestion built + vector index repaired

- **Critical bug fixed**: `ServiceLocator::set()` was never called anywhere, so
  `DashboardRepository::createRagDocument()`'s embedding step ALWAYS failed silently —
  every RAG document was stored **without vectors** and queries fell back to keyword ILIKE.
  Fix: `ServiceLocator::set($container)` right after the container build in `public/index.php`.
  Verified: new uploads get embeddings; query mode = **vector** (score 0.94); all 192 legacy
  chunks backfilled (`bin/backfill-rag-embeddings.php`, resilient: bounded slice for legacy
  oversized chunks) → 0 missing.
- **File/URL ingestion** (`POST /api/rag/documents` now accepts):
  - `file` (multipart, ≤15MB) — PDF via `smalot/pdfparser` (installed), images via vision
    transcription, text/md/csv/json/html direct (+tag stripping)
  - `source_url` (or `file_url`) — fetched with an SSRF guard (public hosts only) + up to 3
    redirects, host re-validated per hop; PDF/PNG/JPEG sniffed from magic bytes
  - images → `meta/llama-3.2-11b-vision-instruct` transcribes to clean markdown
    (env `RAG_PARSE_ENGINE=vision|nemotron-parse`; **nemotron-parse-2.0** emits raw layout tokens
    and needs post-processing — vision is the better default; parse adds no value until then)
  - Service: `src/Application/Rag/DocumentTextExtractor.php`
- **Paid plans more generous (2026-09-28)**: RPM hobby 60→300, starter 120→600, builder 300→1,500,
  scale 1,000→5,000 (docs.html updated); paid fair-use stays unlimited; zen per-user cap 1,000.
- **Stress test (all models)**: `--stress=72 --concurrency=3` → 36/36 OK, **100% success**,
  p50 149ms / p95 4s, 108 cascades (backup chains exercised).

---

## 2026-10-03 reliability revamp — CURRENT invariants (read this first; supersedes dated notes above)

Owner-authorized audit + follow-up by a second agent; deployed and verified. Reports:
`Plugsky/docs/reliability/reliability-audit-2026-10-03.md` + `reliability-followup-2026-10-03.md`; sanitized
evidence `evidence-20261003/`, `evidence-followup-20261003/`. Backups: `/var/www/plugsky/backend/backups/reliability-20261003/`
and `.../reliability-followup-20261003/` (code + pg_dump + per-file rollbacks + local independent copies).

### Routing contract (ModelLadder + RescuePolicy)
- Alias classes: 0 entry / 1 vision-light / 2 professional / 3 frontier. Ranges: `0→[0,0]`, `1→[0,1]`, `2→[2,3]`, `3→[3,3]`.
  Unknown engines = class −1 → never eligible. Eligibility = range + capability profile + serving context
  (`candidateMatches()`, `engineContext()`); `eligiblePrimary()` returns a qualifying engine or a clean 503/context error.
- Live engines (8): nano-omni(262k), gpt-oss-20b(131k), muse-glimmer-30b(131k), ultra-550b(262k), llama-3.2-11b/90b-vision(131k),
  lightning(262k), 3 embeds. **EOL (410) — never re-add:** super-120b, nano-9b, mini-4b, step-3.7-flash, inkling, gemma-3n,
  mistral-nemotron; laguna/glm hang.
- `qwen-vl` accepts only llama-90b. Watchdog peer-heal + `findFallback` obey the same class/capability/context gates.
- Paid plans keep all-model access; frontier may degrade to a professional engine (published policy) — exact 550B not promised.

### Rescue lanes (RescuePolicy + OpencodeClient)
- Reviewed model profiles only (class, context, vision, free, features, prices); unknown/random-router models rejected.
- Providers: opencode go, zen, openrouter, **groq**, selfhost. Round-robin across providers (3 rounds); within a provider,
  sorted by class-distance to the requested alias; candidate budget = half remaining when another candidate remains.
- Cooldowns: `rescue-provider/<base>` 300s on funds/auth; `rescue/<base>/<model>` 3600s retired/404, 15s transient.
- **Context rule (fixed 2026-10-03.004):** the rescue context check is a hard check on the **actual estimated tokens**;
  the primary's serving context is only a **preference** (`contextRank`, same-level first). Requiring the full floor
  dead-ended every 262k-floor alias (micro → 0 candidates → 36 visible errors). Vision aliases pass modality via the
  profile `vision` flag (nano-omni free+paid = vision true; llama-11b-vision:free lane). Tiny budgets (<64) get
  OpenRouter `reasoning:{effort:none}`; <32 promotes `reasoning_content` so a small cap can't yield an empty 200.
- Alerting: `alert-paid-errors.php` pages on `second_provider.failed > 40 AND visible 503s > 5` (attempt noise alone
  never pages; `funds_exhausted>0` and `visible 503s > 15` are separate rules).
- Free users AND capped paid users run `freeOnly` (never escape to entry class to dodge caps). Per-plan daily USD budgets:
  env default (+ hobby 5 / starter 5 / builder 8 / scale 12 / lifetime 12 / enterprise 25).
- Tool strictness enforced post-response (forced choice, allowed names, JSON-object args); OpenRouter pins providers and
  handles auto-only tool_choice (DeepInfra). Response carries `x_plugsky_provider` + `x_plugsky_model` + `x_plugsky_free_lane`.

### Streaming contract (SseCompletionGuard / StreamAbortRecorder)
- JSON/tool streams are assembled + validated **before delivery** (forced choice, allowed names, object args, `[DONE]` +
  visible answer); ordinary text stays incremental; partial output is never replayed. `hasAnswer()` = visible content/tool
  delta — role/reasoning/keepalive bytes do not count (first-answer deadline via `CURLOPT_XFERINFOFUNCTION` + low-speed guard).
- Raw stream lifetime **270s** (< inference FPM 310s); structured candidates ≤150s inside a 240s route budget; buffers 4MiB
  (prelude/structured) + 1MiB incomplete event; split CRLF supported. Committed streams never replay; mid-stream failure sends
  an SSE error event + `[DONE]`; if nothing committed, rescue is delivered as valid SSE chunks.
- Client disconnect → `StreamAbortRecorder` writes exactly ONE error outcome + `stream.client_disconnected` (attributed to the
  served engine); completed requests are not double-written.
- `X-Plugsky-Served-Model` header on blocking AND streaming responses.

### Truthful usage counting (UsageOutcomes)
- One outcome per (user, endpoint, request_id): `ok` > `error` > `fallback`; rows without request_id counted individually.
  `fallback` = attempt ("unresolved"); `success_count` from `ok`. Dashboard shows "N without recorded outcome".
- **Do not judge health from raw usage_events counts or "no 5xx" alone.** The revamp uncovered **11 builder + 27 scale real
  terminal errors/24h** that the old raw metric missed (dedup changes the picture).

### Infra
- NVIDIA concurrency = Redis leases `plugsky:slotlease:<worker>` (12/model, stale 1500s) — never erases active slots; 2s wait → clean 429.
  3 NVIDIA keys with per-instance randomized round-robin. Concurrency slots are a real capacity signal, not fail-open.
- Watchdog: `flock(storage/health-check.lock)` (overlapping crons skip), shared `ModelHealthProbe` (shuffled keys, one real
  probe/engine/job), preserves `last_ok_at`, mail prefers `MAIL_HOST/PORT/USERNAME/PASSWORD` then legacy `SMTP_*` with a 10s
  timeout; `--no-notify` for interactive runs. `refresh-fallback-model.php` is discovery-only.
- Response cache key `plugsky:resp:v3` includes plan + full validated body; `/v1/models` cache v1/v2/v3 dropped on override changes.

### Verification (current baseline)
- `php vendor/bin/phpunit` → 120 tests / 2,491 assertions (117 pass, 3 pre-existing skips) at the follow-up.
- Hardened checks supersede `full-audit.sh` for release claims: `tools/audit/plugsky_audit.mjs` (model battery/stress),
  `tests/Unit/Inference/*` (SseCompletionGuard, RoutingReliability, EquivalentRecovery, UsageOutcomes),
  `tests/Integration/InferenceStreamingTest.php`, plan-matrix structured streams, disconnect/vision/dashboard fixtures.
- Do NOT overclaim: strict JSON Schema mode, sustained peak-load SLO, >270s generation, SMTP message delivery, payments/OTP,
  external integration writes, and legacy Playground/agent SSE (still buffered) are unverified. `/v1/models` is public —
  revoked-key checks must hit protected chat (401).

## 2026-11-06 — security gate for every audit

Run the 7-skill suite before claiming an audit pass:

```bash
cd /var/www/plugsky/backend
bash bin/skills/run-all.sh
```

`skill-security.sh` must have **0 FAIL**: admin password/OTP/TOTP state, no active admin API keys/trusted devices, no `@fldubai.com` accounts, browser/MCP disabled, `/v1/payouts` 404, no `.env`/status secret leakage, JWT length, live-Stripe warning.

The user-journey skill must have **0 FAIL** (WARN = capacity). Paid acceptance is **WARN/Paused** while `PAID_LANES_ENABLED=false`; do not re-enable paid lanes to make it green.

Do not count `/api/playground/browser` as a harmless playground. In the 2026-11-06 incident it was a root-run, no-sandbox, arbitrary-session `evaluate` surface. Keep it disabled until rebuilt.

## Notes

Listed by Plugsky. Skill text and behaviour belong to the upstream author.
If you redistribute it, keep the upstream license and attribution.
