---
name: plugsky-surface-audit
slug: plugsky--plugsky-surface-audit
description: "Systematic full-surface audit of Plugsky \u2014 every client dashboard page, every admin page and all public/landing pages \u2014 using load checks + data-contract checks (API rows must appear in the DOM), plus the paid-user error triage loop. WHEN: \"audit all pages\", \"find broken pages\", \"page shows empty but has data\", \"dashboard bug\", \"systematic audit\", \"check all client/admin pages\", \"paid user error logs\", \"why is this page empty\", \"full surface check\", \"pre-release audit\"."
collection: Plugsky
category: Platform Ops
license: Plugsky
source: https://plugsky.com/skills
install: "Download the Plugsky skills pack (see below)"
---

# plugsky-surface-audit

Systematic full-surface audit of Plugsky — every client dashboard page, every admin page and all public/landing pages — using load checks + data-contract checks (API rows must appear in the DOM), plus the paid-user error triage loop. WHEN: "audit all pages", "find broken pages", "page shows empty but has data", "dashboard bug", "systematic audit", "check all client/admin pages", "paid user error logs", "why is this page empty", "full surface check", "pre-release audit".

## Install

```bash
Download the Plugsky skills pack (see below)
```

## Skill

# Plugsky Surface Audit — find broken pages systematically

Last updated: **2026-11-06** (Round 21). Full audit: `Plugsky/docs/ops/audit-skills.md` and `PLUGSKY-FULL-AUDIT-2026-10-27.md`.

The load-only checks ("section loads, 0 console errors") are **not enough**. Real bugs slip
through: `dashboard#audit` loaded perfectly with 0 errors but showed **"No events yet"** forever,
because the renderer read `d.audit` while `/api/audit` returns `{entries: [...]}`.
The missing layer is the **data contract**: *if the API returned rows, the page must render them*.

Run both layers for every surface: **client dashboard (20 sections) · admin panel (23 sections) ·
public/landing pages (sitemap, ~1789 URLs)**.

## 1. The two-layer method

| Layer | Question answered | Tool |
|---|---|---|
| Load audit | Does the page load, title render, 0 console/page errors? | `Plugsky/tools/audit/user-journey/journey.mjs`, `admin-walk3.mjs` |
| **Data-contract audit** | **Does the page render what the API returned?** | **`Plugsky/tools/audit/surface-audit.mjs`** |
| API sweep | Do all endpoints used by pages answer correctly? | `api_sweep.py` + `admin_sweep.py` (VPS `/tmp`) |
| Public sweep | Do all landing/article pages return 200 with title+content? | `surface-audit.mjs --mode=public` |
| Paid-user triage | Are paying users hitting errors, and are they code or capacity? | `tools/monitoring/collect_errors.py` + DB queries |

## 2. Run the data-contract audit

```bash
# token for dashboard (QA) or admin (promote first)
ssh -i ~/.ssh/plugsky_deploy root@72.61.81.91 \
  "cd /var/www/plugsky/backend && bash /tmp/prep_qa.sh >/dev/null 2>&1; \
   API=https://api.plugsky.com; \
   T=\$(curl -s $API/api/auth/login -H 'Content-Type: application/json' \
     -d '{\"email\":\"qa.audit@plugsky.com\",\"password\":\"QaAuditPass123!\"}' \
     | python3 -c 'import sys,json;print(json.load(sys.stdin)[\"access_token\"])'); echo \$T" > /tmp/token.txt

cp Plugsky/tools/audit/surface-audit.mjs /tmp/tools-audit/   # playwright lives there
PLUGSKY_TOKEN="$(cat /tmp/token.txt)" node /tmp/tools-audit/surface-audit.mjs --mode=dashboard
PLUGSKY_TOKEN="$(cat /tmp/token.txt)" node /tmp/tools-audit/surface-audit.mjs --mode=admin
node /tmp/tools-audit/surface-audit.mjs --mode=public

ssh -i ~/.ssh/plugsky_deploy root@72.61.81.91 'bash /tmp/restore_qa.sh'   # ALWAYS restore
```

Pass criteria: **0 FAIL** in all three modes (WARN = verify manually).
Reports: `/tmp/surface-audit/report-{dashboard,admin,public}.json`.

### What the detector flags
- `pageerror` / render-error phrases (`Failed to load`, `Could not load`).
- **API-has-data-but-UI-empty**: any captured `/api/*` JSON list with rows > 0, the DOM contains an
  empty-state phrase (`No events yet`, `No data`, …) and none of the first rows' values appear in the DOM.
- WARN when API data isn't matched in the DOM but no empty phrase (usually dropdown-only data).

### False positives to know
- Locked sections (trial plan) legitimately make 0 API calls — no data, no failure.
- Catalog endpoints used only for pickers (e.g. `/v1/models` on the playground) may WARN — verify manually.
- Empty tables are honest when the API returns 0 rows (incidents, payouts) — that is a PASS.

## 3. When the detector fires — fix recipe

1. Reproduce the page in the browser, capture the network JSON (Playwright `page.on('response')`).
2. Compare the **API key names** with what the renderer reads:
   `grep -n "api('/api/<endpoint>')" dashboard.html admin.html` then read the renderer lines.
   (The `entries` vs `audit` class of bug.)
3. Patch the renderer (prefer tolerant reads: `d.entries || d.audit || []`).
4. `node --check` the extracted script; deploy the HTML with an explicit single-file `scp`
   (never multi-source `scp` into `frontend/` — it has clobbered index.html before).
5. Re-run `surface-audit.mjs` for that mode; FAIL → OK.
6. Record the fix in `Plugsky/error-logs/RESOLUTIONS.md`.

## 4. Paid-user error loop (systematic)

```bash
cd Plugsky && python3 tools/monitoring/collect_errors.py --hours 24
```
Then classify every signature:
- `Upstream returned HTTP 429` across ≥3 buckets → **capacity**, not code (do not "fix" by removing cooldowns).
- `ResourceExhausted (16/16)` → capacity; check slots + rescue lanes.
- Raw `500`, `SQLSTATE`, `InvalidArgumentException`, unrecognized upstream wordings → **code bug**.
- `second_provider.failed Insufficient account funds` → owner top-up alert.

Per-user paid check:
```sql
SELECT u.id, u.email, u.plan, count(*) total,
       count(*) FILTER (WHERE ue.status='ok') ok,
       count(*) FILTER (WHERE ue.status='error') err,
       count(*) FILTER (WHERE ue.status='fallback') fb,
       round(avg(ue.latency_ms)) avg_ms, max(ue.latency_ms) max_ms
FROM usage_events ue JOIN users u ON u.id=ue.user_id
WHERE ue.created_at > now() - interval '5 days' AND u.plan NOT IN ('free','trial')
GROUP BY 1,2,3 HAVING count(*) FILTER (WHERE ue.status IN ('error','fallback')) > 0
ORDER BY err DESC;
```
Trace a failing request end-to-end before declaring capacity:
`grep "<request_id>" storage/logs/app.log` → every `chat.fallback*` line shows the lanes tried.
A request is only "capacity" when the trace shows every lane attempted (primary → backup → peers →
exhaustive candidates → rescue lanes) and all failed.

**Status semantics rule:** `usage_events.status='error'` must mean *terminal failure only*.
Failed attempts that later succeeded belong in `fallback` (a streaming bug once marked the first
attempt `error`, inflating the queues backlog and moderation queue).

## 5. Cadence
- After every deploy touching frontend renderers or API response shapes → data-contract audit.
- Daily → public sweep + `collect_errors.py`.
- Weekly → full dashboard + admin + API sweep, plus a canary outage drill (run in a quiet window —
  the drill blackholes NVIDIA for all traffic for ~2 min).

## 6. Admin metric audit — verify the denominator, not just the render (2026-10-27)

A KPI can render correctly and still be misleading. The 2026-10-27 audit found four such cases on `#overview`; all were fixed and browser-verified.

| Symptom | Real cause | Fix |
|---|---|---|
| “Models serving 49/51” vs health check “39/39” | 51 rows = **40 alias rows + 11 engine rows**; 39/39 is the public-alias subset | Split the KPI into **Alias routes serving** and **engines**; add a note |
| Requests card showed a large negative % | Compared **today so far** to **yesterday full day** | Label: “today so far · 36,795 yesterday (full day)” |
| GPU success showed 99.6% | Denominator omitted **703 expired** jobs | Include expired; **85.5%** with completed/failed/expired shown |
| “Open tickets 11” | Status query counts open + awaiting | Rename **Unresolved tickets**; show “open + awaiting customer” |

Also check:

- `mrr_today` is **today's subscription receipts**, not MRR. It is 0 when no subscription payment was booked today. Do not label it MRR.
- `usage_events.status='error'` must mean terminal failure only; successful fallbacks are `fallback`.
- Error Watch must separate **paid terminal errors**, **free errors**, and **fallbacks** (fallbacks are successful answers, not errors).
- A 60m table can include a pre-fix tail. Add a note with the clear time; do not compare it to a 5m window without saying so.
- A model health count can move because aliases share an upstream. One cooling engine can flip several alias rows at once.

**Browser verification pattern:** authenticate the Playwright/admin session, load `#overview`, then evaluate:

```js
const failed = performance.getEntriesByType("resource")
  .filter(r => r.name.includes("/api/admin") && (r.responseStatus || 0) >= 400);
const text = document.body.innerText;
return JSON.stringify({ failed, text: text.slice(0, 7000) });
```

Pass criteria: **0 failed admin requests**, and the KPI text matches the database/API denominator. On 2026-10-27 `#overview` and `#errorwatch` both passed with 0 failed admin requests.

## 7. 2026-10-27 audit evidence

- Full suite: **ALL SKILLS PASS**; paid acceptance **100/100** (91.93s final suite / 75.24s standalone before the guard fix).
- Custom skills: **14/14 available**; `skill-quality` **14/14**; router **19/19**; text outputs **9/9**; artifacts **4/4**; user journeys **13/13**.
- Admin `#overview`: 0 failed admin requests; alias/engine split, full-day request comparison, expired GPU jobs, and unresolved-ticket label all verified.
- Admin `#errorwatch`: 0 failed admin requests; **0 paid / 2 free** in 15m, 120 fallbacks, no alerts, 60m pre-fix note.
- Public privacy page: v1.1, 15 subprocessor rows, GPU Share note, retention caveat — browser-verified.

## 2026-11-06 — security surface layer (add to every surface audit)

The 2026-11-06 incident found a surface the page-load and data-contract checks did not cover: an authenticated **admin browser-automation endpoint** that could attach to arbitrary sessions and run `evaluate` as root. Add a security-surface pass to every dashboard/admin/public audit.

| Check | Expected | Evidence |
|---|---|---|
| `POST /api/playground/browser` with a valid key | 403/404/503, never 200 | `skill-security.sh` |
| `POST /mcp` `tools/call` without auth | 401/403 | `skill-security.sh` |
| `GET /api/admin/error-watch` without auth | 401/403 | `skill-security.sh` |
| `GET /api/auth/me`, `/api/keys` without auth | 401/403 | `skill-security.sh` |
| `GET /.env`, `/storage/.env` | no secret fields | `skill-security.sh` |
| `GET /api/status` | no `sk-live-`, secret, password, DB, JWT fields | `skill-security.sh` |
| `GET /v1/payouts` | 404 (Stripe-only endpoint) | `skill-security.sh` |
| Admin users | password + email OTP + TOTP; no active keys/trusted devices | DB + `skill-security.sh` |
| `@fldubai.com` accounts | 0 active | DB |
| Playwright service | inactive; no root process/sudo wrapper | systemd/pgrep |

Run:

```bash
cd /var/www/plugsky/backend
bash bin/skills/skill-security.sh
bash bin/skills/run-all.sh
```

**Metrics caveat:** `error-watch.request_outcomes.ok` previously counted every status below 500, including 429. A window can look 99% “ok” while 77% of clients are rate-limited. Use `success_rate`/`rate_limit_rate` and the 429-by-reason breakdown; do not call a page or service healthy from 5xx alone.

**Paid acceptance:** when `PAID_LANES_ENABLED=false`, `paid-acceptance` is a policy-paused WARN, not a code FAIL. Do not re-enable paid lanes just to make the dashboard green.

**r44 correction (2026-11-06 20:37 UTC):** `ok` is now 2xx only, with explicit `rate_limited`, `client_error`, `success_rate`, and `rate_limit_rate`. The 1-hour SLO no longer reuses a stale 60-minute snapshot; it merges four non-overlapping 15-minute records when no fresh 60-minute record exists and reports the covered minutes in `window_note`. Verify `windows.1h.window_minutes=60` and the `window_note` before quoting a 1-hour rate.

## Notes

Listed by Plugsky. Skill text and behaviour belong to the upstream author.
If you redistribute it, keep the upstream license and attribution.
