---
name: plugsky-full-stack-audit
slug: plugsky--plugsky-full-stack-audit
description: "Comprehensive multi-layer audit for the entire Plugsky project \u2014 infrastructure, AI model chain, every API endpoint, every client/admin/public page (data-contract checks), user journeys, paid-user error logs, performance/latency and security. One command runs all layers. WHEN: \"full audit\", \"audit everything\", \"find all bugs\", \"check all layers\", \"comprehensive audit\", \"why are users getting errors\", \"apply solid fixes for all errors\", \"no bugs\", \"pre-release audit\", \"lags\", \"slow\", \"all pages\", \"all APIs\", \"paid user errors\"."
collection: Plugsky
category: Platform Ops
license: Plugsky
source: https://plugsky.com/skills
install: "Download the Plugsky skills pack (see below)"
---

# plugsky-full-stack-audit

Comprehensive multi-layer audit for the entire Plugsky project — infrastructure, AI model chain, every API endpoint, every client/admin/public page (data-contract checks), user journeys, paid-user error logs, performance/latency and security. One command runs all layers. WHEN: "full audit", "audit everything", "find all bugs", "check all layers", "comprehensive audit", "why are users getting errors", "apply solid fixes for all errors", "no bugs", "pre-release audit", "lags", "slow", "all pages", "all APIs", "paid user errors".

## Install

```bash
Download the Plugsky skills pack (see below)
```

## Skill

# Plugsky Full-Stack Audit — every layer, nothing missed

One command runs all layers and prints a consolidated PASS/FAIL:

```bash
bash Plugsky/tools/audit/full-audit.sh            # all layers (public sweep included)
bash Plugsky/tools/audit/full-audit.sh --skip-public   # faster, skip the 1789-URL sweep
# reports → /tmp/full-audit/ ; exit 0 = all pass
```

The core principle: **loading is not working**. A page/API "loads" fine while rendering the
wrong thing (empty state despite API data), or a chain "works" while silently skipping the
cascade because an upstream error wording is unknown. Every layer therefore checks the
**contract**, not just the response code.

## Layer map

| Layer | What it proves | Tool / command | Pass criteria |
|---|---|---|---|
| **L0 Infrastructure** | API health, `.env` perms (`www-data:www-data 640`), Redis PONG, watchdog freshness (<15m), DB reachable | `full-audit.sh` L0 | all green |
| **L1 Model chain** | 36-model catalog, 8-model battery, provider aliases (`gpt-4o-mini`, `claude-*`, `gemini-*`), embeddings; full 33-model battery via `plugsky_audit.mjs --stress=6` | `full-audit.sh` L1 + `Plugsky/tools/audit/plugsky_audit.mjs` | 33/33 chat 200, stress 100% |
| **L2 API surface** | Every endpoint used by every dashboard/admin page (empty-body probes; billing/security mutations skipped) | `full-audit.sh` L2 → VPS `/tmp/api_sweep.py`, `/tmp/admin_sweep.py` | 0 non-OK |
| **L3 Pages (data contracts)** | Every client/admin/public page renders what its API returned | `Plugsky/tools/audit/surface-audit.mjs --mode=dashboard\|admin\|public` | 0 FAIL |
| **L4 Paid users** | Paying users with >5 errors/hour (actionable window); digest + per-user rates | `full-audit.sh` L4 + `tools/monitoring/collect_errors.py` | no unexplained terminal errors |
| **L5 Error classes** | Upstream error wordings the transient matcher does NOT know (these skip the cascade) | `Plugsky/tools/audit/unmatched-errors.py` | no NEW upstream wordings |
| **L6 Performance** | p50/p95 latency, slowest models, Redis cache hit | `full-audit.sh` L6 | p50 < 5s, p95 < 60s (agent workloads excluded) |
| **L7 Journeys** | signup → login/OTP → plan sync → dashboard → billing → renewal → support | `plugsky-user-journey` skill (`journey.mjs`) | 25/25 sections, 0 errors |
| **L8 Chain resilience** | NVIDIA-outage failover: chat/JSON/tools served by second provider, vision clean 503, restore | `plugsky-audit` skill §5 (`/tmp/outage_drill.sh`) | all lanes exercised, restore OK |
| **L9 Security/compliance** | admin gating (403 non-admin), RAG/Drive scoping, security headers, no secrets in logs, DKIM/SPF/DMARC | `plugsky-security` skill + `dig` checks | no leaks, DNS records present |

## The fix loop (applies to every layer)

1. **Reproduce with evidence** — capture the exact request/response/log line; never fix from a symptom.
2. **Classify**:
   - capacity (`429`, `ResourceExhausted 16/16`) → routing/cooldown work, never "fix" by removing cooldowns;
   - code bug (`500`, `SQLSTATE`, `InvalidArgumentException`, unrecognized upstream wording) → fix;
   - data contract (API rows > 0, UI empty) → fix the renderer key.
3. **Trace the full path** — `grep "<request_id>" storage/logs/app.log` shows every lane attempted
   (`chat.fallback*`, `chat.second_provider*`, `chat.free_rescue*`). A failure is only "capacity"
   when the trace shows every lane attempted.
4. **Patch minimally** — one concern per change; backup first (`cp f f.bak-$(date +%s)`), lint
   (`php -l` / `node --check`), reload (`systemctl reload php8.4-fpm` — never restart).
5. **Verify live** — re-run the failing layer; the check must flip FAIL → OK.
6. **Record** — `Plugsky/error-logs/RESOLUTIONS.md` (signature, root cause, fix, files, verification).

## High-value automated detectors (don't skip these)

- **`unmatched-errors.py`** — extracts the live `$isTransient` regex from `ChatService.php`, pulls
  distinct error messages from the VPS logs and prints the ones that DON'T match. Those are exactly
  the classes that skip the cascade (`Upstream request failed.`, `Internal server error`,
  `ValueError: multimodal…` were all found this way). Own-message entries (`Upstream provider error.`)
  and fixed-code signals (`SQLSTATE`, `Unknown plugsky model`, `_plugsky_no_reasoning`) are expected;
  a NEW upstream wording is a bug → add it to the matcher.
- **Surface data-contract audit** — the `entries` vs `audit` class of bug (API had 100 rows, UI showed
  "No events yet") is invisible to load checks. The detector fails when API rows > 0, an empty-state
  phrase is visible, and none of the row values render.
- **Status semantics** — `usage_events.status='error'` must mean terminal failure only. Failed
  attempts that later succeed belong in `fallback`; otherwise the queues backlog and moderation
  queue lie.

## Paid-user error runbook

```bash
cd Plugsky && python3 tools/monitoring/collect_errors.py --hours 24
```
- Group by signature; capacity vs code per the table above.
- Per-user query (paid, 5 days) and request-id tracing: see `plugsky-error-triage` skill §1/§3.
- If a user's error rate is high, reproduce **their exact workload** (e.g. faisal: multimodal content
  to a text model, `max_tokens` large, streaming) — the chain must cascade to a capable peer and
  return 200. If it doesn't, that is a code bug, not capacity.

## Cadence & automation

| Frequency | Run |
|---|---|
| After every deploy | L3 surface audit (all modes) + L5 unmatched errors |
| Daily | `full-audit.sh --skip-public` + `collect_errors.py --hours 24` |
| Weekly | `full-audit.sh` (full, incl. public) + outage drill (quiet window) + journey harness |
| Before release | all layers + security skill |

## Guardrails (learned the hard way)

- `.env` must be `www-data:www-data 640` — a root `cp/mv` breaks FPM with placeholder-secret 500s.
- Never multi-source `scp` into `frontend/` (it has clobbered `index.html`); use explicit paths.
- The outage drill blackholes NVIDIA for ALL traffic (~2 min) — run it in a quiet window.
- Never call billing mutations or key-reveal endpoints in sweeps; skip and note why.
- macOS `tar` adds `._*` AppleDouble files — use `COPYFILE_DISABLE=1` and clean after extraction.
- DNS email: SPF ✅ `_spf.mail.hostinger.com`, DMARC ✅ `p=none`, DKIM ✅ `hostingermail1._domainkey`
  (verified 2026-10-01). Consider raising DMARC to `quarantine` once volume is stable.

---

## 2026-10-03 reliability revamp — new invariants (supersedes older audit claims)

Deployed by the owner-authorized reliability audit + follow-up (`Plugsky/docs/reliability/*-2026-10-03.md`).
Audit these alongside the layers above; several supersede earlier checks:

1. **Class-preserving routing** — every served engine must sit inside the alias class range (`0→[0,0]`, `1→[0,1]`,
   `2→[2,3]`, `3→[3,3]`); verify via **`X-Plugsky-Served-Model`** on blocking AND streaming responses. Unknown engines
   (class −1) must never serve. Eligibility also gates capabilities + serving context (`candidateMatches`/`engineContext`).
2. **Structured streaming validated before delivery** — `SseCompletionGuard` assembles JSON/tool streams (forced choice,
   tool names, object args, `[DONE]` + visible answer) and only then delivers; text streams stay incremental and partial
   output is never replayed. Test that upstream invalid-JSON/tool fixtures cannot leak to clients.
3. **Truthful usage** — count outcomes with `UsageOutcomes` (one per user+endpoint+request_id; ok > error > fallback;
   rows without request_id individually). Raw `usage_events` sums overcount attempts; “0 HTTP 5xx” alone is NOT a health
   claim — the revamp found 11 builder + 27 scale real terminal errors/24h missed by that metric.
4. **Disconnect recording** — closing a live stream after first byte yields exactly one error outcome +
   `stream.client_disconnected`; completed streams are not double-written.
5. **`full-audit.sh` is a smoke tool, not release evidence** — use the hardened nonce / served-engine / SSE checks:
   `tools/audit/plugsky_audit.mjs`, `tests/Unit/Inference/*` (SseCompletionGuard, RoutingReliability, EquivalentRecovery,
   UsageOutcomes), `tests/Integration/InferenceStreamingTest.php`, plus the cross-plan structured-stream matrix.
6. **Policy wording**: paid plans keep all-model access; frontier may receive a professional engine under the published
   failover policy — exact 550B is not promised. `/v1/models` is public; revoked-key checks must hit protected chat (401).

## 2026-10-12 — tooling notes

- Admin `#customers` now pages: `GET /api/admin/customers?limit=&offset=` returns `self_serve_total` (was hardcoded
  LIMIT 200 of 2,000+). Page shows "Showing X of Y" + Load more.
- L5 filter list in full-audit.sh includes "Daily fair-use limit" (own-message; without it L5 false-fails).
- Battery must run with QA upgraded to admin+enterprise BEFORE creating the key (`prep_qa.sh`), else paid models
  return 400 "not available on your trial plan".

## 2026-11-06 — security layer is now mandatory

`bin/skills/run-all.sh` now runs **7 skills**, including `skill-security.sh`:

```bash
cd /var/www/plugsky/backend
bash bin/skills/run-all.sh
```

Layer 7 checks: admin password/OTP/2FA state, admin API-key hygiene, trusted devices, `@fldubai.com` scanner accounts, Playwright/browser service, sudo wrapper, MCP no-auth behavior, `/api/playground/browser` disabled, `/v1/payouts` absent, `.env`/status leakage, JWT length, live Stripe key warning, paid-lane policy.

Rules:

- Do **not** claim a full audit passed if `skill-security.sh` has a FAIL.
- `skill-journey.sh` must have **0 FAIL**. WARN from upstream capacity is not a code pass/fail; rerun when the provider wave clears and record both results.
- Paid acceptance is **WARN/Paused** while `PAID_LANES_ENABLED=false`; its score is printed but does not fail the code suite.
- The 2026-11-06 incident was an application-level admin/browser exposure, not a page-load bug; page-load audits alone would have missed it.

## Notes

Listed by Plugsky. Skill text and behaviour belong to the upstream author.
If you redistribute it, keep the upstream license and attribution.
