---
name: plugsky-error-triage
slug: plugsky--plugsky-error-triage
description: "Use when watching Plugsky production health \u2014 read error logs/digests, find user-facing failures, fix bugs before customers complain or ask for refunds. WHEN: error logs, errors, bugs, logs, triage, incident, users complaining, refund risk, 429, rate limit, failing requests, upstream errors, webhook failures, watch users, production health, error digest, fix errors."
collection: Plugsky
category: Platform Ops
license: Plugsky
source: https://plugsky.com/skills
install: "Download the Plugsky skills pack (see below)"
---

# plugsky-error-triage

Use when watching Plugsky production health — read error logs/digests, find user-facing failures, fix bugs before customers complain or ask for refunds. WHEN: error logs, errors, bugs, logs, triage, incident, users complaining, refund risk, 429, rate limit, failing requests, upstream errors, webhook failures, watch users, production health, error digest, fix errors.

## Install

```bash
Download the Plugsky skills pack (see below)
```

## Skill

# Plugsky Error Triage — find and fix production errors before users complain

Workflow for an AI agent to proactively monitor Plugsky, classify failures, and ship
solid fixes with low risk.

## 1. Collect the latest errors (30 seconds)

```bash
cd /Users/mmarhama/Documents/GitHub/DarkShark/Plugsky
python3 tools/monitoring/collect_errors.py --hours 24     # full day
python3 tools/monitoring/collect_errors.py --hours 2      # quick check
```

Outputs into `Plugsky/error-logs/<YYYY-MM-DD>/`:
| File | Contents |
|---|---|
| `digest.md` (also `error-logs/latest.md`) | human summary: usage failures, top users, real error messages, log signatures, 5xx |
| `usage-errors.json` | failures grouped by status+endpoint+models |
| `top-users.json` | per-user failure counts + opencode (paid) call counts |
| `error-messages.json` | real messages joined from backend logs via request_id |
| `app-errors.json` | normalized backend log signatures |
| `http-5xx.json` | nginx 5xx |

## 2. Classify before touching anything

| Signature | Meaning | Action |
|---|---|---|
| `Upstream returned HTTP 429` across ≥3 buckets | NVIDIA account capacity crunch | NOT a code bug. Check pressure smoothing + opencode caps; consider capacity work. Never "fix" by removing cooldowns. |
| `AuthException Invalid or revoked API key` from one IP, high volume | scanner / stale client | Confirm IP is not a customer (DB/nginx). Block IP, keep 4xx logs at info. |
| `RateLimitException` on one user, high volume | user's client retrying in a loop, or plan RPM | Check their volume vs plan; proactive outreach draft if abnormal. |
| `chat.second_provider.failed Insufficient account funds` | zen out of balance | Alert owner; opencode is last-resort only (`OPENCODE_LAST_RESORT_ONLY`). |
| `playground.error` / 500 on dashboard APIs | real bug | Reproduce, fix with guardrails below. |
| `stripe`/`webhook` errors | billing risk — refunds/complaints | Highest priority: verify with `bin/stripe_webhook_e2e.sh`; check `bin/reconcile-plans.php`. |
| `usage_events status=error` spike with one endpoint | regression after a deploy | Compare with last deploy time; roll back if needed. |

### 2b. Count outcomes correctly (2026-10-03 revamp — mandatory)

Raw `usage_events` rows are **attempts**, not requests. One subscriber request can write several `fallback` rows
plus one `ok`/`error` row. Always count with the shared dedup CTE (`src/Infrastructure/Database/UsageOutcomes.php`):
one outcome per `(user_id, endpoint, request_id)` — `ok` > `error` > `fallback`, rows without `request_id` counted
individually. `fallback` rows = "unresolved attempts" (client aborted / missing outcome), **not** customer-visible
failures. The dashboard Usage page uses the same rule (`success_count` / `unresolved_count`).

- Do not claim platform health from "0 HTTP 5xx" alone — the 2026-10-03 revamp found **11 builder + 27 scale real
  terminal errors/24h** that the old raw metric missed.
- New/extended log signals: `stream.client_disconnected` (client abort recorded once, attributed to served engine),
  `chat.second_provider.*` (rescue lanes; `rescue-provider/<base>` cooldowns on funds/401-403), and
  `chat.fallback.failed/ok` with `x_plugsky_provider`/`x_plugsky_model` context. The old opencode
  `opencode-nofunds.until` marker file is gone — funds failures now cool the provider 300s instead.
- When checking "did requests fail", join by `request_id` and prefer `X-Plugsky-Served-Model` / usage `status='ok'`
  over counting attempt rows.

## 3. Fix with guardrails (mandatory)

1. **Backup**: `cp file file.bak-$(date +%Y%m%d-%H%M%S)`.
2. **Patch minimally** — one concern per change.
3. **Lint**: `php -l <file>` (PHP) or `node --check` (JS).
4. **Apply**: `systemctl reload php8.4-fpm` (graceful — never restart during announced test windows).
5. **Smoke test immediately**:
   ```bash
   curl -sk -o /dev/null -w "healthz=%{http_code}\n" -H "Host: plugsky.com" https://127.0.0.1/healthz
   curl -sk -o /dev/null -w "models=%{http_code}\n"  -H "Host: plugsky.com" https://127.0.0.1/v1/models
   curl -sk -o /dev/null -w "login=%{http_code}\n" -X POST -H "Host: plugsky.com" -H "Content-Type: application/json" -d '{"email":"x@y.z","password":"x"}' https://127.0.0.1/api/auth/login
   ```
6. **Verify the error trend**: re-run the collector (`--hours 1`) after 15–30 min; the signature count must drop.
7. **Record it** in `Plugsky/error-logs/RESOLUTIONS.md` (date, signature, root cause, fix, files, verification).

## 4. Access reference

```bash
# Plugsky VPS
ssh -i ~/.ssh/plugsky_deploy root@72.61.81.91
# backend: /var/www/plugsky/backend   frontend: /var/www/plugsky/frontend
# logs: backend/storage/logs/{app,error,health,fallback-model,reconcile}.log
# DB: PostgreSQL 127.0.0.1:5433 plugsky (creds in backend/.env — never print them)
# Redis: 127.0.0.1:6390 (cooldowns: plugsky:cooldown:*, opencode caps: plugsky:oc:*)
# crons: health-check */5, collect-status */5, reconcile-plans */15
```

Useful queries:
```sql
-- failures last hour by endpoint
SELECT endpoint, model, status, count(*) FROM usage_events
WHERE created_at > now() - interval '1 hour' AND status IN ('error','rate_limited','blocked')
GROUP BY 1,2,3 ORDER BY 4 DESC;
-- paying users affected (refund risk)
SELECT u.email, u.plan, count(*) FROM usage_events e JOIN users u ON u.id=e.user_id
WHERE e.status='error' AND e.created_at > now() - interval '24 hours'
GROUP BY 1,2 ORDER BY 3 DESC;
```

## 5. Rules

- **Never** log or print secrets, API keys, or `.env` contents; keep `.env` `www-data:www-data 640`.
- Prefer `systemctl reload` over `restart`.
- Don't delete log lines to "fix" a digest — fix the cause or reduce noise at the source (see ErrorMiddleware expected-4xx handling).
- Upstream 429s are capacity, not bugs: check `UpstreamCooldown::coolingCount()`, pressure smoothing, and opencode caps before changing routing.
- Paying users with errors > their plan RPM → proactive outreach draft in `tools/audit/case-hy-20260924/replies.md`.

## 6. Deploy gotchas (learned the hard way)

- **macOS tar creates `._*` AppleDouble files** on the server (390 junk files once, 39 of them
  polluted `news-sitemap.xml`). Always deploy with `COPYFILE_DISABLE=1 tar czf …` and/or run
  `find /var/www/plugsky/frontend -name "._*" -delete` after extraction.
- **Never extract a bundle's `index.html` into the frontend root** — the articles index belongs at
  `articles/index.html`; the root `index.html` is the marketing homepage (it was clobbered once).
- After any frontend deploy: rebuild `sitemap.xml` / `tools-sitemap.xml` / `news-sitemap.xml`
  (tools must include all 145 pages), then run the link checker and a 200-check on every new URL.
- `nginx` clean URLs: canonical form is **no trailing slash** (`/articles/<slug>`); `/articles/<slug>/` 404s.

## 7. Fair-use controls (deployed 2026-09-26)

- **Free plan**: `FAIRUSE_FREE_DAILY=500` requests/day · **Trial**: `FAIRUSE_TRIAL_DAILY=2000` · paid = unlimited fair-use.
  Exceeding → `429` `rate_limit_error` "Daily fair-use limit reached…" with `Retry-After: 3600`.
- **Failure backoff (all plans)**: 8 consecutive upstream failures (`FAIRUSE_FAIL_THRESHOLD`) → 60s block
  (`FAIRUSE_BLOCK_SECONDS`) → `429` "Too many consecutive upstream failures… use exponential backoff".
  Redis keys: `plugsky:fu:block:<user>`, `plugsky:fu:fail:<user>`, `plugsky:fu:day:<user>:<date>`.
  Success resets the failure counter; cached responses and client-side errors do not consume budget.
- **Degraded workers**: upstreams returning "DEGRADED function cannot be invoked" are now marked cooling
  (UpstreamCooldown regex) so tool-call requests skip them instead of retrying every time.
- When a user's failure count explodes (e.g. 5k+/hour), check their client retry loop → send the outreach
  draft (replies.md) and let the backoff cool them; do not raise their limits.

## 8. nginx gotchas (learned 2026-09-26)

- **Never leave backups in `/etc/nginx/sites-enabled/`** — nginx includes `sites-enabled/*`, so a
  `plugsky.com.bak-*` file loads as a second server block → "conflicting server name" warnings and the
  wrong block may serve traffic. Backups go to `/etc/nginx/backups/`.
- **`try_files` + `rewrite` in the same location**: a `rewrite` directive resets `$1` captures, so
  `try_files /x/$1 ...` silently 404s. Use `try_files $uri $uri.html $uri/index.html =404;` instead.
- Nested path routes follow the /articles pattern:
  `location ~ ^/locations/(.+)$ { rewrite ^(/locations/.+)/$ $1 permanent; try_files $uri $uri.html $uri/index.html =404; }`
- After nginx edits: `nginx -t` → `systemctl reload nginx` → curl test both slash/no-slash forms.

## 9. Client-side error telemetry (added 2026-09-26)

Browser errors are captured automatically:
- **Endpoint**: `POST /api/telemetry/error` (public, throttled) → appends JSON lines to
  `backend/storage/logs/client-errors.log` (rotates at 5 MB to `.1`).
- **Reporters**: `articles/article.js` (all article/tool/location pages), `dashboard.html`, and the
  key landing pages (`/`, `/contact`, `/billing`, `/billing-success`, `/pricing`, `/free-trial`).
  Captures `window.onerror` + `unhandledrejection`, dedupes per message+source+line, capped payloads.
- **Collector**: `collect_errors.py` now writes `client-errors.json` and a "Client-side errors" digest
  section — check it first for SPA bugs (e.g. `openEnterpriseForm is not defined` came from a stale
  cached dashboard; the fix was a global function + `Cache-Control: no-store` on `/dashboard`).
- **Workflow**: run collector → group client errors by message+url → reproduce in Playwright on the
  affected page → fix with backups/lint → verify the error stops appearing in the log.


## Known bug classes & detection playbooks (added 2026-09-28)

Run these in order during any triage; each has a concrete query and a fix path.

### 1. Plan-sync mismatch — customer paid, API says free  (top support issue)
```sql
SELECT w.id, w.plan AS ws_plan, u.email, u.plan AS user_plan, s.plan AS sub_plan, s.status,
       s.stripe_price_id, s.current_period_end::date, s.canceled_at::date
  FROM workspaces w LEFT JOIN users u ON u.id=w.owner_id
  LEFT JOIN subscriptions s ON s.workspace_id=w.id
 WHERE w.plan = 'free' AND s.status IN ('active','trialing') AND s.plan <> 'free';
```
- Any row = a paying customer stuck on free. Verify the price maps to a plan (`STRIPE_PRICE_*` in `.env`),
  check `stripe_events` for `checkout.session.completed`/`subscription.created`, then run
  `php bin/reconcile-plans.php` and re-check. If the paid period already lapsed, apply a goodwill comp
  (`PlanOverrides::set` + set `workspaces.plan`/`users.plan`, see the support-tickets skill) and reply.
- Reconcile cron: `*/15 * * * * php bin/reconcile-plans.php` → `storage/logs/reconcile.log`
  (expect `N checked, N already correct, 0 fixed`; ignore the `sub_guard_*` fetch warning on ws 1).

### 2. Admin access missing (owner gets bounced from /admin)
```sql
SELECT id, email, role FROM users WHERE role='admin';
```
Only these can use the admin panel/API (admin.html redirects silently to `/dashboard?error=not_admin`).
Promote the owner/agent if missing; AuthMiddleware reads the role from the DB, so live tokens pick it up.

### 3. Support-ticket enum drift (statuses that can't be set / never close)
```sql
SELECT status, COUNT(*) FROM support_tickets GROUP BY status;
```
Valid enum: **open / awaiting / closed**. If code or UI offers `pending`/`resolved`, it's drift → fix the
whitelist (reply endpoint) and the UI select. Also check `resolved_at` is set on close / cleared on reopen.

### 4. Billing interval inversion (monthly ↔ annual)
- Verify Stripe prices: monthly `STRIPE_PRICE_BUILDER`, annual `STRIPE_PRICE_BUILDER_ANNUAL`
  (Stripe `Price::retrieve` → unit_amount + recurring.interval).
- Frontend signal: `dashboard.html` CTA handler reads the annual button's active state
  (`style.background !== '' && !classList.contains('ghost')`). If that check is inverted or missing,
  customers choosing monthly land on the annual price → immediately fix + check recent subs with
  `interval='annual'` for complaints, and check `subscriptions.status='pending'` (never charged).

### 5. API key friction (create/limits)
```sql
SELECT u.email, COUNT(*) FILTER (WHERE k.revoked_at IS NULL) AS active_keys
  FROM users u JOIN api_keys k ON k.user_id=u.id GROUP BY u.email ORDER BY active_keys DESC LIMIT 10;
```
Policy: **unlimited keys** per account (scopes still filtered by plan). Revoke is instant. If the API returns
`plan_limit`, the cap code wasn't updated (PlanLimits `api_keys`). Creation test: `POST /api/keys`.

### 6. Tickets aging without an answer
```sql
SELECT id, reference, subject, status, created_at::date, updated_at::date
  FROM support_tickets WHERE status='open' AND updated_at < NOW() - INTERVAL '3 days'
 ORDER BY updated_at;
```
These need a reply (support-tickets skill: investigate → reply → awaiting).

## 2026-11-06 — success vs 429 vs 5xx (mandatory correction)

`error-watch.request_outcomes.ok` was calculated as `total - terminal_5xx`, so **429 responses were counted as “ok.”** On 2026-11-06 a 10-minute window showed 14,323 “ok” while 11,061 requests were 429. A low 5xx rate can therefore hide a rate-limit storm.

Use/require these separate signals:

- `success_rate` = 2xx / total (target >99% paid, >99.5% free).
- `rate_limit_rate` = 429 / total; page when >20% or when `signatures.rate_limit > 1000` per window.
- `rate_limit` by reason: `plan_rpm_limit`, `plan_concurrency_limit`, `account_cooldown`, `daily_request_cap`, `daily_rescue_budget_exceeded`, `fair_use_heavy_agent`, `free_plan_limit`.
- `terminal_5xx_rate` remains the code/outage signal; do not let a 429 flood make it look green.

New signatures that must be reviewed together:

- `cascade_budget_exhausted` / `cascade_one_try_zero` — a paid cascade failed after one engine attempt.
- `stream_attempt_failed` / `stream_exhausted` — streaming lane/attempt failures.
- `free_phase_expired` / `free_phase_starved` — tool rescue reserved time for a paused paid tail; `free_phase_starved > 0` pages.
- `rate_limit` — high count now pages; inspect the top user/IP and retry-storm breaker.

Current incident baseline (2026-11-06, 10m): 14,378 final requests, 55 paid 5xx, 0 free-plan 5xx, **11,061 429s**. 1h paid terminal-5xx was **7.25%** before the paid-storm fixes; 24h was ~2.46%. Do not call this smooth until the 429 rate is under control.

When a window is “5xx green but 429 red,” classify as **capacity/load-shedding**, not a code pass. Check:

```sql
SELECT error_code, count(*) FROM request_logs
WHERE created_at > now() - interval '15 minutes' AND status=429
GROUP BY 1 ORDER BY 2 DESC;
```
(If request logs are only in `app.log`, use the `request` lines; `error-watch` already groups `plan_rejections`.)

Paid acceptance while `PAID_LANES_ENABLED=false` is a **WARN/Paused**, not a code failure. The skill suite prints its score/failed checks; use journey/API/security skills for code coverage.

### r44 deployed — verify these fields, not just the 5xx rate

`php bin/error-watch.php --window=30 --quiet` now returns:

```json
"request_outcomes": {
  "total": 10853, "ok": 7650, "rate_limited": 2882,
  "client_error": 248, "error": 71,
  "success_rate": 0.704874, "rate_limit_rate": 0.265549,
  "paid_total": 3083, "paid_error": 71
}
```

- `ok` is **2xx only**. Never describe a window as healthy from `ok` alone; require `success_rate` and `rate_limit_rate`.
- The SLO 1-hour window now ignores a 60-minute record older than 20 minutes and merges four non-overlapping 15-minute records. Always check `window_note` and `windows.1h.window_minutes` before quoting the rate.
- r44 verification: 1h = 22,638 requests, paid 120/6,855 = **1.75%**, free 20/15,783 = **0.13%**, status `ok`. The previous file reported a stale 14:31 snapshot (7.25%) at 20:31.
- 30m watch: 11,470 requests, 3,300 429 (28.8%), 102 5xx (82 paid, 20 free), paid final 5xx **2.43%**, paid-provider spend **$0**. A 70% success rate in this window is a load-shedding incident, not a green service.

## 2026-11-07 — billing reconcile + rate-limit storm breaker (r45)

### Billing plan revert (PLG-147)

A Starter → Builder upgrade was correctly synced by the webhook at 23:59 UTC, then the `reconcile-plans.php` cron reverted it to Starter at 00:00 because it read stale `metadata.plan_slug` before the current Stripe price. The fix makes the **current price** the source of truth in reconcile, matching the webhook. `skill-errors.php` now checks every active subscription's `plan` against its `stripe_price_id`; any mismatch is a FAIL.

Verification after the fix: workspace `1629009` restored to Builder; `PlanCaps::for()` returns requests=0, tokens=0, concurrent=24, rpm=1500, max_input=262144, max_output=32768, tpm=0. A full dry-run now reports 65 checked, 64 correct, 0 changes.

### Free-plan retry loop

User `1745` (free, one Codex key) sent ~338 requests/minute from `85.239.245.74`: 310 malformed `400 invalid_request`, then **3,071 `plan_rpm_limit` 429s** in 10 hours. That was over half of the 5,529 total 429s in the 21:22–07:22 window. It stopped around 22:38; no traffic in the last hour.

r45 adds a bounded storm breaker in `RateLimiter`:

- After `RATE_LIMIT_STORM_THRESHOLD` (default 20) RPM/TPM limit hits in the block window, the user/key gets a `429 rate_limit_storm` with `Retry-After` = `RATE_LIMIT_STORM_BLOCK_SECONDS` (default 300).
- IP scope is never blocked (shared NAT/carrier safety).
- The strike counter resets after the block, so a recovered client gets a fresh window.
- The error code is distinct from `plan_rpm_limit`/`account_cooldown`; page on it if it appears for multiple users.

Controlled test: use the acceptance user, set a temporary Redis RPM override, send a burst, confirm `rate_limit_storm` and the block key, then remove the override and all `plugsky:rl:storm:*:<user>` keys.

### 2026-11-07 10h baseline (21:22–07:22 UTC)

- 167,521 final requests; 5,529 429 (3.3%); 1,033 5xx (0.62%).
- Paid: 8,163 requests; 6,673 2xx (81.75%); 460 429 (5.64%); **1,018 5xx (12.47%)**.
- Starter: 5,875 / 4,919 2xx (83.7%) / 352 429 / 599 5xx (10.2%).
- Hobby: 2,171 / 1,676 2xx (77.2%) / 108 429 / 380 5xx (17.5%).
- Builder: 117 / 78 2xx / 39 5xx (33.3%).
- Free: 6,327 / 724 2xx (11.4%) / 5,068 429 (80.1%) / 10 5xx.
- Top paid 5xx: 6402 (345), 1801 (311), 1726 (230), 6612 (53 before fix), 1369 (39).
- 5xx codes: upstream_unavailable 925; upstream_timeout 106.
- Top paths: `/v1/plugsky/usage` 97,276; `/api/nodes/jobs/claim` 28,912; `/v1/chat/completions` 14,619; `/mcp` 13,305.
- The paid 503s are long streaming tool/agent requests; p95 ~32–40s. Capacity, not a plan cap.

### 2026-11-07 30m live watch (07:19:47–07:49:47 UTC)

- 9,031 requests; 8,658 2xx (95.9%); 136 429 (1.5%); **52 5xx (0.58%)**; 139 404, 19 401, 11 400.
- Paid: 332 requests, **52 5xx = 15.66%**; free 5xx **0**.
- 429s: `free_plan_limit` 131 (user 953), `account_cooldown` 5. The 1745 retry storm is gone.
- Paid 5xx: **1726 (25), 6402 (16), 1801 (11)**; Nemotron Ultra, MiniMax M3.1, Mistral Large 4; all upstream capacity.
- Paid attempts: 15m 207 / 71 ok / 106 fallback / 30 errors (14.49%); 60m 884 / 351 / 439 / 94 (10.63%).
- Spend: **0 metered / $0**; last-resort 48 / $10.37 estimated; GPU 1.
- SLO 1h at close: paid 5xx **15.72%** (breach). Last hour after the watch: 112/449 = **24.94%**, concentrated in 6402/1801/1726/1369. Do not call this recovered.
- Security: 27/2/0 (TOTP + Stripe key WARNs); legacy caps 21/5/1, 0 violations; no new admin/scanner accounts.
- r45 suite: ALL SKILLS PASS; journey 13/13; API 34/1/0; errors 5/4/0; paid acceptance 95/100 WARN/Paused (`large_context` only).

### What still cannot be fixed in code

The paid 503s are free-tool-lane exhaustion while paid fallback is paused. The only real fix is restoring paid capacity (approved budget + positive provider balances) or adding a funded tool lane. Do not "fix" this by removing cooldowns, silently downgrading a paid alias to a free entry model, or raising plan concurrency. Keep paid lanes off until the canary gate passes.

## Notes

Listed by Plugsky. Skill text and behaviour belong to the upstream author.
If you redistribute it, keep the upstream license and attribution.
