Windsurf rules

1---
2name: harden-vps
3model: haiku
4description: "Harden a production Linux VPS for your application across three layers — application (systemd hardening, monitoring alerts, backup automation), Ubuntu OS (UFW firewall, fail2ban SSH, unattended-upgrades, SSH hardening), and VPS provider (health checks, daily backups, monthly snapshots). Use when the user wants to secure a production server, harden a VPS, audit server security, or mentions production hardening, VPS security, or harden the server."
5---
6 
7# Harden VPS
8 
9Three-layer production hardening for a self-hosted app on any VPS. Each layer independently verifiable. Apply OS first (firewall blocks attacks immediately), then your application (alerts + backups), then your VPS provider (snapshots). See [REFERENCE.md](REFERENCE.md) for full script bodies, systemd unit template, and gotchas.
10 
11## Quick start
12 
13SSH as root into the VPS. Find credentials in the your VPS provider Customer Control Panel.
14 
15> **HARD GATE** — Run `ufw status` first. No firewall = layer 1 takes priority over everything.
16 
17## Layer 1 — Ubuntu OS
18 
19```bash
20# UFW
21ufw default deny incoming && ufw default allow outgoing
22ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw enable
23# → verify: ufw status | grep -q active
24 
25# fail2ban
26apt install -y fail2ban
27# Configure /etc/fail2ban/jail.local: sshd, maxretry=3, bantime=3600, findtime=600
28systemctl restart fail2ban
29# → verify: fail2ban-client status sshd
30 
31# unattended-upgrades
32apt install -y unattended-upgrades && dpkg-reconfigure -plow unattended-upgrades
33# → verify: systemctl is-active unattended-upgrades | grep -q active
34 
35# SSH: PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes
36# → verify: sshd -T | grep -E 'permitrootlogin no|passwordauthentication no'
37 
38# Deploy healthcheck.sh → /opt/your-app/scripts/healthcheck.sh
39# Crontab: */5 * * * * /opt/your-app/scripts/healthcheck.sh
40```
41 
42## Layer 2 — applicationlication
43 
44```bash
45# systemd: User=your-app, NoNewPrivileges=yes, ProtectSystem=full,
46#   ProtectKernelTunables=yes, ProtectKernelModules=yes,
47#   ProtectControlGroups=yes, RestrictAddressFamilies=AF_INET AF_INET6,
48#   RestrictRealtime=yes, PrivateTmp=yes, LimitNOFILE=65536
49# → verify: systemctl show your-app -p NoNewPrivileges -p ProtectSystem -p User
50 
51# Alerts (your application requires auth; insert via SQLite)
52sqlite3 /opt/your-app/data/your-app.db "
53INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
54VALUES ('a1','Disk >80%','disk_used_percent',80,'gt',1,300);
55INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
56VALUES ('a2','CPU >90%','cpu_percent',90,'gt',1,60);
57INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
58VALUES ('a3','RAM >85%','mem_used_percent',85,'gt',1,120);
59"
60systemctl restart your-app
61 
62# Backup crontab (root):
63# 0 2 * * * cp /opt/your-app/data/your-app.db /backup/your-app-$(date +\%Y\%m\%d).db
64# 0 3 * * * find /backup/ -name "your-app-*.db" -mtime +90 -delete
65```
66 
67## Layer 3 — VPS provider
68 
69```bash
70# cntb CLI
71curl -sL "$(curl -sL https://api.github.com/repos/contabo/cntb/releases/latest \
72  | grep browser_download_url.*linux_amd64.tar.gz | head -1 | cut -d'"' -f4)" \
73  | tar xz -C /usr/local/bin
74 
75# Snapshot script → /opt/your-app/scripts/contabo-snapshot.sh (reads from /opt/your-app/.env)
76# Credentials as env vars in /opt/your-app/.env (deployed by GitHub Actions):
77#   CONTABO_CLIENT_ID, CONTABO_CLIENT_SECRET, CONTABO_API_USER, CONTABO_API_PASSWORD
78# Crontab: 0 4 1 * * /opt/your-app/scripts/contabo-snapshot.sh
79 
80# > HARD GATE — Snapshot cron silently fails until env vars are set in .env.
81# Credentials source: your VPS provider Customer Panel → API Details.
82# Local dev: add to .envrc. Production: GitHub Secrets → deploy → /opt/your-app/.env
83```
84 
85## CRITICAL GOTCHAS
86 
871. **Shell escaping in Orca terminals:** `$VAR`, `$(…)`, and `%` get eaten by the local shell. Always use base64: `echo '<base64>' | base64 -d > script.sh`
882. **Crontab `%`:** cron interprets `%` as newline. Escape as `\%` in `$(date +\%Y\%m\%d)`
893. **fail2ban exit 255:** means a jail references a missing log file. Remove the broken jail, restart.
904. **your application alerts need auth:** POST to `/api/monitoring/alerts` requires Bearer token. Workaround: insert directly into SQLite, then restart your application.
91 
92## Verify all 8 gates
93 
94```bash
95ufw status|grep -q active||echo FAIL:ufw
96fail2ban-client status sshd&gt;/dev/null 2&gt;&amp;1||echo FAIL:fail2ban
97systemctl is-active unattended-upgrades|grep -q active||echo FAIL:unattended
98sshd -T|grep -q 'permitrootlogin no'||echo FAIL:sshd
99systemctl show your-app -p NoNewPrivileges|grep -q yes||echo FAIL:systemd
100systemctl is-active your-app|grep -q active||echo FAIL:your-app
101sqlite3 /opt/your-app/data/your-app.db &quot;SELECT count(*) FROM monitoring_alerts&quot;|grep -q 3||echo FAIL:alerts
102crontab -l|grep -q healthcheck&&crontab -l|grep -q your-app.db&&crontab -l|grep -q contabo-snapshot||echo FAIL:crontab
103echo ALL 8 GATES PASSED
104```
105 
106→ verify: # requires VPS SSH — run the 8-gate one-liner on the VPS manually
107 
108---
109 
110# Harden VPS — Reference
111 
112## Health check script
113 
114Deploy to `/opt/your-app/scripts/healthcheck.sh`:
115 
116```bash
117#!/bin/bash
118LOG=/var/log/your-app/health.log
119mkdir -p /var/log/your-app
120DISK=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
121RAM=$(free | awk '/Mem:/ {printf "%.0f", $3/$2*100}')
122NOW=$(date -Iseconds)
123[ &quot;$DISK&quot; -gt 85 ] && echo &quot;$NOW ALERT: Disk ${DISK}%&quot; &gt;&gt; &quot;$LOG&quot;
124[ &quot;$RAM&quot; -gt 90 ]   && echo &quot;$NOW ALERT: RAM ${RAM}%&quot; &gt;&gt; &quot;$LOG&quot;
125systemctl is-active --quiet your-app || { echo &quot;$NOW ALERT: your application DOWN&quot; &gt;&gt; &quot;$LOG&quot;; systemctl restart your-app; }
126[ -f /var/run/reboot-required ] && echo &quot;$NOW WARN: kernel reboot pending&quot; &gt;&gt; &quot;$LOG&quot;
127```
128 
129Make executable: `chmod +x /opt/your-app/scripts/healthcheck.sh`
130 
131## your application systemd unit
132 
133Full hardened unit at `/etc/systemd/system/your-app.service`:
134 
135```ini
136[Unit]
137Description=your application BaaS Platform
138Documentation=https://github.com/example-org/example-app
139After=network.target caddy.service
140Wants=caddy.service
141
142[Service]
143Type=simple
144User=your-app
145Group=your-app
146WorkingDirectory=/opt/your-app
147ExecStart=/opt/your-app/bin/your-app serve \
148    --port 8080 \
149    --db /opt/your-app/data/your-app.db \
150    --sites-domain <your-domain.example.com>
151EnvironmentFile=-/opt/your-app/.env
152Environment=APP_HOME=/opt/your-app
153Environment=HOME=/opt/your-app
154Environment=NPM_CONFIG_CACHE=/opt/your-app/.npm
155Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
156Restart=always
157RestartSec=5
158StartLimitIntervalSec=60
159StartLimitBurst=3
160TimeoutStopSec=30
161KillSignal=SIGTERM
162NoNewPrivileges=yes
163PrivateTmp=yes
164ProtectSystem=full
165ProtectHome=yes
166ProtectKernelTunables=yes
167ProtectKernelModules=yes
168ProtectControlGroups=yes
169RestrictAddressFamilies=AF_INET AF_INET6
170RestrictRealtime=yes
171ReadWritePaths=/opt/your-app/data
172ReadWritePaths=/opt/your-app/backups
173ReadWritePaths=/opt/your-app/secrets
174ReadWritePaths=/opt/your-app/.npm
175ReadWritePaths=/opt/your-app/logs
176LimitNOFILE=65536
177
178[Install]
179WantedBy=multi-user.target
180```
181 
182## fail2ban jail.local
183 
184```ini
185[sshd]
186enabled = true
187port = ssh
188filter = sshd
189logpath = /var/log/auth.log
190maxretry = 3
191bantime = 3600
192findtime = 600
193```
194 
195Note: your application auth jail deferred — your application logs to journald, not a file. To enable, use `backend = systemd` with a journald filter.
196 
197## provider-snapshot.sh — worked example (Contabo API)
198 
199> Snapshot APIs are provider-specific. This is one worked example, not a
200> bigpowers default; swap the auth and endpoint calls for your own provider.
201 
202Deploy to `/opt/your-app/scripts/contabo-snapshot.sh`. Reads credentials from `/opt/your-app/.env` (same file used by your application systemd unit):
203 
204```bash
205#!/bin/bash
206set -e
207LOG=/var/log/your-app/snapshot.log
208ENVFILE=/opt/your-app/.env
209INSTANCE_ID=&lt;your-contabo-instance-id&gt;
210 
211[ -f &quot;$ENVFILE&quot; ] || { echo &quot;$(date -Iseconds) ERROR: $ENVFILE missing&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
212set -a; source &quot;$ENVFILE&quot;; set +a
213 
214[ -z &quot;$CONTABO_CLIENT_ID&quot; ] && { echo &quot;$(date -Iseconds) ERROR: CONTABO_CLIENT_ID not set&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
215 
216TOKEN=$(curl -s -d &quot;client_id=$CONTABO_CLIENT_ID&quot; \
217  -d &quot;client_secret=$CONTABO_CLIENT_SECRET&quot; \
218  --data-urlencode &quot;username=$CONTABO_API_USER&quot; \
219  --data-urlencode &quot;password=$CONTABO_API_PASSWORD&quot; \
220  -d 'grant_type=password' \
221  'https://auth.contabo.com/auth/realms/contabo/protocol/openid-connect/token' \
222  | jq -r '.access_token')
223 
224[ -z &quot;$TOKEN&quot; ] || [ &quot;$TOKEN&quot; = &quot;null&quot; ] && { echo &quot;$(date -Iseconds) ERROR: auth failed&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
225
226curl -s -X POST \
227  -H &quot;Authorization: Bearer $TOKEN&quot; \
228  -H &quot;Content-Type: application/json&quot; \
229  -H &quot;x-request-id: $(uuidgen 2&gt;/dev/null || echo $RANDOM)&quot; \
230  &quot;https://api.contabo.com/v1/compute/instances/${INSTANCE_ID}/snapshots&quot; \
231  | tee -a &quot;$LOG&quot;
232
233echo &gt;&gt; &quot;$LOG&quot;
234echo &quot;$(date -Iseconds) Snapshot requested for $INSTANCE_ID&quot; &gt;&gt; &quot;$LOG&quot;
235```
236 
237## your VPS provider credentials
238 
239Add to `/opt/your-app/.env` (same file used by your application systemd unit, deployed by GitHub Actions):
240 
241```
242CONTABO_CLIENT_ID=
243CONTABO_CLIENT_SECRET=
244CONTABO_API_USER=
245CONTABO_API_PASSWORD=
246```
247 
248Source: your VPS provider Customer Control Panel → API Details. ClientId and ClientSecret are generated there. API User is your email. API Password is set separately in the panel.
249 
250**Pipeline:** GitHub Secrets → deploy workflow → `/opt/your-app/.env` on VPS. For local dev (`cntb get instances`), add to `.envrc`.
251 
252## your VPS provider instance info template
253 
254Fill in your instance details from the your VPS provider Customer Control Panel:
255 
256```
257IP:           <your-instance-ip>
258IPv6:         <your-instance-ipv6>
259Region:       <your-region>
260OS:           <your-os-version>
261Disk:         <your-disk-size>
262Default user: root
263Customer ID:  <your-customer-id>
264```
265 
266## your application monitoring alert SQL
267 
268```sql
269INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
270VALUES ('alert-001', 'VPS Disk above 80%', 'disk_used_percent', 80, 'gt', 1, 300);
271 
272INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
273VALUES ('alert-002', 'VPS CPU above 90%', 'cpu_percent', 90, 'gt', 1, 60);
274 
275INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
276VALUES ('alert-003', 'VPS RAM above 85%', 'mem_used_percent', 85, 'gt', 1, 120);
277```
278 
279Alerts are loaded at your application startup. Restart with `systemctl restart your-app` after inserting.
280 
281## Base64 encoding workaround
282 
283When sending scripts through Orca terminal `--text`, the local bash shell interprets `$`, `(`, and `%`. Encode locally and decode remotely:
284 
285```bash
286# Local
287cat script.sh | base64
288 
289# Remote terminal
290echo '<base64-output>' | base64 -d &gt; script.sh
291```
292