Cursor rule

1---
2description: "Harden a production Linux VPS for your application across three layers — application (systemd hardening, monitoring alerts, backup automation), Ubuntu OS (UFW firewall, fail2ban SSH, unattended-upgrades, SSH hardening), and VPS provider (health checks, daily backups, monthly snapshots). Use when the user wants to secure a production server, harden a VPS, audit server security, or mentions production hardening, VPS security, or harden the server."
3alwaysApply: false
4---
5 
6# Harden VPS
7 
8Three-layer production hardening for a self-hosted app on any VPS. Each layer independently verifiable. Apply OS first (firewall blocks attacks immediately), then your application (alerts + backups), then your VPS provider (snapshots). See [REFERENCE.md](REFERENCE.md) for full script bodies, systemd unit template, and gotchas.
9 
10## Quick start
11 
12SSH as root into the VPS. Find credentials in the your VPS provider Customer Control Panel.
13 
14> **HARD GATE** — Run `ufw status` first. No firewall = layer 1 takes priority over everything.
15 
16## Layer 1 — Ubuntu OS
17 
18```bash
19# UFW
20ufw default deny incoming && ufw default allow outgoing
21ufw allow 22/tcp && ufw allow 80/tcp && ufw allow 443/tcp && ufw enable
22# → verify: ufw status | grep -q active
23 
24# fail2ban
25apt install -y fail2ban
26# Configure /etc/fail2ban/jail.local: sshd, maxretry=3, bantime=3600, findtime=600
27systemctl restart fail2ban
28# → verify: fail2ban-client status sshd
29 
30# unattended-upgrades
31apt install -y unattended-upgrades && dpkg-reconfigure -plow unattended-upgrades
32# → verify: systemctl is-active unattended-upgrades | grep -q active
33 
34# SSH: PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes
35# → verify: sshd -T | grep -E 'permitrootlogin no|passwordauthentication no'
36 
37# Deploy healthcheck.sh → /opt/your-app/scripts/healthcheck.sh
38# Crontab: */5 * * * * /opt/your-app/scripts/healthcheck.sh
39```
40 
41## Layer 2 — applicationlication
42 
43```bash
44# systemd: User=your-app, NoNewPrivileges=yes, ProtectSystem=full,
45#   ProtectKernelTunables=yes, ProtectKernelModules=yes,
46#   ProtectControlGroups=yes, RestrictAddressFamilies=AF_INET AF_INET6,
47#   RestrictRealtime=yes, PrivateTmp=yes, LimitNOFILE=65536
48# → verify: systemctl show your-app -p NoNewPrivileges -p ProtectSystem -p User
49 
50# Alerts (your application requires auth; insert via SQLite)
51sqlite3 /opt/your-app/data/your-app.db "
52INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
53VALUES ('a1','Disk >80%','disk_used_percent',80,'gt',1,300);
54INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
55VALUES ('a2','CPU >90%','cpu_percent',90,'gt',1,60);
56INSERT INTO monitoring_alerts (id,name,metric,threshold,operator,enabled,duration_seconds)
57VALUES ('a3','RAM >85%','mem_used_percent',85,'gt',1,120);
58"
59systemctl restart your-app
60 
61# Backup crontab (root):
62# 0 2 * * * cp /opt/your-app/data/your-app.db /backup/your-app-$(date +\%Y\%m\%d).db
63# 0 3 * * * find /backup/ -name "your-app-*.db" -mtime +90 -delete
64```
65 
66## Layer 3 — VPS provider
67 
68```bash
69# cntb CLI
70curl -sL "$(curl -sL https://api.github.com/repos/contabo/cntb/releases/latest \
71  | grep browser_download_url.*linux_amd64.tar.gz | head -1 | cut -d'"' -f4)" \
72  | tar xz -C /usr/local/bin
73 
74# Snapshot script → /opt/your-app/scripts/contabo-snapshot.sh (reads from /opt/your-app/.env)
75# Credentials as env vars in /opt/your-app/.env (deployed by GitHub Actions):
76#   CONTABO_CLIENT_ID, CONTABO_CLIENT_SECRET, CONTABO_API_USER, CONTABO_API_PASSWORD
77# Crontab: 0 4 1 * * /opt/your-app/scripts/contabo-snapshot.sh
78 
79# > HARD GATE — Snapshot cron silently fails until env vars are set in .env.
80# Credentials source: your VPS provider Customer Panel → API Details.
81# Local dev: add to .envrc. Production: GitHub Secrets → deploy → /opt/your-app/.env
82```
83 
84## CRITICAL GOTCHAS
85 
861. **Shell escaping in Orca terminals:** `$VAR`, `$(…)`, and `%` get eaten by the local shell. Always use base64: `echo '<base64>' | base64 -d > script.sh`
872. **Crontab `%`:** cron interprets `%` as newline. Escape as `\%` in `$(date +\%Y\%m\%d)`
883. **fail2ban exit 255:** means a jail references a missing log file. Remove the broken jail, restart.
894. **your application alerts need auth:** POST to `/api/monitoring/alerts` requires Bearer token. Workaround: insert directly into SQLite, then restart your application.
90 
91## Verify all 8 gates
92 
93```bash
94ufw status|grep -q active||echo FAIL:ufw
95fail2ban-client status sshd&gt;/dev/null 2&gt;&amp;1||echo FAIL:fail2ban
96systemctl is-active unattended-upgrades|grep -q active||echo FAIL:unattended
97sshd -T|grep -q 'permitrootlogin no'||echo FAIL:sshd
98systemctl show your-app -p NoNewPrivileges|grep -q yes||echo FAIL:systemd
99systemctl is-active your-app|grep -q active||echo FAIL:your-app
100sqlite3 /opt/your-app/data/your-app.db &quot;SELECT count(*) FROM monitoring_alerts&quot;|grep -q 3||echo FAIL:alerts
101crontab -l|grep -q healthcheck&&crontab -l|grep -q your-app.db&&crontab -l|grep -q contabo-snapshot||echo FAIL:crontab
102echo ALL 8 GATES PASSED
103```
104 
105→ verify: # requires VPS SSH — run the 8-gate one-liner on the VPS manually
106 
107---
108 
109# Harden VPS — Reference
110 
111## Health check script
112 
113Deploy to `/opt/your-app/scripts/healthcheck.sh`:
114 
115```bash
116#!/bin/bash
117LOG=/var/log/your-app/health.log
118mkdir -p /var/log/your-app
119DISK=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
120RAM=$(free | awk '/Mem:/ {printf "%.0f", $3/$2*100}')
121NOW=$(date -Iseconds)
122[ &quot;$DISK&quot; -gt 85 ] && echo &quot;$NOW ALERT: Disk ${DISK}%&quot; &gt;&gt; &quot;$LOG&quot;
123[ &quot;$RAM&quot; -gt 90 ]   && echo &quot;$NOW ALERT: RAM ${RAM}%&quot; &gt;&gt; &quot;$LOG&quot;
124systemctl is-active --quiet your-app || { echo &quot;$NOW ALERT: your application DOWN&quot; &gt;&gt; &quot;$LOG&quot;; systemctl restart your-app; }
125[ -f /var/run/reboot-required ] && echo &quot;$NOW WARN: kernel reboot pending&quot; &gt;&gt; &quot;$LOG&quot;
126```
127 
128Make executable: `chmod +x /opt/your-app/scripts/healthcheck.sh`
129 
130## your application systemd unit
131 
132Full hardened unit at `/etc/systemd/system/your-app.service`:
133 
134```ini
135[Unit]
136Description=your application BaaS Platform
137Documentation=https://github.com/example-org/example-app
138After=network.target caddy.service
139Wants=caddy.service
140
141[Service]
142Type=simple
143User=your-app
144Group=your-app
145WorkingDirectory=/opt/your-app
146ExecStart=/opt/your-app/bin/your-app serve \
147    --port 8080 \
148    --db /opt/your-app/data/your-app.db \
149    --sites-domain <your-domain.example.com>
150EnvironmentFile=-/opt/your-app/.env
151Environment=APP_HOME=/opt/your-app
152Environment=HOME=/opt/your-app
153Environment=NPM_CONFIG_CACHE=/opt/your-app/.npm
154Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
155Restart=always
156RestartSec=5
157StartLimitIntervalSec=60
158StartLimitBurst=3
159TimeoutStopSec=30
160KillSignal=SIGTERM
161NoNewPrivileges=yes
162PrivateTmp=yes
163ProtectSystem=full
164ProtectHome=yes
165ProtectKernelTunables=yes
166ProtectKernelModules=yes
167ProtectControlGroups=yes
168RestrictAddressFamilies=AF_INET AF_INET6
169RestrictRealtime=yes
170ReadWritePaths=/opt/your-app/data
171ReadWritePaths=/opt/your-app/backups
172ReadWritePaths=/opt/your-app/secrets
173ReadWritePaths=/opt/your-app/.npm
174ReadWritePaths=/opt/your-app/logs
175LimitNOFILE=65536
176
177[Install]
178WantedBy=multi-user.target
179```
180 
181## fail2ban jail.local
182 
183```ini
184[sshd]
185enabled = true
186port = ssh
187filter = sshd
188logpath = /var/log/auth.log
189maxretry = 3
190bantime = 3600
191findtime = 600
192```
193 
194Note: your application auth jail deferred — your application logs to journald, not a file. To enable, use `backend = systemd` with a journald filter.
195 
196## provider-snapshot.sh — worked example (Contabo API)
197 
198> Snapshot APIs are provider-specific. This is one worked example, not a
199> bigpowers default; swap the auth and endpoint calls for your own provider.
200 
201Deploy to `/opt/your-app/scripts/contabo-snapshot.sh`. Reads credentials from `/opt/your-app/.env` (same file used by your application systemd unit):
202 
203```bash
204#!/bin/bash
205set -e
206LOG=/var/log/your-app/snapshot.log
207ENVFILE=/opt/your-app/.env
208INSTANCE_ID=&lt;your-contabo-instance-id&gt;
209 
210[ -f &quot;$ENVFILE&quot; ] || { echo &quot;$(date -Iseconds) ERROR: $ENVFILE missing&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
211set -a; source &quot;$ENVFILE&quot;; set +a
212 
213[ -z &quot;$CONTABO_CLIENT_ID&quot; ] && { echo &quot;$(date -Iseconds) ERROR: CONTABO_CLIENT_ID not set&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
214 
215TOKEN=$(curl -s -d &quot;client_id=$CONTABO_CLIENT_ID&quot; \
216  -d &quot;client_secret=$CONTABO_CLIENT_SECRET&quot; \
217  --data-urlencode &quot;username=$CONTABO_API_USER&quot; \
218  --data-urlencode &quot;password=$CONTABO_API_PASSWORD&quot; \
219  -d 'grant_type=password' \
220  'https://auth.contabo.com/auth/realms/contabo/protocol/openid-connect/token' \
221  | jq -r '.access_token')
222 
223[ -z &quot;$TOKEN&quot; ] || [ &quot;$TOKEN&quot; = &quot;null&quot; ] && { echo &quot;$(date -Iseconds) ERROR: auth failed&quot; &gt;&gt; &quot;$LOG&quot;; exit 1; }
224
225curl -s -X POST \
226  -H &quot;Authorization: Bearer $TOKEN&quot; \
227  -H &quot;Content-Type: application/json&quot; \
228  -H &quot;x-request-id: $(uuidgen 2&gt;/dev/null || echo $RANDOM)&quot; \
229  &quot;https://api.contabo.com/v1/compute/instances/${INSTANCE_ID}/snapshots&quot; \
230  | tee -a &quot;$LOG&quot;
231
232echo &gt;&gt; &quot;$LOG&quot;
233echo &quot;$(date -Iseconds) Snapshot requested for $INSTANCE_ID&quot; &gt;&gt; &quot;$LOG&quot;
234```
235 
236## your VPS provider credentials
237 
238Add to `/opt/your-app/.env` (same file used by your application systemd unit, deployed by GitHub Actions):
239 
240```
241CONTABO_CLIENT_ID=
242CONTABO_CLIENT_SECRET=
243CONTABO_API_USER=
244CONTABO_API_PASSWORD=
245```
246 
247Source: your VPS provider Customer Control Panel → API Details. ClientId and ClientSecret are generated there. API User is your email. API Password is set separately in the panel.
248 
249**Pipeline:** GitHub Secrets → deploy workflow → `/opt/your-app/.env` on VPS. For local dev (`cntb get instances`), add to `.envrc`.
250 
251## your VPS provider instance info template
252 
253Fill in your instance details from the your VPS provider Customer Control Panel:
254 
255```
256IP:           <your-instance-ip>
257IPv6:         <your-instance-ipv6>
258Region:       <your-region>
259OS:           <your-os-version>
260Disk:         <your-disk-size>
261Default user: root
262Customer ID:  <your-customer-id>
263```
264 
265## your application monitoring alert SQL
266 
267```sql
268INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
269VALUES ('alert-001', 'VPS Disk above 80%', 'disk_used_percent', 80, 'gt', 1, 300);
270 
271INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
272VALUES ('alert-002', 'VPS CPU above 90%', 'cpu_percent', 90, 'gt', 1, 60);
273 
274INSERT INTO monitoring_alerts (id, name, metric, threshold, operator, enabled, duration_seconds)
275VALUES ('alert-003', 'VPS RAM above 85%', 'mem_used_percent', 85, 'gt', 1, 120);
276```
277 
278Alerts are loaded at your application startup. Restart with `systemctl restart your-app` after inserting.
279 
280## Base64 encoding workaround
281 
282When sending scripts through Orca terminal `--text`, the local bash shell interprets `$`, `(`, and `%`. Encode locally and decode remotely:
283 
284```bash
285# Local
286cat script.sh | base64
287 
288# Remote terminal
289echo '<base64-output>' | base64 -d &gt; script.sh
290```
291