CLAUDE.md

1# CLAUDE.md - Dawarich Development Guide
2 
3This file contains essential information for Claude to work effectively with the Dawarich codebase.
4 
5## Project Overview
6 
7**Dawarich** is a self-hostable web application built with Ruby on Rails 8.0 that serves as a replacement for Google Timeline (Google Location History). It allows users to track, visualize, and analyze their location data through an interactive web interface.
8 
9### Key Features
10- Location history tracking and visualization
11- Interactive maps with multiple layers (heatmap, points, lines, fog of war)
12- Import from various sources (Google Maps Timeline, OwnTracks, Strava, GPX, GeoJSON, photos)
13- Export to GeoJSON and GPX formats
14- Statistics and analytics (countries visited, distance traveled, etc.)
15- Public sharing of monthly statistics with time-based expiration
16- Trips management with photo integration
17- Areas and visits tracking
18- Integration with photo management systems (Immich, Photoprism)
19 
20## Technology Stack
21 
22### Backend
23- **Framework**: Ruby on Rails 8.0
24- **Database**: PostgreSQL with PostGIS extension
25- **Background Jobs**: Sidekiq with Redis
26- **Authentication**: Devise
27- **Authorization**: Pundit
28- **API Documentation**: rSwag (Swagger)
29- **Monitoring**: Prometheus, Sentry
30- **File Processing**: AWS S3 integration
31 
32### Frontend
33- **CSS Framework**: Tailwind CSS with DaisyUI components
34- **JavaScript**: Stimulus, Turbo Rails, Hotwired
35- **Maps**: Leaflet.js
36- **Charts**: Chartkick
37 
38## Conventions
39- **Enums over strings:** Prefer Rails enums (integer columns) over string columns for status/type fields. Use `enum :field_name, { ... }, prefix: :field_name` to get scoped predicate methods and avoid name collisions.
40- **Turbo first:** Follow Rails 8 conventions — use Turbo Frames and Turbo Streams/broadcasts wherever appropriate to avoid full page reloads and provide smooth, in-place UI updates.
41- **SVGs as files:** Never inline SVG markup in views. Instead, save SVGs to `app/assets/svg/icons` and use `inline_svg_tag "name.svg"` to render them. This keeps views clean and SVGs reusable. Use `rails_icons` to manage SVG assets and ensure consistent styling.
42 
43## Code Style
44 
45- Follow rubocop conventions (see `.rubocop.yml`)
46- Rails defaults: convention over configuration
47- Prefer Hotwire (Turbo Frames/Streams + Stimulus) over custom JS
48- Use importmap for JS dependencies — no npm/yarn
49 
50### Key Gems
51- `activerecord-postgis-adapter` - PostgreSQL PostGIS support
52- `geocoder` - Geocoding services
53- `rgeo` - Ruby Geometric Library
54- `gpx` - GPX file processing
55- `parallel` - Parallel processing
56- `sidekiq` - Background job processing
57- `chartkick` - Chart generation
58 
59## Project Structure
60 
61```
62├── app/
63│   ├── controllers/     # Rails controllers
64│   ├── models/         # ActiveRecord models with PostGIS support
65│   ├── views/          # ERB templates
66│   ├── services/       # Business logic services
67│   ├── jobs/           # Sidekiq background jobs
68│   ├── queries/        # Database query objects
69│   ├── policies/       # Pundit authorization policies
70│   ├── serializers/    # API response serializers
71│   ├── javascript/     # Stimulus controllers and JS
72│   └── assets/         # CSS and static assets
73├── config/             # Rails configuration
74├── db/                 # Database migrations and seeds
75├── docker/             # Docker configuration
76├── spec/               # RSpec test suite
77└── swagger/            # API documentation
78```
79 
80## Core Models
81 
82### Primary Models
83- **User**: Authentication and user management
84- **Point**: Individual location points with coordinates and timestamps
85- **Track**: Collections of related points forming routes
86- **Area**: Geographic areas drawn by users
87- **Visit**: Detected visits to areas
88- **Trip**: User-defined travel periods with analytics
89- **Import**: Data import operations
90- **Export**: Data export operations
91- **Stat**: Calculated statistics and metrics with public sharing capabilities
92 
93### Geographic Features
94- Uses PostGIS for advanced geographic queries
95- Implements distance calculations and spatial relationships
96- Supports various coordinate systems and projections
97 
98## Development Environment
99 
100### Setup
1011. **Docker Development**: Use `docker-compose -f docker/docker-compose.yml up`
1022. **DevContainer**: VS Code devcontainer support available
1033. **Local Development**:
104   - `bundle exec rails db:prepare`
105   - `bundle exec sidekiq` (background jobs)
106   - `bundle exec bin/dev` (main application)
107 
108### Default Credentials
109- Username: `demo@dawarich.app`
110- Password: `safepassword`
111 
112## Testing
113 
114### Test Suite
115- **Framework**: RSpec
116- **System Tests**: Capybara + Selenium WebDriver
117- **E2E Tests**: Playwright
118- **Coverage**: SimpleCov
119- **Factories**: FactoryBot
120- **Mocking**: WebMock
121 
122### Test Commands
123```bash
124bundle exec rspec                    # Run all specs
125bundle exec rspec spec/models/       # Model specs only
126npx playwright test                  # E2E tests
127```
128 
129### Testing Best Practices — Test Behavior, Not Implementation
130 
131When writing or modifying tests, always test **observable behavior** (return values, state changes, side effects) rather than **implementation details** (which internal methods are called, in what order, with what exact arguments).
132 
133**Anti-patterns to AVOID:**
134 
1351. **Never mock the object under test** — `allow(subject).to receive(:internal_method)` makes the test a tautology
1362. **Never test private methods via `send()`** — test through the public interface instead; if creating a user triggers a trial, test by creating the user and checking `user.trial?`, not by calling `user.send(:start_trial)`
1373. **Never use `receive_message_chain`** — `allow(x).to receive_message_chain(:a, :b, :c)` breaks on any scope reorder; use real data instead
1384. **Avoid over-stubbing** — if every collaborator is mocked, the test proves nothing; mock only at external boundaries (HTTP, geocoder, external APIs)
1395. **Don't test wiring without outcomes** — `expect(Service).to receive(:new).with(args)` only proves a method was called, not that it works; verify the returned data or state change instead
1406. **Prefer `have_enqueued_job` over `expect(Job).to receive(:perform_later)`** — the former tests real ActiveJob integration; the latter just tests a mock
1417. **Don't assert on cache key formats or internal metric JSON shapes** — test that caching works (2nd call doesn't requery) or that metrics fire, not exact internal formats
1428. **Use real factory data over `allow(user).to receive(:active?).and_return(true)`** — set the actual user state: `create(:user, status: :active)`
143 
144**Good test pattern:**
145```ruby
146# Test behavior: creating an export enqueues processing
147it 'enqueues processing job' do
148  expect { create(:export, file_type: :points) }.to have_enqueued_job(ExportJob)
149end
150```
151 
152**Bad test pattern:**
153```ruby
154# Tests implementation: mocks the callback interaction
155it 'enqueues processing job' do
156  expect(ExportJob).to receive(:perform_later)  # mock, not real
157  build(:export).save!
158end
159```
160 
161## Background Jobs
162 
163### Sidekiq Jobs
164- **Import Jobs**: Process uploaded location data files
165- **Calculation Jobs**: Generate statistics and analytics
166- **Notification Jobs**: Send user notifications
167- **Photo Processing**: Extract EXIF data from photos
168 
169### Key Job Classes
170- `Tracks::ParallelGeneratorJob` - Generate track data in parallel
171- Various import jobs for different data sources
172- Statistical calculation jobs
173 
174## Public Sharing System
175 
176### Overview
177Dawarich includes a comprehensive public sharing system that allows users to share their monthly statistics with others without requiring authentication. This feature enables users to showcase their location data while maintaining privacy control through configurable expiration settings.
178 
179### Key Features
180- **Time-based expiration**: Share links can expire after 1 hour, 12 hours, 24 hours, or be permanent
181- **UUID-based access**: Each shared stat has a unique, unguessable UUID for security
182- **Public API endpoints**: Hexagon map data can be accessed via API without authentication when sharing is enabled
183- **Automatic cleanup**: Expired shares are automatically inaccessible
184- **Privacy controls**: Users can enable/disable sharing and regenerate sharing URLs at any time
185 
186### Technical Implementation
187- **Database**: `sharing_settings` (JSONB) and `sharing_uuid` (UUID) columns on `stats` table
188- **Routes**: `/shared/month/:uuid` for public viewing, `/stats/:year/:month/sharing` for management
189- **API**: `/api/v1/maps/hexagons` supports public access via `uuid` parameter
190- **Controllers**: `Shared::StatsController` handles public views, sharing management integrated into existing stats flow
191 
192### Security Features
193- **No authentication bypass**: Public sharing only exposes specifically designed endpoints
194- **UUID-based access**: Sharing URLs use unguessable UUIDs rather than sequential IDs
195- **Expiration enforcement**: Automatic expiration checking prevents access to expired shares
196- **Limited data exposure**: Only monthly statistics and hexagon data are publicly accessible
197 
198### Usage Patterns
199- **Social sharing**: Users can share interesting travel months with friends and family
200- **Portfolio/showcase**: Travel bloggers and photographers can showcase location statistics
201- **Data collaboration**: Researchers can share aggregated location data for analysis
202- **Public demonstrations**: Demo instances can provide public examples without compromising user data
203 
204## API Documentation
205 
206- **Framework**: rSwag (Swagger/OpenAPI)
207- **Location**: `/api-docs` endpoint
208- **Authentication**: API key (Bearer) for API access, UUID-based access for public shares
209 
210## Database Schema
211 
212### Key Tables
213- `users` - User accounts and settings
214- `points` - Location points with PostGIS geometry
215- `tracks` - Route collections
216- `areas` - User-defined geographic areas
217- `visits` - Detected area visits
218- `trips` - Travel periods
219- `imports`/`exports` - Data transfer operations
220- `stats` - Calculated metrics with sharing capabilities (`sharing_settings`, `sharing_uuid`)
221 
222### PostGIS Integration
223- Extensive use of PostGIS geometry types
224- Spatial indexes for performance
225- Geographic calculations and queries
226 
227## Configuration
228 
229### Environment Variables
230See `.env.template` for available configuration options including:
231- Database configuration
232- Redis settings
233- AWS S3 credentials
234- External service integrations
235- Feature flags
236 
237### Key Config Files
238- `config/database.yml` - Database configuration
239- `config/sidekiq.yml` - Background job settings
240- `config/schedule.yml` - Cron job schedules
241- `docker/docker-compose.yml` - Development environment
242 
243## Deployment
244 
245### Docker
246- Production: `docker/docker-compose.production.yml`
247- Development: `docker/docker-compose.yml`
248- Multi-stage Docker builds supported
249 
250### Procfiles
251- `Procfile` - Production Heroku deployment
252- `Procfile.dev` - Development with Foreman
253- `Procfile.production` - Production processes
254 
255## Code Quality
256 
257### Tools
258- **Ruby Linting**: RuboCop with Rails extensions
259- **JS/CSS Linting**: Biome (formatting, lint, import sorting)
260- **Security**: Brakeman, bundler-audit
261- **Dependencies**: Strong Migrations for safe database changes
262- **Performance**: Stackprof for profiling
263 
264### Commands
265```bash
266bundle exec rubocop                  # Ruby linting
267npx @biomejs/biome check --write .   # JS/CSS auto-fix (safe fixes)
268npx @biomejs/biome check --write --unsafe .  # JS/CSS auto-fix (all fixes)
269npx @biomejs/biome ci .              # JS/CSS CI check (read-only)
270bundle exec brakeman                 # Security scan
271bundle exec bundle-audit             # Dependency security
272```
273 
274### Lint Rules
275- **Always run RuboCop** on modified Ruby files before committing: `bundle exec rubocop <files>`
276- **Always run Biome** on modified JS/CSS files before committing: `npx @biomejs/biome check --write <files>`
277- If Biome `--write` leaves remaining errors, use `--write --unsafe` to apply fixes like `parseInt` radix and `Number.isNaN`
278- CI runs `biome ci --changed --since=dev` — it compares against the `dev` branch, not `master`
279- The `noStaticOnlyClass` warning is acceptable and does not fail CI
280- Tailwind CSS files (`*.tailwind.css`) have `@import` position rules disabled in `biome.json` because `@tailwind` directives must come first
281 
282## Frontend: Hotwire-First Approach
283 
284**Always prefer Turbo + Stimulus over custom JavaScript.** This project uses the Hotwire stack (Turbo Drive, Turbo Frames, Turbo Streams, Stimulus) as its primary frontend architecture. Direct `fetch()` calls, manual DOM manipulation, and standalone JS modules should only be used when Hotwire cannot handle the use case (e.g., map rendering with Leaflet/MapLibre).
285 
286### Decision Hierarchy
287 
288When adding frontend behavior, follow this order of preference:
289 
2901. **Turbo Drive** — Default. Links and forms work as SPAs with zero JS.
2912. **Turbo Frames** — Partial page updates. Wrap a section in `<turbo-frame>` and target it from links/forms.
2923. **Turbo Streams** — Server-pushed DOM updates. Use for CRUD operations that need to update multiple page sections. Respond with `turbo_stream` format from controllers.
2934. **Stimulus controller** — Client-side behavior that Turbo can't handle (toggles, form validation, UI interactions). Keep controllers thin.
2945. **Direct JS** — Last resort. Only for complex map interactions, canvas rendering, or third-party library integration (Leaflet, MapLibre, Chartkick).
295 
296### Turbo Stream Responses
297 
298For CRUD actions (create, update, destroy), respond with Turbo Streams instead of redirects or JSON:
299 
300```ruby
301# Controller
302def create
303  @area = current_user.areas.new(area_params)
304  if @area.save
305    respond_to do |format|
306      format.turbo_stream
307      format.html { redirect_to areas_path }
308    end
309  end
310end
311 
312# app/views/areas/create.turbo_stream.erb
313<%= turbo_stream.prepend "areas-list", partial: "areas/area", locals: { area: @area } %>
314<%= stream_flash(:notice, "Area created successfully") %>
315```
316 
317Use the `FlashStreamable` concern (included in controllers) to send flash messages via Turbo Streams:
318 
319```ruby
320include FlashStreamable
321 
322# In turbo_stream responses:
323stream_flash(:notice, "Success message")
324stream_flash(:error, "Error message")
325```
326 
327### Flash Messages
328 
329- **Server-side (Turbo Stream):** Use `stream_flash` from the `FlashStreamable` concern. This appends a flash partial to the `#flash-messages` container.
330- **Client-side (Stimulus/JS):** Import `Flash` from `flash_controller.js` and call `Flash.show(type, message)`:
331```javascript
332  import Flash from "./flash_controller"
333  Flash.show("notice", "Operation completed")
334  Flash.show("error", "Something went wrong")
335```
336- **Never** use raw `alert()`, `console.log` for user-facing messages, or create ad-hoc notification DOM elements.
337 
338### Stimulus Controllers
339 
340- Location: `app/javascript/controllers/`
341- Naming: `<name>_controller.js` maps to `data-controller="<name>"` in HTML
342- Use `static targets` for DOM references, `static values` for data from HTML attributes
343- Always clean up in `disconnect()` (event listeners, timers, subscriptions)
344- Prefer `data-action` attributes in HTML over `addEventListener` in JS
345- For forms, prefer `this.formTarget.requestSubmit()` over manual `fetch()` calls — this preserves Turbo form handling, CSRF tokens, and Turbo Stream responses
346 
347### File Uploads
348 
349Use the unified `upload` controller (`upload_controller.js`) for all file upload forms. Configure via `data-upload-*-value` attributes:
350 
351```erb
352<%= form_with data: {
353  controller: "upload",
354  upload_url_value: rails_direct_uploads_url,
355  upload_field_name_value: "import[files][]",
356  upload_multiple_value: true,
357  upload_target: "form"
358} do |f| %>
359```
360 
361### What NOT to Do
362 
363- **No `fetch()` for form submissions** — Use `form_with` with Turbo. If you need custom headers (API key), use Stimulus to submit the form via `requestSubmit()`.
364- **No `document.getElementById()` for updates** — Use Turbo Frames/Streams to replace DOM sections server-side.
365- **No `showFlashMessage()` or ad-hoc flash functions** — Use `Flash.show()` (client) or `stream_flash` (server).
366- **No ActionCable subscriptions for CRUD updates** — Use Turbo Stream broadcasts from models/controllers instead.
367- **No separate upload controllers per form** — Use the unified `upload` controller with value attributes for configuration.
368 
369### When Direct JS Is Acceptable
370 
371- **Map rendering**: Leaflet (Maps v1) and MapLibre GL JS (Maps v2) require imperative JS for layers, markers, and interactions.
372- **Chart rendering**: Chartkick handles its own DOM.
373- **Third-party integrations**: Libraries that don't have Hotwire adapters.
374- **Complex client-side computation**: Haversine distance, coordinate transforms, etc.
375 
376Even in these cases, wrap the integration in a Stimulus controller and connect it to the DOM via `data-controller`.
377 
378## Important Notes for Development
379 
3801. **Location Data**: Always handle location data with appropriate precision and privacy considerations
3812. **PostGIS**: Leverage PostGIS features for geographic calculations rather than Ruby-based solutions
3822.1 **Coordinates**: Use `lonlat` column in `points` table for geographic calculations
3833. **Background Jobs**: Use Sidekiq for any potentially long-running operations
3844. **Testing**: Include both unit and integration tests for location-based features
3855. **Performance**: Consider database indexes for geographic queries
3866. **Security**: Never log or expose user location data inappropriately
3877. **Migrations**: Put all migrations (schema and data) in `db/migrate/`, not `db/data/`. Data manipulation migrations use the same `ActiveRecord::Migration` class and should run in the standard migration sequence.
3888. **Public Sharing**: When implementing features that interact with stats, consider public sharing access patterns:
389   - Use `public_accessible?` method to check if a stat can be publicly accessed
390   - Support UUID-based access in API endpoints when appropriate
391   - Respect expiration settings and disable sharing when expired
392   - Only expose minimal necessary data in public sharing contexts
393 
394### Route Drawing Implementation (Critical)
395 
396⚠️ **IMPORTANT: Unit Mismatch in Route Splitting Logic**
397 
398Both Map v1 (Leaflet) and Map v2 (MapLibre) contain an **intentional unit mismatch** in route drawing that must be preserved for consistency:
399 
400**The Issue**:
401- `haversineDistance()` function returns distance in **kilometers** (e.g., 0.5 km)
402- Route splitting threshold is stored and compared as **meters** (e.g., 500)
403- The code compares them directly: `0.5 > 500` = always **FALSE**
404 
405**Result**:
406- The distance threshold (`meters_between_routes` setting) is **effectively disabled**
407- Routes only split on **time gaps** (default: 60 minutes between points)
408- This creates longer, more continuous routes that users expect
409 
410**Code Locations**:
411- **Map v1**: `app/javascript/maps/polylines.js:390`
412  - Uses `haversineDistance()` from `maps/helpers.js` (returns km)
413  - Compares to `distanceThresholdMeters` variable (value in meters)
414 
415- **Map v2**: `app/javascript/maps_maplibre/layers/routes_layer.js:82-104`
416  - Has built-in `haversineDistance()` method (returns km)
417  - Intentionally skips `/1000` conversion to replicate v1 behavior
418  - Comment explains this is matching v1's unit mismatch
419 
420**Critical Rules**:
4211. ❌ **DO NOT "fix" the unit mismatch** - this would break user expectations
4222. ✅ **Keep both versions synchronized** - they must behave identically
4233. ✅ **Document any changes** - route drawing changes affect all users
4244. ⚠️ If you ever fix this bug:
425   - You MUST update both v1 and v2 simultaneously
426   - You MUST migrate user settings (multiply existing values by 1000 or divide by 1000 depending on direction)
427   - You MUST communicate the breaking change to users
428 
429**Additional Route Drawing Details**:
430- **Time threshold**: 60 minutes (default) - actually functional
431- **Distance threshold**: 500 meters (default) - currently non-functional due to unit bug
432- **Sorting**: Map v2 sorts points by timestamp client-side; v1 relies on backend ASC order
433- **API ordering**: Map v2 must request `order: 'asc'` to match v1's chronological data flow
434 
435## Plan System (Lite vs Pro)
436 
437Dawarich Cloud has a two-tier plan system. Self-hosted instances bypass all plan restrictions (`DawarichSettings.self_hosted?` returns true, all users effectively have Pro).
438 
439### Plans
440 
441- **Pro** (`plan: :pro`, enum value `1`) — Full access to all features, no data window
442- **Lite** (`plan: :lite`, enum value `0`) — Free tier with restricted feature set
443 
444Plan is stored as an integer enum on the `users` table. New cloud users start on Lite via trial flow.
445 
446### Lite Plan Restrictions
447 
448**Data visibility window (12 months):**
449- Lite users only see data from the last 12 months (`DawarichSettings::LITE_DATA_WINDOW`)
450- Implemented as a query-time filter in `PlanScopable` concern (`app/models/concerns/plan_scopable.rb`)
451- Scoped methods: `scoped_points`, `scoped_tracks`, `scoped_visits`, `scoped_stats`
452- Data is **never deleted** — only filtered from UI and API reads. Export uses unscoped `user.points` etc.
453- `plan_restricted?` returns `true` only when `!self_hosted? && lite?`
454 
455**Disabled map layers (Pro-only):**
456- Heatmap, Fog of War, Scratch Map, Globe View
457- Lite users get a 20-second timed preview, then auto-hide with upgrade prompt
458- Gating logic: `app/javascript/maps_maplibre/utils/layer_gate.js`
459- UI components: `Toast` (countdown) and `UpgradeBanner` (post-preview CTA)
460 
461**API restrictions:**
462- Write API returns 403 (`require_write_api!` in `ApiController`)
463- Read API scopes results to 12-month window (`apply_plan_scope` in `ApiController`)
464- Rate limit: 200 req/hr (Lite) vs 1,000 req/hr (Pro) via `rack-attack` (`config/initializers/rack_attack.rb`)
465 
466**Disabled features:**
467- Integrations (Immich, Photoprism)
468- Public sharing of stats
469- Full digest view
470 
471**Plan endpoint:** `GET /api/v1/plan` returns current plan and feature flags (`Api::V1::PlanController`)
472 
473### Archival Warning System
474 
475`Lite::ArchivalWarningJob` runs daily for Lite users and sends warnings at three thresholds:
4761. **11 months** — In-app notification warning data will archive in 30 days
4772. **11.5 months** — Email notification
4783. **12 months** — In-app notification that data has been archived (hidden from view)
479 
480Warnings are deduped via `settings['archival_warnings']` JSONB on the user record.
481 
482### Development Guidelines for Plan Gating
483 
484- Use `user.plan_restricted?` to check if restrictions apply (returns false for self-hosted)
485- Use `user.scoped_*` methods instead of `user.points`/`user.tracks` etc. for plan-aware queries
486- Use `require_pro_api!` or `require_write_api!` before_actions in API controllers
487- Use `apply_plan_scope(relation)` when scoping points that don't start from `user.points`
488- Frontend: use `isGatedPlan(userPlan)` and `gatedToggle()` from `layer_gate.js` for map layer toggling
489- Export must always use unscoped relations — users can export all their data regardless of plan
490 
491## Contributing
492 
493- **Main Branch**: `master`
494- **Development**: `dev` branch for pull requests
495- **Issues**: GitHub Issues for bug reports
496- **Discussions**: GitHub Discussions for feature requests
497- **Community**: Discord server for questions
498 
499## Resources
500 
501- **Documentation**: https://dawarich.app/docs/
502- **Repository**: https://github.com/Freika/dawarich
503- **Discord**: https://discord.gg/pHsBjpt5J8
504- **Changelog**: See CHANGELOG.md for version history
505- **Development Setup**: See DEVELOPMENT.md
506