CLAUDE.md

1# CLAUDE.md
2 
3This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4 
5Refer to llm-shared/project_tech_stack.md for core technology choices, build system configuration, and library preferences.
6 
7## Key Commands
8 
9- `task` or `task build` - Build the application (runs tests and lint first)
10- `task test` - Run tests with coverage report (generates coverage/coverage.html)
11- `task lint` - Run golangci-lint
12- `task clean` - Clean build artifacts
13- `task upgrade-deps` - Update all dependencies
14- `task build-linux` - Cross-compile for Linux
15- `task test-ci` - Run tests for CI (with ci build tag)
16 
17**Running the application:**
18 
19- `./build/hermes --help` - View available commands
20- `./build/hermes import goodreads --help` - View importer-specific options
21- `./build/hermes enhance --help` - View enhance command options
22- `go run . import goodreads -f goodreads_library_export.csv` - Run directly without building
23- `go run . import goodreads --automated --headful` - Automated Goodreads export download (Chrome required)
24- `go run . import goodreads --automated --dry-run` - Test automation without import
25- `go run . enhance -d markdown/imdb --tmdb-generate-content` - Enhance existing notes with TMDB data
26 
27**Development workflow:**
28 
29- Tests must pass before builds (enforced by Taskfile dependencies)
30- Always run `goimports -w .` after modifying Go files
31- Use `go test ./...` for quick test runs during development
32 
33## Architecture Overview
34 
35Hermes is a data import/export tool that converts exports from various sources (Goodreads, IMDb, Letterboxd, Steam) into unified formats (JSON, Markdown, SQLite/Datasette). It also provides an `enhance` command to enrich existing markdown notes with TMDB data.
36 
37**Key Components:**
38 
39- `cmd/root.go` - Kong-based CLI structure with nested commands
40- `cmd/{source}/` - Each data source has its own package (goodreads, imdb, letterboxd, steam)
41- `cmd/enhance/` - Enhance existing markdown notes with TMDB data
42- `internal/` - Shared utilities and packages
43  - `cache/` - API response caching
44  - `cmdutil/` - Command setup helpers
45  - `config/` - Global configuration management
46  - `datastore/` - SQLite/Datasette integration
47  - `enrichment/` - TMDB enrichment functionality
48  - `errors/` - Custom error types
49  - `fileutil/` - File operations, markdown/JSON utilities
50  - `tmdb/` - TMDB API client
51  - `tui/` - Interactive terminal UI for TMDB selection
52 
53**Standard Importer Structure:**
54 
55```plain
56cmd/{source}/
57├── cmd.go          # Command setup and execution
58├── parser.go       # Input data parsing
59├── types.go        # Data models
60├── {api}.go        # External API integration (e.g., omdb.go, openlibrary.go)
61├── cache.go        # API response caching
62├── json.go         # JSON output formatting
63├── markdown.go     # Markdown output formatting
64├── testdata/       # Test fixtures and expected outputs
65└── *_test.go       # Unit tests
66```
67 
68## CLI Framework (Kong)
69 
70- Uses Kong for command-line parsing (defined in `cmd/root.go`)
71- CLI structure: `CLI` struct contains global flags and `ImportCmd` with subcommands
72- Each importer command (GoodreadsCmd, IMDBCmd, etc.) implements a `Run() error` method
73- Kong automatically handles help text, usage messages, and error handling
74- Commands read from config file if CLI flags not provided (CLI flags take precedence)
75 
76**Adding a new importer command:**
77 
781. Add command struct to `cmd/root.go` (e.g., `type NewSourceCmd struct`)
792. Add struct field to `ImportCmd` with cmd and help tags
803. Implement `Run() error` method that calls the importer package
814. Follow pattern of reading from config with CLI flag override
82 
83## Enhance Command
84 
85The `enhance` command enriches existing markdown notes with TMDB data without re-importing from original sources.
86 
87**Usage:**
88 
89```bash
90# Basic usage - enhance all notes in a directory
91./build/hermes enhance -d markdown/imdb
92 
93# Recursive scan with TMDB content generation
94./build/hermes enhance -d markdown/letterboxd -r --tmdb-generate-content
95 
96# Download cover images and generate content
97./build/hermes enhance -d markdown/imdb --tmdb-download-cover --tmdb-generate-content
98 
99# Interactive mode for manual TMDB selection
100./build/hermes enhance -d markdown/letterboxd --tmdb-interactive
101 
102# Dry run to see what would be enhanced
103./build/hermes enhance -d markdown/imdb --dry-run
104```
105 
106**Key Features:**
107 
108- Scans directory for markdown files with YAML frontmatter
109- Extracts title, year, and IMDB ID from existing notes
110- Searches TMDB for matching content
111- Updates frontmatter with TMDB ID, runtime, genres, etc.
112- Optionally downloads cover images
113- Optionally generates TMDB content sections (cast, crew, similar titles, etc.)
114- Supports interactive TUI for selecting from multiple TMDB matches
115- Skips notes that already have TMDB data (unless `--overwrite` flag is used)
116- Dry-run mode to preview changes without modifying files
117 
118**Implementation:**
119 
120- `cmd/enhance/cmd.go` - Command logic, file discovery, and processing
121- `cmd/enhance/parser.go` - YAML frontmatter parsing and markdown rebuilding
122- Uses `internal/enrichment` for TMDB API integration
123- Leverages existing TMDB client and TUI components
124 
125## Configuration
126 
127- Primary config: `config.yaml` (YAML format, auto-generated on first run)
128- CLI flags override config file values
129- Global settings in `cmd/root.go`: output directories, overwrite flag, datasette config
130- Command-specific config uses namespaced keys (e.g., `goodreads.csvfile`, `steam.apikey`)
131- Viper manages config loading with defaults in `initConfig()`
132 
133## Development Patterns
134 
135**Adding New Importers:**
136 
1371. Create new package under `cmd/{source}/`
1382. Implement standard structure (cmd.go, parser.go, types.go, etc.)
1393. Add command struct and Run() method to `cmd/root.go`
1404. Add struct field to `ImportCmd` in `cmd/root.go`
1415. Follow existing patterns for API integration, caching, and output formatting
1426. Add tests in `testdata/` subdirectory
143 
144**Common Utilities:**
145 
146- `internal/cmdutil` - Command setup helpers
147- `internal/fileutil` - File operations, markdown/JSON utilities
148- `internal/config` - Global configuration management
149- `internal/datastore` - SQLite/Datasette integration
150- `internal/cache` - API response caching
151- `internal/errors` - Custom error types (e.g., RateLimitError)
152 
153**API Integration:**
154 
155- All API responses cached in SQLite database (`cache.db` in project root, configurable via `cache.dbfile`)
156- Cache TTL defaults to 720h (30 days), configurable via `cache.ttl`
157- Implement API client logic within relevant command package (e.g., `cmd/goodreads/openlibrary.go`)
158- Respect API rate limits using delays (`time.Sleep`) or by handling specific rate limit errors
159- Handle common API errors gracefully (log warnings for 404s, retry or fail on persistent errors)
160- Use existing patterns from OMDB or OpenLibrary integration
161 
162**Error Handling:**
163 
164- Use standard Go error handling (`errors.New`, `fmt.Errorf`)
165- Return errors up the call stack for handling by Kong's command execution
166- Wrap errors with context: `fmt.Errorf("failed to process item %s: %w", itemID, err)`
167- Use custom error types from `internal/errors/` for specific conditions
168- Log significant errors but return them to let top level handle exit codes
169 
170## Caching Patterns
171 
172Use the appropriate caching strategy from `internal/cache`:
173 
174- **`cache.GetOrFetch()`** - Cache all responses with global TTL (default 30 days)
175  - Use for: Steam games, TMDB details by ID
176  - Example: `cache.GetOrFetch("steam_cache", appID, fetchFunc)`
177 
178- **`cache.GetOrFetchWithPolicy()`** - Cache only certain responses (conditional)
179  - Use for: TMDB searches (don't cache empty results)
180  - Example: `cache.GetOrFetchWithPolicy("tmdb_cache", key, fetchFunc, shouldCache)`
181 
182- **`cache.GetOrFetchWithTTL()`** - Different TTLs for different result types (negative caching)
183  - Use for: Goodreads books (7 days for "not found", 30 days for successful)
184  - Helper: `cache.SelectNegativeCacheTTL(func(r *CachedResult) bool { return r.NotFound })`
185  - Example: See `cmd/goodreads/cache.go` for reference implementation
186 
187**When adding new cached operations:**
1881. Choose appropriate strategy (GetOrFetch, GetOrFetchWithPolicy, or GetOrFetchWithTTL)
1892. Design deterministic cache keys (normalize if needed)
1903. Add table schema to `internal/cache/schema.go`
1914. Add table name to `ValidCacheTableNames` map
1925. Add source to cache invalidation in `internal/cache/cmd.go`
193 
194**For detailed guidance:**
195- Developer guide: `docs/cache-architecture.md`
196- User documentation: `docs/caching.md`
197- Architecture decisions: `docs/decisions/001-cache-architecture.md`
198 
199## Testing Requirements
200 
201- Write unit tests for parsing logic, API interaction, and output generation
202- Test files in same package with `_test.go` suffix
203- Tests must pass before builds (enforced by Taskfile)
204- Use `testdata/` subdirectory within each command package for fixtures
205- Employ table-driven tests for validating multiple input cases
206- Use `//go:build !ci` to skip tests in CI that require external dependencies
207 
208## Datasette Integration
209 
210- Supports both local SQLite and remote Datasette storage
211- `internal/datastore` provides unified interface
212- Configuration under `datasette:` key in config (enabled, mode, dbfile, remote_url, api_token)
213- Local mode writes to `hermes.db`, remote mode uses API
214- Enable via `--datasette` flag or config file
215 
216## Logging
217 
218- Uses `log/slog` with custom handler from `github.com/lepinkainen/humanlog`
219- Initialized in `cmd/root.go` with `slog.LevelInfo` default
220- `InfoLevel` for progress messages (starting import, items processed)
221- `DebugLevel` for verbose debugging info (API request/response details, cache hits/misses)
222- `WarnLevel` for recoverable issues (skipping items but continuing)
223- `ErrorLevel` for significant problems before returning an error
224 
225## Output Formats & Handling
226 
227- Default output directories (`markdown/`, `json/`) set in `cmd/root.go`, configurable via `config.yaml`
228- Commands allow specifying subdirectories for output via `-o` flag (e.g., `markdown/goodreads/`)
229- Use `internal/fileutil` for writing files (WriteMarkdownFile for Markdown, WriteJSONFile for JSON); compose markdown notes with `internal/obsidian` (Frontmatter, TagSet, BuildNoteMarkdown)
230- Markdown files use YAML frontmatter (Obsidian-compatible)
231- Follow existing patterns for Markdown frontmatter and JSON structure for each data type
232- Respect `--overwrite` flag logic
233 
234## Important Notes
235 
236- Go-only project (Go 1.24+)
237- Follows standard Go project layout and idioms
238- Each importer handles its own data enrichment and API integration
239- Use `goimports -w .` after making changes (not gofmt)
240- Use shared utility functions from `internal/` packages
241- Build artifacts go to `build/` directory
242- Cache stored in SQLite database (`cache.db`, safe to delete for cache invalidation)
243