AGENTS.md

1# LLM Context Guide for Apache Superset
2 
3Apache Superset is a data visualization platform with Flask/Python backend and React/TypeScript frontend.
4 
5## Run Pre-commit Before Pushing
6 
7Always run pre-commit against the files changed by the current branch before
8pushing. This matches CI and keeps unrelated failures already present on
9`master` from blocking otherwise independent work.
10 
11```bash
12# Stage your changes first
13git add .
14 
15# Run pre-commit on staged files
16pre-commit run
17 
18# If there are auto-fixes, stage them and commit
19git add .
20git commit --amend  # or new commit
21```
22 
23Use `pre-commit run --all-files` when auditing or repairing the repository-wide
24baseline. If that check finds failures in files untouched by the current branch,
25fix them in a separate branch rather than adding unrelated changes to the
26current pull request.
27 
28Common pre-commit failures:
29- **Formatting** - black, oxfmt, eslint will auto-fix
30- **Type errors** - mypy failures need manual fixes
31- **Linting** - ruff, pylint issues need manual fixes
32 
33## ⚠️ CRITICAL: Ongoing Refactors (What NOT to Do)
34 
35**These migrations are actively happening - avoid deprecated patterns:**
36 
37### Frontend Modernization
38- **NO `any` types** - Use proper TypeScript types
39- **NO JavaScript files** - Convert to TypeScript (.ts/.tsx)
40- **Use @superset-ui/core** - Don't import Ant Design directly, prefer Ant Design component wrappers from @superset-ui/core/components
41- **Use antd theming tokens** - Prefer antd tokens over legacy theming tokens
42- **Avoid custom css and styles** - Follow antd best practices and avoid styling and custom CSS whenever possible
43 
44### Testing Strategy Migration
45- **Prefer unit tests** over integration tests
46- **Prefer integration tests** over end-to-end tests
47- **Use Playwright for E2E tests** - Migrating from Cypress
48- **Cypress is deprecated** - Will be removed once migration is completed
49- **Use Jest + React Testing Library** for component testing
50- **Use `test()` instead of `describe()`** - Follow [avoid nesting when testing](https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) principles
51 
52### Backend Type Safety
53- **Add type hints** - All new Python code needs proper typing
54- **MyPy compliance** - Run `pre-commit run mypy` to validate
55- **SQLAlchemy typing** - Use proper model annotations
56 
57### UUID Migration
58- **Prefer UUIDs over auto-incrementing IDs** - New models should use UUID primary keys
59- **External API exposure** - Use UUIDs in public APIs instead of internal integer IDs
60- **Existing models** - Add UUID fields alongside integer IDs for gradual migration
61 
62## Security and Threat Model
63 
64Before evaluating any code path for security issues, read [`SECURITY.md`](SECURITY.md). It is the canonical, authoritative source for Apache Superset's security model and is referenced by both human reporters and automated scanners.
65 
66In short, the test for whether a finding is in scope is one question:
67 
68> *Does it let a principal perform an action the role and capability matrix in `SECURITY.md` does not entitle them to?*
69 
70If yes, it is in scope. If no, it is not.
71 
72The three trust boundaries are:
73 
741. **The Admin role** is a fully trusted operational principal. Anything an Admin can do through documented configuration, API, or UI is an intended capability, not a vulnerability.
752. **The operator** owns deployment-time decisions (secrets, network exposure, feature-flag selection, connector and codec choices, notification destinations, third-party plugins). Misconfiguration at this layer is a deployment defect, not a Superset vulnerability.
763. **The codebase** is responsible for enforcing the role and capability matrix wherever it exposes functionality to a principal: API routes, command and DAO layers, UI handlers, background jobs, and any other entry point. A missing or incorrect enforcement check is in scope no matter where it lives.
77 
78The security model assumes that operator-controlled infrastructure, including the metadata database, cache backends, message brokers, secret stores, and deployment environment, remains within the operator's trust boundary. Vulnerabilities must demonstrate a security boundary violation by an attacker who does not already control those systems.
79 
80Route-level authorization in this codebase uses one of three Flask-AppBuilder decorators depending on the route type:
81 
82- `@protect()` for REST API routes (`ModelRestApi` / `BaseApi`)
83- `@has_access_api` for legacy view routes
84- `@has_access` for legacy HTML view routes
85 
86Object-level authorization via `security_manager.raise_for_access(...)` applies to data-bearing resources: dashboards, charts, datasets and datasources, queries, database and table access, and query contexts. Other resources (annotations, tags, CSS templates, reports, RLS rules, and similar) rely on the route-level decorator plus DAO `base_filters` for ownership scoping; the absence of `raise_for_access` on these resources is by design, not a finding. Code that omits the per-object gate on a route that returns or mutates a specific data-bearing object is in scope; code that follows the correct pattern for its resource class can still contain injection, SSRF, XSS, or other classes of finding unrelated to authorization, which are evaluated separately.
87 
88The full role and capability matrix, in-scope and out-of-scope class lists, and CVE aggregation rules are in [`SECURITY.md`](SECURITY.md). Defer to that document for any specifics.
89 
90**Requirements for findings filed by automated tooling**
91 
92Automated scanners (LLM-based code scanners, static analyzers, dependency tools) that file findings against this codebase must, in each finding, name:
93 
941. The specific role and capability matrix row in [`SECURITY.md`](SECURITY.md) the finding believes is violated.
952. The principal the finding assumes the attacker holds (Public, Gamma, sql_lab, Alpha, Admin, Embedded guest token, or a custom role with explicit capability grants).
96 
97Findings that cannot identify both should be filed as questions, not vulnerabilities. This requirement exists to ensure every reported issue is testable against the published security model and to keep speculative or pattern-match-only reports out of the triage queue.
98 
99## Key Directories
100 
101```
102superset/
103├── superset/                    # Python backend (Flask, SQLAlchemy)
104│   ├── views/api/              # REST API endpoints
105│   ├── models/                 # Database models
106│   └── connectors/             # Database connections
107├── superset-frontend/src/       # React TypeScript frontend
108│   ├── components/             # Reusable components
109│   ├── explore/                # Chart builder
110│   ├── dashboard/              # Dashboard interface
111│   └── SqlLab/                 # SQL editor
112├── superset-frontend/packages/
113│   └── superset-ui-core/       # UI component library (USE THIS)
114├── tests/                      # Python/integration tests
115├── docs/                       # Documentation (UPDATE FOR CHANGES)
116└── UPDATING.md                 # Breaking changes log
117```
118 
119## Code Standards
120 
121### TypeScript Frontend
122- **Avoid `any` types** - Use proper TypeScript, reuse existing types
123- **Functional components** with hooks
124- **@superset-ui/core** for UI components (not direct antd)
125- **Jest** for testing (NO Enzyme)
126- **Redux** for global state where it exists, hooks for local
127 
128### Python Backend  
129- **Type hints required** for all new code
130- **MyPy compliant** - run `pre-commit run mypy`
131- **SQLAlchemy models** with proper typing
132- **pytest** for testing
133 
134### Apache License Headers
135- **New files require ASF license headers** - When creating new code files, include the standard Apache Software Foundation license header
136- **LLM instruction files are excluded** - Files like AGENTS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead
137 
138### Code Comments
139- **Avoid time-specific language** - Don't use words like "now", "currently", "today" in code comments as they become outdated
140- **Write timeless comments** - Comments should remain accurate regardless of when they're read
141 
142## Documentation Requirements
143 
144- **docs/**: Update for any user-facing changes
145- **UPDATING.md**: Add breaking changes here
146- **Docstrings**: Required for new functions/classes
147 
148## Developer Portal: Storybook-to-MDX Documentation
149 
150The Developer Portal auto-generates MDX documentation from Storybook stories. **Stories are the single source of truth.**
151 
152### Core Philosophy
153- **Fix issues in the STORY, not the generator** - When something doesn't render correctly, update the story file first
154- **Generator should be lightweight** - It extracts and passes through data; avoid special cases
155- **Stories define everything** - Props, controls, galleries, examples all come from story metadata
156 
157### Story Requirements for Docs Generation
158- Use `export default { title: '...' }` (inline), not `const meta = ...; export default meta;`
159- Name interactive stories `Interactive${ComponentName}` (e.g., `InteractiveButton`)
160- Define `args` for default prop values
161- Define `argTypes` at the story level (not meta level) with control types and descriptions
162- Use `parameters.docs.gallery` for size×style variant grids
163- Use `parameters.docs.sampleChildren` for components that need children
164- Use `parameters.docs.liveExample` for custom live code blocks
165- Use `parameters.docs.staticProps` for complex object props that can't be parsed inline
166 
167### Generator Location
168- Script: `docs/scripts/generate-superset-components.mjs`
169- Wrapper: `docs/src/components/StorybookWrapper.jsx`
170- Output: `docs/developer_docs/components/`
171 
172## Architecture Patterns
173 
174### Security & Features
175- **Security model**: see the top-level [Security and Threat Model](#security-and-threat-model) section and [`SECURITY.md`](SECURITY.md)
176- **RBAC**: Role-based access via Flask-AppBuilder
177- **Feature flags**: Control feature rollouts
178- **Row-level security**: SQL-based data access control
179 
180## Test Utilities
181 
182### Python Test Helpers
183- **`SupersetTestCase`** - Base class in `tests/integration_tests/base_tests.py`
184- **`@with_config`** - Config mocking decorator
185- **`@with_feature_flags`** - Feature flag testing
186- **`login_as()`, `login_as_admin()`** - Authentication helpers
187- **`create_dashboard()`, `create_slice()`** - Data setup utilities
188 
189### TypeScript Test Helpers
190- **`superset-frontend/spec/helpers/testing-library.tsx`** - Custom render() with providers
191- **`createWrapper()`** - Redux/Router/Theme wrapper
192- **`selectOption()`** - Select component helper
193- **React Testing Library** - NO Enzyme (removed)
194 
195### Test Database Patterns
196- **Mock patterns**: Use `MagicMock()` for config objects, avoid `AsyncMock` for synchronous code
197- **API tests**: Update expected columns when adding new model fields
198 
199### Running Tests
200```bash
201# Frontend
202npm run test                           # All tests
203npm run test -- filename.test.tsx     # Single file
204 
205# E2E Tests (Playwright - NEW)
206npm run playwright:test                # All Playwright tests
207npm run playwright:ui                  # Interactive UI mode
208npm run playwright:headed              # See browser during tests
209npx playwright test tests/auth/login.spec.ts  # Single file
210npm run playwright:debug tests/auth/login.spec.ts  # Debug specific file
211 
212# E2E Tests (Cypress - DEPRECATED)
213cd superset-frontend/cypress-base
214npm run cypress-run-chrome             # All Cypress tests (headless)
215npm run cypress-debug                  # Interactive Cypress UI
216 
217# Backend  
218pytest                                 # All tests
219pytest tests/unit_tests/specific_test.py  # Single file
220pytest tests/unit_tests/               # Directory
221 
222# If pytest fails with database/setup issues, ask the user to run test environment setup
223```
224 
225## Environment Validation
226 
227**Quick Setup Check (run this first):**
228 
229```bash
230# Verify Superset is running
231curl -f http://localhost:8088/health || echo "❌ Setup required - see https://superset.apache.org/docs/contributing/development#working-with-llms"
232```
233 
234**If health checks fail:**
235"It appears you aren't set up properly. Please refer to the [Working with LLMs](https://superset.apache.org/docs/contributing/development#working-with-llms) section in the development docs for setup instructions."
236 
237**Key Project Files:**
238- `superset-frontend/package.json` - Frontend build scripts (`npm run dev` on port 9000, `npm run test`, `npm run lint`)
239- `pyproject.toml` - Python tooling (ruff, mypy configs)
240- `requirements/` folder - Python dependencies (base.txt, development.txt)
241 
242## SQLAlchemy Query Best Practices  
243- **Use negation operator**: `~Model.field` instead of `== False` to avoid ruff E712 errors
244- **Example**: `~Model.is_active` instead of `Model.is_active == False`
245 
246## Pull Request Guidelines
247 
248**When creating pull requests:**
249 
2501. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
2512. **Use the template sections**: Include all sections from the template (SUMMARY, BEFORE/AFTER, TESTING INSTRUCTIONS, ADDITIONAL INFORMATION)
2523. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
253   - Format: `type(scope): description`
254   - Example: `fix(dashboard): load charts correctly`
255   - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
256 
257**Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time.
258 
259## Pre-commit Validation
260 
261**Use pre-commit hooks for quality validation:**
262 
263```bash
264# Install hooks
265pre-commit install
266 
267# IMPORTANT: Stage your changes first!
268git add .                        # Pre-commit only checks staged files
269 
270# Quick validation (faster than --all-files)
271pre-commit run                   # Staged files only
272pre-commit run mypy              # Python type checking
273pre-commit run format            # Code formatting
274pre-commit run eslint            # Frontend linting
275```
276 
277**Important pre-commit usage notes:**
278- **Stage files first**: Run `git add .` before `pre-commit run` to check only changed files (much faster)
279- **Virtual environment**: Activate your Python virtual environment before running pre-commit
280```bash
281  # Common virtual environment locations (yours may differ):
282  source .venv/bin/activate      # if using .venv
283  source venv/bin/activate       # if using venv
284  source ~/venvs/superset/bin/activate  # if using a central location
285```
286  If you get a "command not found" error, ask the user which virtual environment to activate
287- **Auto-fixes**: Some hooks auto-fix issues (e.g., trailing whitespace). Re-run after fixes are applied
288 
289## Common File Patterns
290 
291### API Structure
292- **`/api.py`** - REST endpoints with decorators and OpenAPI docstrings
293- **`/schemas.py`** - Marshmallow validation schemas for OpenAPI spec
294- **`/commands/`** - Business logic classes with @transaction() decorators
295- **`/models/`** - SQLAlchemy database models
296- **OpenAPI docs**: Auto-generated at `/swagger/v1` from docstrings and schemas
297 
298### Migration Files
299- **Location**: `superset/migrations/versions/`
300- **Naming**: `YYYY-MM-DD_HH-MM_hash_description.py`
301- **Utilities**: Use helpers from `superset.migrations.shared.utils` for database compatibility
302- **Pattern**: Import utilities instead of raw SQLAlchemy operations
303 
304## Platform-Specific Instructions
305 
306- **[CLAUDE.md](CLAUDE.md)** - For Claude/Anthropic tools
307- **[.github/copilot-instructions.md](.github/copilot-instructions.md)** - For GitHub Copilot  
308- **[GEMINI.md](GEMINI.md)** - For Google Gemini tools
309- **[GPT.md](GPT.md)** - For OpenAI/ChatGPT tools
310- **[.cursor/rules/dev-standard.mdc](.cursor/rules/dev-standard.mdc)** - For Cursor editor
311 
312---
313 
314**LLM Note**: This codebase is actively modernizing toward full TypeScript and type safety. Always run `pre-commit run` to validate changes. Follow the ongoing refactors section to avoid deprecated patterns.
315