---
description:
globs:
alwaysApply: true
---
# Testing Guide for Script Kit App

## Avoid Running the App

Never, ever run the app. Ask me to start it, then you can check the logs.

## Test Configuration

### Performance Optimizations
- Use `bail: 1` for fast failure (stop on first test failure)
- Enable parallel execution with `singleThread: false` and `isolate: true` for speed
- Set reasonable thread limits: `maxThreads: 4`, `minThreads: 2`
- Use `vitest run` instead of `vitest` for single test runs

### Environment Setup
- **Main process tests**: Use `node` environment
- **Renderer tests**: Use `jsdom` environment
- Configure separate vitest workspace configs for different environments

### Test Script Configuration
```json
{
  "scripts": {
    "test": "vitest run"  // Single run, not watch mode
  }
}
```

## Test Execution Strategy

### Concurrent vs Sequential Testing
Choose the right execution strategy based on test complexity:

#### ✅ Parallel Tests (Use `describe.concurrent()`)
- Simple file operations (create, read, delete)
- Basic validation tests
- Mock-heavy tests with no external dependencies
- Tests that don't require complex timing

#### ✅ Sequential Tests (Use regular `describe()`)
- File system operations that require exclusive access
- Directory renames and complex file operations
- Tests with intricate timing dependencies
- Resource-intensive operations

### Load-Resilient Test Design
Tests must work under both isolated and concurrent execution:

```typescript
// ❌ Fragile - only works in isolation
const events = await collectEvents(500, async () => {
  await writeFile(path, content);
});

// ✅ Robust - works under system load
const events = await collectEventsIsolated(1500, async (events, dirs) => {
  await writeFile(path, content);
  // Wait for file system under load
  await new Promise(resolve => setTimeout(resolve, 300));
});
```

### Test Environment Differences
Be aware that `pnpm test` creates different conditions than individual file execution:

- **Individual files**: Lower resource contention, faster execution
- **Full test suite**: Higher system load, requires longer timeouts
- **Solution**: Design tests to be resilient to both conditions

## Mocking Strategies

### ✅ What Works

#### Inline Mocks in Test Files
Prefer inline mocks over shared mock utilities to avoid circular dependencies:

```typescript
vi.mock('electron', () => ({
  app: {
    getPath: vi.fn((name: string) => {
      switch (name) {
        case 'userData': return '/Users/test/Library/Application Support/ScriptKit';
        default: return '/Users/test';
      }
    }),
    // ... other app methods
  },
  powerMonitor: {
    on: vi.fn(),
    addListener: vi.fn(),  // Important: Include both on AND addListener
    listeners: vi.fn(() => []),
  }
}));
```

#### Essential Electron APIs to Mock
- `app.getPath()` - Critical for path resolution
- `powerMonitor.addListener()` - Not just `on()`, include both
- `nativeTheme` - Often required by components
- `BrowserWindow` - With full webContents mock
- `crashReporter.start()` - Often called during initialization

#### Electron-log Mock Structure
```typescript
vi.mock('electron-log', () => ({
  default: {
    transports: {
      file: { level: 'info' },
      console: { level: false },
      ipc: { level: false },  // Essential - prevents "Cannot set properties of undefined"
    },
    info: vi.fn(),
    error: vi.fn(),
    // ... other log methods
  }
}));
```

#### Node.js Module Mocking
For `node:os` and other Node modules, provide complete API surface:
```typescript
vi.mock('node:os', () => ({
  default: {
    homedir: vi.fn(() => '/Users/test'),
    platform: vi.fn(() => 'darwin'),
    // ... all other os methods
    constants: {
      signals: { /* full signal definitions */ }
    }
  }
}));
```

### ❌ What Doesn't Work

#### Shared Mock Utilities
Avoid creating shared mock files that are imported across tests:
```typescript
// ❌ Don't do this - causes circular dependencies
import { setupCommonMocks } from './src/test-utils/mocks';
```

#### Incomplete Mock Objects
Missing required properties cause runtime errors:
```typescript
// ❌ Incomplete - missing required properties
const testScript = {
  filePath: '/test/path/script.ts',
  system: 'resume'
  // Missing: command, id, name
};
```

#### Complex Debounce/Timing Tests
Tests that rely on complex timing with fake timers and lodash debounce are fragile:
```typescript
// ❌ Fragile - timing-dependent
vi.advanceTimersByTime(250);
expect(debouncedFunction).toHaveBeenCalledTimes(2);
```

## Test Object Requirements

### Script Objects
When creating test script objects, include all required properties:
```typescript
const testScript = {
  filePath: '/test/path/script.ts',
  kenv: '',
  system: 'resume' as const,
  type: ProcessType.System,
  command: 'node',      // Required
  id: 'test-script',    // Required
  name: 'test-script'   // Required for Choice interface
};
```

### System Event Strings
Use proper system event syntax:
```typescript
system: 'suspend lock-screen' as const  // Multiple events
system: 'resume' as const               // Single event
```

## Mock State Management

### Between Tests
```typescript
beforeEach(() => {
  vi.useFakeTimers();
  // Don't clear mocks if they're used by debounced functions
});

afterEach(() => {
  vi.useRealTimers();
  vi.clearAllMocks(); // Clear after tests complete
});
```

### PowerMonitor Event Handling
For tests that need to simulate events:
```typescript
const mockElectronBase = vi.hoisted(() => {
  const handlers = new Map();
  return {
    powerMonitor: {
      addListener: vi.fn((event: string, handler: Function) => {
        if (!handlers.has(event)) handlers.set(event, []);
        handlers.get(event).push(handler);
      }),
      listeners: vi.fn((event: string) => handlers.get(event) || [])
    }
  };
});
```

## Integration vs Unit Tests

### Skip Integration Tests
Integration tests that require external dependencies should be skipped in the main test suite:
```typescript
it.skip('should run external command', async () => {
  // Test requires pnpm, specific file paths, etc.
});
```

### Focus on Unit Logic
Test the core business logic rather than external integrations:
- File watching logic (mocked file operations)
- Event registration/deregistration
- State management
- Component rendering

## Performance Testing

### File System Tests
Use temporary directories and proper cleanup:
```typescript
const testDir = vi.hoisted(() =>
  import('tmp-promise').then(({ dir }) => dir({ unsafeCleanup: true }))
);
```

### Isolated Directory Pattern
For file system tests that need complete isolation from each other:

```typescript
/**
 * Create isolated test directories for parallel-safe testing
 */
async function createIsolatedTestDirs(testName: string) {
  const { dir } = await import('tmp-promise');
  const tmpDir = await dir({
    unsafeCleanup: true,
    prefix: `test-${testName}-`,
  });

  const isolatedDirs = {
    root: tmpDir.path,
    kit: path.join(tmpDir.path, '.kit'),
    kenv: path.join(tmpDir.path, '.kenv'),
    scripts: path.join(tmpDir.path, '.kenv', 'scripts'),
    // ... other directories
    cleanup: tmpDir.cleanup,
  };

  // Create directory structure
  await Promise.all([
    ensureDir(isolatedDirs.kit),
    ensureDir(isolatedDirs.kenv),
    ensureDir(isolatedDirs.scripts),
  ]);

  return isolatedDirs;
}

/**
 * Isolated test execution with environment variable override
 */
async function collectEventsIsolated(
  duration: number,
  action: (events: TestEvent[], dirs: any) => Promise<void>,
  testName: string,
): Promise<TestEvent[]> {
  const isolatedDirs = await createIsolatedTestDirs(testName);

  // Override environment variables for this test
  const originalKIT = process.env.KIT;
  const originalKENV = process.env.KENV;
  process.env.KIT = isolatedDirs.kit;
  process.env.KENV = isolatedDirs.kenv;

  try {
    // Execute test logic...
    return events;
  } finally {
    // Restore environment variables
    process.env.KIT = originalKIT;
    process.env.KENV = originalKENV;
    await isolatedDirs.cleanup();
  }
}
```

### Test Isolation Strategy
Choose isolation level based on test requirements:

- **Shared test directory**: Fast, but potential cross-test interference
- **Isolated directories**: Slower setup, but complete isolation
- **Environment variable override**: Essential for file system tests

### Timing Guidelines
Adjust timeouts based on execution context:

```typescript
// ❌ Fixed timing - breaks under load
it('should detect file changes', async () => {
  // Always fails when system is busy
  const events = await collectEvents(500, ...);
}, 3000);

// ✅ Load-aware timing
it('should detect file changes', async () => {
  const events = await collectEventsIsolated(
    1500, // Longer collection time for concurrent environment
    async (events, dirs) => {
      await writeFile(filePath, content);
      // Extra wait for file system under load
      await new Promise(resolve => setTimeout(resolve, 300));
    },
    'test-name'
  );
}, 8000); // Longer overall timeout
```

## Common Pitfalls

1. **Missing `addListener` in powerMonitor** - Electron uses both `on` and `addListener`
2. **Incomplete electron-log transports** - Must include `ipc` transport
3. **Type mismatches in test objects** - Include all required Script properties
4. **Mock state bleeding between tests** - Timing of `vi.clearAllMocks()`
5. **Integration test failures** - Skip tests requiring external setup
6. **Resource contention in parallel tests** - Move complex operations to sequential execution
7. **Fixed timing assumptions** - Tests fail under system load, use load-aware timeouts
8. **Ignoring test environment differences** - `pnpm test` vs individual files require different strategies
9. **Over-engineering solutions** - Sometimes moving tests is better than complex timing fixes
10. **Insufficient isolation** - File system tests interfere without proper directory isolation

## Problem-Solving Strategy

When tests fail intermittently:

1. **First, try the obvious solution** - Move timing-sensitive tests to sequential execution
2. **Increase timeouts progressively** - 500ms → 1500ms → 2000ms until stable
3. **Add load-aware waits** - Extra delays between file operations under concurrent load
4. **Use isolated directories** - Prevent cross-test contamination
5. **Check execution context** - Individual vs full test suite may need different approaches

## Performance Optimization Results

### Actual Optimization Case Study (Chokidar Tests)
- **Original**: 30.9 seconds, multiple failures
- **After timing optimization**: 17.0 seconds, 100% passing
- **After strategic test placement**: 18.1 seconds, 100% reliable
- **Total improvement**: 42% faster + 100% reliability

### Key Strategies That Worked
1. **Strategic test categorization** - Parallel vs sequential based on complexity
2. **Load-resilient timing** - Timeouts that work under system load
3. **Isolated directory pattern** - Complete test isolation
4. **Pragmatic problem solving** - Move problematic tests rather than fix timing

## Success Metrics

A well-optimized test suite should achieve:
- ✅ 75+ tests passing (expanded coverage)
- ✅ ~21 second execution time (full suite)
- ✅ 100% reliability under both individual and concurrent execution
- ✅ Clean exit code (0) in all environments
- ✅ Strategic test placement (parallel vs sequential)
- ✅ Load-resilient timing patterns
- ✅ Comprehensive file system coverage with proper isolation

Focus on **reliability over speed** - better to have slightly slower tests that always pass than fast tests that fail intermittently.

## Benchmark Testing and Naming Conventions

### File Naming Standards

#### ✅ `.bench.ts` - Pure Performance Benchmarks
- Industry standard for dedicated benchmark files
- Uses `bench()` functions for performance measurement
- Separate from regular test suite
- Run with: `pnpm vitest bench file.bench.ts --run`
- Requires vitest config to include `bench` in file patterns

#### ✅ `.test.ts` - Performance Tests with Assertions
- Performance validation using `it()` + `expect()`
- Part of regular test suite with performance assertions
- Includes detailed console output + validation
- Run with: `pnpm vitest run file.test.ts`

### Vitest Configuration for Benchmarks
Update [vitest.config.ts](mdc:vitest.config.ts) to include benchmark files:
```typescript
test: {
  include: ['**/*.{test,spec,bench}.{js,mjs,cjs,ts,mts,cts,jsx,tsx}'],
  benchmark: {
    // Add tinybench options here if needed
  }
}
```

### Benchmark Execution Commands
```bash
# Pure benchmarks (single-run, not watch mode)
pnpm vitest bench src/main/file.bench.ts --run

# Performance tests with assertions
pnpm vitest run src/main/file.test.ts

# Both together via script
node scripts/bench-search.js
```

## Search Performance Testing

### Large Dataset Testing (10,000+ Choices)
For testing search performance with realistic datasets:

```typescript
// Generate realistic mock choices
function generateMockChoices(count: number): Choice[] {
  const categories = ['File Operations', 'Git Tools', 'Development'];
  const prefixes = ['Quick', 'Advanced', 'Simple', 'Super'];
  const actions = ['Manager', 'Tool', 'Helper', 'Runner'];

  return Array.from({ length: count }, (_, i) => ({
    id: `choice-${i}`,
    name: `${prefixes[i % prefixes.length]} ${actions[i % actions.length]} ${i}`,
    keyword: `keyword${i}`,
    group: categories[i % categories.length],
    // Add variety: 10% shortcodes, 1% info choices, 2% hidden
    info: i % 100 === 0,
    hideWithoutInput: i % 50 === 0,
  }));
}

// Performance measurement utility
class PerformanceTracker {
  private measurements: Map<string, number[]> = new Map();

  startTimer(operation: string): () => number {
    const start = performance.now();
    return () => {
      const duration = performance.now() - start;
      if (!this.measurements.has(operation)) {
        this.measurements.set(operation, []);
      }
      this.measurements.get(operation)!.push(duration);
      return duration;
    };
  }

  getStats(operation: string) {
    const times = this.measurements.get(operation) || [];
    if (times.length === 0) return null;

    const sorted = times.slice().sort((a, b) => a - b);
    return {
      avg: times.reduce((a, b) => a + b, 0) / times.length,
      min: sorted[0],
      max: sorted[sorted.length - 1],
      p95: sorted[Math.floor(sorted.length * 0.95)],
    };
  }
}
```

### Search Performance Benchmarks
Essential benchmarks to include:

```typescript
// Progressive typing simulation
bench('Progressive typing simulation', () => {
  const progressiveTerms = ['f', 'fi', 'fil', 'file', 'file m'];
  progressiveTerms.forEach(term => measureSearch(term));
});

// Real-world usage patterns
bench('Real-world usage pattern', () => {
  const searchTerms = ['quick', 'file', 'git', '', ' ', 'nonexistent'];
  searchTerms.forEach(term => measureSearch(term));
});

// Scaling tests
it('should benchmark different choice set sizes', () => {
  const sizes = [1000, 2500, 5000, 7500, 10000];
  sizes.forEach(size => {
    const subset = choices.slice(0, size);
    setChoices(mockPrompt, subset, { preload: false });
    const result = measureSearch('file manager');
    expect(result.duration).toBeLessThan(size * 0.02); // Max 0.02ms per choice
  });
});
```

### Performance Assertions and Targets
Set realistic performance targets:

```typescript
// Individual search performance
expect(result.duration).toBeLessThan(100); // Should be under 100ms

// Average performance across many searches
expect(avgDuration).toBeLessThan(50); // Average under 50ms

// Memory usage validation
expect(memoryDiff.heapUsed).toBeLessThan(200 * 1024 * 1024); // Under 200MB growth

// Scaling requirements
expect(slowSearches.length).toBeLessThan(results.length * 0.1); // <10% slow searches
```

## Integration Testing Patterns

### UI Input → IPC → Search Flow Testing
Test the complete user interaction flow:

```typescript
describe('UI Input Integration', () => {
  // Test Channel.INPUT message handling
  it('should handle user typing via Channel.INPUT', () => {
    const inputMessage: InputMessage = {
      input: 'file manager',
      from: 'user-typing'
    };

    // Simulate IPC message
    handleInputMessage(mockPrompt, inputMessage);

    // Verify search was invoked
    expect(mockInvokeSearch).toHaveBeenCalledWith(
      mockPrompt,
      'file manager',
      'user-typing'
    );

    // Verify results sent back
    expect(mockSendToPrompt).toHaveBeenCalledWith(
      Channel.SET_SCORED_CHOICES,
      expect.any(Array)
    );
  });

  // Test search state management
  it('should update search state correctly', () => {
    const choices = generateMockChoices(1000);
    setChoices(mockPrompt, choices, { preload: false });

    // Test hasGroup setting for info choices
    mockPrompt.kitSearch.hasGroup = true;

    invokeSearch(mockPrompt, 'test', 'integration');

    expect(mockPrompt.kitSearch.input).toBe('test');
    expect(mockPrompt.kitSearch.choices).toHaveLength(1000);
  });
});
```

### End-to-End Search Integration
Simulate complete user workflows:

```typescript
// Helper to simulate complete UI → IPC → Search flow
const simulateUserTyping = (
  input: string,
  choices: Choice[],
  options: { mode?: Mode; ui?: UI; expectSearch?: boolean } = {}
) => {
  // Setup choices
  mockPrompt.kitSearch.choices = choices;
  mockPrompt.kitSearch.hasGroup = choices.some(c => !!c.group);

  // Mock QuickScore for realistic search
  const mockQs = {
    search: vi.fn((searchInput: string) => {
      return choices
        .filter(choice => {
          if (choice.hideWithoutInput && (!searchInput || searchInput.trim() === '')) {
            return false;
          }
          return choice.name?.toLowerCase().includes(searchInput.toLowerCase()) ||
                 choice.keyword?.toLowerCase().includes(searchInput.toLowerCase()) ||
                 choice.info === true;
        })
        .map(choice => ({
          item: choice,
          score: 0.8,
          matches: { name: [[0, searchInput.length]] },
          _: ''
        }));
    })
  };
  mockPrompt.kitSearch.qs = mockQs as any;

  // Execute search
  invokeSearch(mockPrompt, input, 'test');

  // Return results for analysis
  const scoredChoicesMessage = sentMessages.find(m => m.channel === Channel.SET_SCORED_CHOICES);
  return (scoredChoicesMessage?.data as ScoredChoice[]) || [];
};
```

### Search-Specific Mock Requirements
Essential mocks for search testing:

```typescript
// Required search mocks
vi.mock('./search', () => ({
  invokeSearch: vi.fn(),
  setChoices: vi.fn(),
  setShortcodes: vi.fn(),
}));

vi.mock('./messages', () => ({
  cacheChoices: vi.fn()
}));

vi.mock('./state', () => ({
  kitCache: {
    choices: [],
    scripts: [],
    triggers: new Map(),
    keywords: new Map(),
    shortcodes: new Map(),
  },
  kitState: {
    kenvEnv: {
      KIT_SEARCH_MAX_ITERATIONS: '3',
      KIT_SEARCH_MIN_SCORE: '0.6',
    },
  },
}));

// Fix lodash debounce mock for search
vi.mock('lodash-es', () => ({
  debounce: vi.fn((fn) => {
    const mockDebounced = vi.fn(fn) as any;
    mockDebounced.cancel = vi.fn();
    return mockDebounced;
  }),
}));
```

### Search Test Object Requirements
Ensure complete test objects for search functionality:

```typescript
// Complete KitPrompt mock for search testing
const mockPrompt: KitPrompt = {
  ui: UI.arg,
  pid: 12345,
  scriptPath: '/test/script.ts',
  sendToPrompt: mockSendToPrompt,
  kitSearch: {
    input: '',
    inputRegex: undefined,
    keyword: '',
    keywordCleared: false,
    generated: false,
    flaggedValue: '',
    choices: [],
    scripts: [],
    qs: null,
    hasGroup: false, // Critical for info choice handling
    keys: ['name', 'keyword', 'tag'],
    keywords: new Map(),
    triggers: new Map(),
    postfixes: new Map(),
    shortcodes: new Map(),
  },
  flagSearch: {
    input: '',
    choices: [],
    hasGroup: false,
    qs: null,
  },
  updateShortcodes: vi.fn(),
} as unknown as KitPrompt;
```

## Performance Testing Success Metrics

### Search Performance Targets
- ✅ Average search time < 50ms (10,000 choices)
- ✅ Individual searches < 100ms
- ✅ Progressive typing responsive (< 75ms average)
- ✅ Memory growth < 200MB during intensive testing
- ✅ <10% of searches should be considered "slow" (>100ms)
- ✅ Scaling: Max 0.02ms per choice

### Integration Testing Coverage
- ✅ UI input → IPC → Search → Results flow
- ✅ Search state management (hasGroup, choices, etc.)
- ✅ Info choice prioritization
- ✅ Hide-without-input behavior
- ✅ Empty/whitespace input handling
- ✅ Progressive typing simulation
- ✅ Memory leak prevention

### Example Performance Test Files
- [search-performance.bench.ts](mdc:src/main/search-performance.bench.ts) - Pure benchmarks
- [search-performance.test.ts](mdc:src/main/search-performance.test.ts) - Performance tests with assertions
- [search-integration.test.ts](mdc:src/main/search-integration.test.ts) - End-to-end integration
- [ipc-input.test.ts](mdc:src/main/ipc-input.test.ts) - IPC message handling