Cursor rule

1---
2description: Translation workflow and OCR processing pipeline
3globs: ["**/*.cs"]
4alwaysApply: false
5---
6 
7# Translation Workflow and API Integration Guide
8 
9## Translation Pipeline Overview
10 
11The translation process in UGTLive follows this pipeline:
121. **Screen Capture** → 2. **OCR Processing** → 3. **Block Detection** → 4. **Translation** → 5. **Display**
13 
14## Screen Capture
15 
16### Monitor Window ([src/MonitorWindow.xaml.cs](mdc:src/MonitorWindow.xaml.cs))
17- Captures screen region at configurable FPS
18- Uses `System.Drawing` for bitmap operations
19- Sends captured images to OCR services
20- Provides visual feedback of capture area
21 
22### Capture Methods
23```csharp
24// Main capture triggered by timer
25private void Timer_Tick(object sender, EventArgs e)
26// Captures bitmap from screen region
27private Bitmap CaptureScreen(int x, int y, int width, int height)
28```
29 
30## OCR Processing
31 
32### OCR Service Selection
33UGTLive supports six OCR backends configured in [src/ConfigManager.cs](mdc:src/ConfigManager.cs):
34 
351. **Windows OCR** ([src/WindowsOCRManager.cs](mdc:src/WindowsOCRManager.cs))
36   - Uses Windows.Media.Ocr API
37   - Faster but less accurate for some languages
38   - No external dependencies
39 
402. **EasyOCR** (via PythonServicesManager)
41   - Python server running locally
42   - Better accuracy for Asian languages
43   - Requires conda environment setup
44 
453. **MangaOCR** (via PythonServicesManager)
46   - Specialized for vertical Japanese manga text
47   - YOLO-based text detection
48   - Configurable region size and overlap settings
49 
504. **PaddleOCR** (via PythonServicesManager)
51   - Multi-language OCR with 100+ language support
52   - Optional angle classification for rotated text
53   - GPU acceleration support
54 
555. **docTR** (via PythonServicesManager)
56   - Great for non-Asian languages
57   - Document-oriented OCR
58   - High accuracy for printed text
59 
606. **Google Vision** ([src/GoogleVisionOCRService.cs](mdc:src/GoogleVisionOCRService.cs))
61   - Cloud-based OCR service
62   - Requires API key and costs money
63   - High accuracy but not local
64 
65### OCR Data Flow
66```
67Bitmap → OCR Service → List<TextObject> → UniversalBlockDetector
68```
69 
70### Python OCR Services
71All Python OCR services are managed by `PythonServicesManager`:
72- Services discovered automatically on startup
73- Each service runs on its own port
74- Services can be installed/uninstalled via UI
75- Health checks and diagnostics available
76 
77## Block Detection ([src/UniversalBlockDetector.cs](mdc:src/UniversalBlockDetector.cs))
78 
79The UniversalBlockDetector groups individual characters/words/lines into meaningful text blocks:
80 
81### Key Features
82- **Universal Detection**: Handles mixed input types (Words, Lines, Characters)
83- **Intelligent Grouping**: Groups based on proximity, alignment, and reading order
84- **Configurable Thresholds**: Per-OCR method settings for glue distances
85- **Height Similarity**: Prevents merging text with very different sizes
86- **Large Gap Detection**: Splits text at large horizontal gaps
87 
88### Key Parameters
89- **Block Detection Scale**: Controls grouping aggressiveness (0.0 - 1.0)
90- **Horizontal/Vertical Glue**: Per-OCR method glue distances
91- **Height Similarity Threshold**: Percentage for height matching
92- **Settle Time**: Time to wait before processing blocks
93 
94### Grouping Algorithm
951. Sorts text objects by position
962. Groups based on proximity and alignment
973. Merges overlapping blocks
984. Filters by minimum size
995. Handles mixed input types intelligently
100 
101## Translation Services
102 
103### Service Interface ([src/ITranslationService.cs](mdc:src/ITranslationService.cs))
104```csharp
105public interface ITranslationService
106{
107    Task<TranslationResult?> TranslateAsync(
108        string sourceText, 
109        string targetLanguage, 
110        string context);
111}
112```
113 
114### Available Services
115 
116#### Gemini ([src/GeminiTranslationService.cs](mdc:src/GeminiTranslationService.cs))
117- Endpoint: `https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent`
118- Models: gemini-1.5-flash, gemini-1.5-pro, gemini-2.0-flash
119- Supports custom prompts and context
120 
121#### ChatGPT ([src/ChatGptTranslationService.cs](mdc:src/ChatGptTranslationService.cs))
122- Endpoint: `https://api.openai.com/v1/chat/completions`
123- Models: gpt-4o, gpt-4o-mini, o1-preview, o1-mini
124- Supports streaming responses
125- Configurable max completion tokens
126 
127#### Ollama ([src/OllamaTranslationService.cs](mdc:src/OllamaTranslationService.cs))
128- Local endpoint: `http://localhost:11434/api/generate`
129- Models: Various local models (llama, gemma, etc.)
130- Privacy-focused, no cloud dependency
131- Configurable URL and port
132 
133#### Google Translate ([src/GoogleTranslateService.cs](mdc:src/GoogleTranslateService.cs))
134- Can use Cloud API or free web API
135- Auto language mapping support
136- Fast translation for simple use cases
137 
138#### llama.cpp ([src/LlamaCppTranslationService.cs](mdc:src/LlamaCppTranslationService.cs))
139- Local llama.cpp server endpoint
140- Configurable URL and port
141- Privacy-focused local translation
142 
143### Translation Request Format
144All services receive:
145- Source text with position data
146- Target language
147- Previous context (configurable length)
148- Game name for context
149- Custom prompt template
150 
151## Display Systems
152 
153### ChatBox Window ([src/ChatBoxWindow.xaml.cs](mdc:src/ChatBoxWindow.xaml.cs))
154- Overlay window for displaying translations
155- Customizable appearance (color, transparency, font)
156- Auto-scroll and history management
157- Can be positioned anywhere on screen
158 
159### Overlay System
160- Text overlays positioned at original text locations
161- Configurable clear delay
162- Passthrough mode for interaction
163- Multiple overlay modes
164 
165### Text Rendering
166- Uses WPF TextBlock with formatting
167- Supports multiple fonts and sizes
168- Color-coded by translation service
169- Maintains translation history
170 
171## API Key Management
172 
173API keys are stored securely in [src/ConfigManager.cs](mdc:src/ConfigManager.cs):
174- Stored in plain text config files (local only)
175- Never logged (masked in console output)
176- Validated on settings save
177 
178## Context Management
179 
180### Previous Context System
181- Stores recent translations for context
182- Configurable maximum context length
183- Filters small UI elements (buttons, menus)
184- Improves translation accuracy
185 
186### Context Flow
187```
188Previous Translations → Context Buffer → Translation Request → LLM
189```
190 
191## Audio Features
192 
193### Text-to-Speech Services
194- **Google TTS** ([src/GoogleTTSService.cs](mdc:src/GoogleTTSService.cs))
195- **ElevenLabs** ([src/ElevenLabsService.cs](mdc:src/ElevenLabsService.cs))
196- Voice selection dialogs
197- Audio preloading for source/target languages
198 
199### Real-time Audio Transcription
200- Uses OpenAI Realtime API ([src/OpenAIRealtimeAudioService.cs](mdc:src/OpenAIRealtimeAudioService.cs))
201- WebSocket connection for streaming
202- Supports voice activity detection
203 
204### Audio Playback
205- Managed by `AudioPlaybackManager`
206- Preloading via `AudioPreloadService`
207- Queue management and playback control
208 
209## Error Handling
210 
211### Common Error Points
2121. **OCR Failures**: Logged, skips frame
2132. **Translation API Errors**: Displays error in ChatBox or via ErrorPopupManager
2143. **Network Issues**: Retries with exponential backoff
2154. **Invalid API Keys**: Shows settings prompt
216 
217### Logging
218All errors logged via [src/LogManager.cs](mdc:src/LogManager.cs):
219```csharp
220LogManager.Instance.LogError("Error description", exception);
221```
222 
223### Error Popups
224User-friendly error messages via [src/ErrorPopupManager.cs](mdc:src/ErrorPopupManager.cs):
225```csharp
226ErrorPopupManager.ShowError("Title", "Message");
227```
228 
229## Performance Considerations
230 
231### OCR Optimization
232- Configurable capture FPS
233- Region-based capture (not full screen)
234- Caching of unchanged regions
235- Per-OCR method confidence thresholds
236 
237### Translation Optimization
238- Batches small text blocks
239- Caches recent translations
240- Parallel processing where possible
241- Pause OCR while translating option
242 
243### Memory Management
244- Disposes bitmaps after use
245- Limits translation history size
246- Clears old context periodically
247- Audio preloading with size limits
248 
249## Testing Translation Services
250 
251### Manual Testing
2521. Set API key in settings
2532. Select service and model
2543. Use Monitor window to capture text
2554. Check ChatBox for results
2565. Review logs for errors
257 
258### Common Issues
259- **Empty translations**: Check OCR output
260- **Wrong language**: Verify language settings
261- **Slow performance**: Reduce capture area/FPS
262- **API errors**: Validate API key and quota
263- **Python service errors**: Check service diagnostics dialog
264