Copilot instructions

1---
2applyTo: "eng/helix_xharness.proj,eng/pipelines/**/device-tests*.yml,eng/pipelines/**/stage-device-tests.yml,src/**/DeviceTests/**"
3description: "Guidelines for running and configuring .NET MAUI device tests on Helix infrastructure"
4---
5 
6# Helix Device Tests Guidelines
7 
8This document provides guidance for working with .NET MAUI device tests that run on Helix infrastructure using XHarness.
9 
10## Overview
11 
12.NET MAUI uses [.NET Engineering Services Helix](https://helix.dot.net) with XHarness to run device tests across multiple platforms in parallel. This provides cloud-based device testing infrastructure.
13 
14### Device Test Projects
15 
16- `Controls.DeviceTests` - UI control tests
17- `Core.DeviceTests` - Core framework tests
18- `Graphics.DeviceTests` - Graphics and drawing tests
19- `Essentials.DeviceTests` - Platform API tests
20- `MauiBlazorWebView.DeviceTests` - Blazor WebView tests
21 
22### Available Helix Queues
23 
24Current configuration uses:
25- **iOS**: `osx.15.arm64.maui.open`
26- **Mac Catalyst**: `osx.15.arm64.maui.open`
27- **Android**: `ubuntu.2204.amd64.android.33.open`
28 
29Check available queues at [helix.dot.net](https://helix.dot.net).
30 
31## Key Configuration Files
32 
33| File | Purpose |
34|------|---------|
35| `eng/helix_xharness.proj` | Main Helix configuration - defines scenarios, queues, and work items |
36| `eng/pipelines/common/stage-device-tests.yml` | Pipeline template for device tests |
37| `eng/test-configuration.json` | Test retry configuration |
38 
39## iOS Category Splitting
40 
41For iOS, Controls.DeviceTests heavy categories are split into separate Helix work items. This mirrors the old cake-based approach and enables parallel execution for the slowest tests.
42 
43**How it works:**
441. Heavy categories are defined in `ControlsTestCategoriesToSkipForRestOfTests` property in `helix_xharness.proj`
452. The `ControlsTestCategoriesToRunIndividually` ItemGroup is populated from that property
463. Each heavy category becomes a separate Helix work item
474. All other Controls tests run together in a single "General" work item
485. XHarness passes `--set-env="TestFilter=Category=X"` for individual categories
496. XHarness passes `--set-env="TestFilter=SkipCategories=X;Y;Z"` for the "General" work item
507. Core.DeviceTests runs as a single work item (no splitting)
51 
52**Heavy categories that run separately:**
53- CollectionView, Shell, HybridWebView
54 
55**Keep in sync:** If adding new heavy categories, update the `ControlsTestCategoriesToSkipForRestOfTests` property in `eng/helix_xharness.proj`.
56 
57## Running Device Tests Locally
58 
59### Prerequisites
60 
61From the repository root:
62 
63```bash
64# 1. Restore dotnet tools
65dotnet tool restore
66 
67# 2. Build MSBuild tasks (required)
68./build.sh -restore -build -configuration Release -projects $(PWD)/Microsoft.Maui.BuildTasks.slnf /bl:BuildBuildTasks.binlog -warnAsError false
69 
70# 3. Build device tests
71./build.sh -restore -build -configuration Release /p:BuildDeviceTests=true /bl:BuildDeviceTests.binlog -warnAsError false
72```
73 
74### Using the run-device-tests Skill (Recommended)
75 
76The easiest way to run device tests locally is using the `run-device-tests` skill:
77 
78```bash
79# Run Controls tests on iOS simulator
80pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios
81 
82# Run only Button category tests
83pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform ios -TestFilter "Category=Button"
84 
85# Run on Android emulator
86pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Controls -Platform android -TestFilter "Category=Button"
87 
88# Run on MacCatalyst
89pwsh .github/skills/run-device-tests/scripts/Run-DeviceTests.ps1 -Project Core -Platform maccatalyst
90```
91 
92See `.github/skills/run-device-tests/SKILL.md` for full documentation.
93 
94### Submit to Helix
95 
96Set required environment variables:
97 
98```bash
99export BUILD_REASON=pr
100export BUILD_REPOSITORY_NAME=maui
101export BUILD_SOURCEBRANCH=main
102export SYSTEM_TEAMPROJECT=dnceng
103export SYSTEM_ACCESSTOKEN=''
104```
105 
106Submit tests:
107 
108```bash
109# Android
110./eng/common/msbuild.sh ./eng/helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=android /bl:sendhelix_android.binlog
111 
112# iOS
113./eng/common/msbuild.sh ./eng/helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=ios /bl:sendhelix_ios.binlog
114 
115# Mac Catalyst
116./eng/common/msbuild.sh ./eng/helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=maccatalyst /bl:sendhelix_catalyst.binlog
117```
118 
119### Windows Commands
120 
121```cmd
122set BUILD_REASON=pr
123set BUILD_REPOSITORY_NAME=maui
124set BUILD_SOURCEBRANCH=main
125set SYSTEM_TEAMPROJECT=dnceng
126set SYSTEM_ACCESSTOKEN=
127 
128.\build.cmd -restore -build -configuration Release -projects ".\Microsoft.Maui.BuildTasks.slnf" /bl:BuildBuildTasks.binlog -warnAsError false
129.\build.cmd -restore -build -configuration Release /p:BuildDeviceTests=true /bl:BuildDeviceTests.binlog -warnAsError false
130.\eng\common\msbuild.cmd .\eng\helix_xharness.proj /restore /p:TreatWarningsAsErrors=false /t:Test /p:TargetOS=android /bl:sendhelix.binlog
131```
132 
133### Validate MSBuild Logic Only
134 
135To validate the helix proj without submitting (requires built artifacts):
136 
137```bash
138dotnet msbuild eng/helix_xharness.proj /t:DiscoverTestBundles /p:TargetOS=ios /p:_MauiDotNetTfm=net10.0 /p:RepoRoot=$(pwd)/ -v:n
139```
140 
141## Test Filtering Implementation
142 
143Test category filtering is implemented in `src/Core/tests/DeviceTests.Shared/DeviceTestSharedHelpers.cs`. The `GetExcludedTestCategories()` method reads the `TestFilter` value and converts it to a list of categories to skip.
144 
145### Filter Syntax
146 
147| Format | Description | Example |
148|--------|-------------|---------|
149| `Category=X` | Run only category X (skip all others) | `Category=Button` |
150| `SkipCategories=X,Y,Z` | Skip specific categories | `SkipCategories=Shell,CollectionView` |
151 
152### Platform-Specific Filter Passing
153 
154| Platform | XHarness Argument | How App Reads It |
155|----------|-------------------|------------------|
156| **iOS/MacCatalyst** | `--set-env=TestFilter=...` | `NSProcessInfo.ProcessInfo.Environment["TestFilter"]` |
157| **Android** | `--arg TestFilter=...` | `MauiTestInstrumentation.Current.Arguments.GetString("TestFilter")` |
158| **Windows** | `--filter "Category=..."` | Native vstest filter |
159 
160**Important**: iOS uses `--set-env` (environment variable), while Android uses `--arg` (instrumentation argument). These are NOT interchangeable.
161 
162### Example XHarness Commands with Filters
163 
164```bash
165# iOS - uses --set-env
166xharness apple test --target ios-simulator-64_18.5 --device UDID --set-env=TestFilter=Category=Button ...
167 
168# Android - uses --arg
169xharness android test --package-name com.microsoft.maui.controls.devicetests --arg TestFilter=Category=Button ...
170```
171 
172## Configuration Details
173 
174The `eng/helix_xharness.proj` configuration includes:
175 
176- **Timeouts**: 2-hour work item timeout, 1-hour 15-min test timeout for category splits
177- **Test Discovery**: Automatically discovers test bundles for each scenario
178- **Platform Targeting**: Uses `TargetOS` property (ios, maccatalyst, android)
179- **Queue Selection**: Platform-appropriate Helix queues
180- **XHarness Integration**: Uses XHarness CLI for device orchestration
181 
182### CustomCommands for Category Filtering
183 
184When using category splitting, `CustomCommands` metadata overrides the default xharness invocation:
185 
186```xml
187<CustomCommands>xharness apple test --target "$target" --app "$app" --output-directory "$output_directory" --timeout "$timeout" --launch-timeout "$launch_timeout" --set-env="TestFilter=Category=CategoryName"</CustomCommands>
188```
189 
190**Important**: Keep CustomCommands as a single line. Multi-line commands with `set -ex` can cause parse errors.
191 
192## Troubleshooting
193 
194### Common Issues
195 
1961. **Build failures**: Ensure MSBuild tasks are built first
1972. **Missing devices**: Check queue availability at [helix.dot.net](https://helix.dot.net)
1983. **Authentication**: For CI, ensure proper Azure DevOps access tokens
1994. **Timeouts**: Adjust `TestTimeout` and `WorkItemTimeout` for complex scenarios
2005. **CustomCommands parse errors**: Keep commands on single line, avoid shell constructs like `set -ex`
201 
202### Logging and Diagnostics
203 
204- Use `/bl:filename.binlog` for detailed MSBuild logs
205- Add `-verbosity:diag` for maximum diagnostic output
206- Check Helix job results at the URL provided after submission
207- Individual work item logs available at `https://helix.dot.net/api/2019-06-17/jobs/{jobId}/workitems/{workItemName}/console`
208 
209## Additional Resources
210 
211- [XHarness on Helix Documentation](https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/Readme.md)
212- [Helix SDK Documentation](https://github.com/dotnet/arcade/blob/main/src/Microsoft.DotNet.Helix/Sdk/Readme.md)
213- [Example Helix Run](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1115383&view=results)
214