Cursor rule

1---
2alwaysApply: false
3---
4 
5# Property-Based Testing Rules
6 
7## Overview
8 
9These property-based testing (PBT) rules are cross-cutting constraints that apply across applicable AI-DLC phases. They ensure that code with identifiable properties is tested using property-based techniques, complementing (not replacing) traditional example-based tests.
10 
11Property-based testing defines invariants that must hold for all valid inputs, then uses a framework to generate random inputs and search for counterexamples. When a failure is found, the framework shrinks the input to a minimal reproducing case. This approach uncovers edge cases and subtle bugs that example-based testing routinely misses.
12 
13**Enforcement**: At each applicable stage, the model MUST verify compliance with these rules before presenting the stage completion message to the user.
14 
15### Blocking PBT Finding Behavior
16 
17A **blocking PBT finding** means:
181. The finding MUST be listed in the stage completion message under a "PBT Findings" section with the PBT rule ID and description
192. The stage MUST NOT present the "Continue to Next Stage" option until all blocking findings are resolved
203. The model MUST present only the "Request Changes" option with a clear explanation of what needs to change
214. The finding MUST be logged in `aidlc-docs/audit.md` with the PBT rule ID, description, and stage context
22 
23If a PBT rule is not applicable to the current project or unit (e.g., PBT-06 when no stateful components exist), mark it as **N/A** in the compliance summary — this is not a blocking finding.
24 
25### Default Enforcement
26 
27All rules in this document are **blocking** by default. If any rule's verification criteria are not met, it is a blocking PBT finding — follow the blocking finding behavior defined above.
28 
29### Partial Enforcement Mode
30 
31If the user selected **Partial** enforcement during opt-in, only rules PBT-02, PBT-03, PBT-07, PBT-08, and PBT-09 are enforced. All other rules are treated as advisory (non-blocking). Log the enforcement mode in `aidlc-docs/aidlc-state.md` under `## Extension Configuration`.
32 
33### Verification Criteria Format
34 
35Verification items in this document are plain bullet points describing compliance checks. Each item should be evaluated as compliant or non-compliant during review.
36 
37---
38 
39## Rule PBT-01: Property Identification During Design
40 
41**Rule**: Every unit containing business logic, data transformations, or algorithmic operations MUST be analyzed for testable properties during the Functional Design stage. The analysis MUST identify which of the following property categories apply:
42 
43| Category | Description | Example |
44|---|---|---|
45| Round-trip | An operation paired with its inverse yields the original value | serialize → deserialize = identity |
46| Invariant | A transformation preserves some measurable characteristic | sort preserves collection size and elements |
47| Idempotence | Applying an operation twice yields the same result as once | dedup(dedup(list)) = dedup(list) |
48| Commutativity | Different operation orderings produce the same result | add(a, b) = add(b, a) |
49| Oracle | A reference implementation or simplified model can verify results | optimized algorithm vs brute-force |
50| Induction | A property proven for smaller inputs extends to larger ones | recursive structures, divide-and-conquer |
51| Easy verification | The result is hard to compute but easy to check | maze solver output can be walked to verify |
52 
53The identified properties MUST be documented in the functional design artifacts for the unit, and carried forward into code generation as PBT test requirements.
54 
55**Verification**:
56- Functional design artifacts include a "Testable Properties" section listing identified properties per component
57- Each identified property references one of the categories above
58- Components with no identifiable properties are explicitly marked as "No PBT properties identified" with a brief rationale
59- The property list is referenced during code generation planning
60 
61---
62 
63## Rule PBT-02: Round-Trip Properties
64 
65**Rule**: Any operation that has a logical inverse MUST have a property-based test verifying the round-trip. This includes but is not limited to:
66- Serialization / deserialization (JSON, XML, Protobuf, binary formats)
67- Encoding / decoding (Base64, URL encoding, compression)
68- Parsing / formatting (date parsing, number formatting, template rendering with structured input)
69- Encryption / decryption (where key is available)
70- Database write / read (for the data transformation layer, not the I/O itself)
71- Any pair of functions where `f_inverse(f(x)) = x` for all valid `x`
72 
73The property-based test MUST generate random valid inputs using a domain-appropriate generator (see PBT-07) and assert that the round-trip produces a value equal to the original input.
74 
75**Verification**:
76- Every serialization/deserialization pair has a round-trip property test
77- Every encoding/decoding pair has a round-trip property test
78- Every parsing/formatting pair has a round-trip property test (or documents why the transformation is lossy)
79- Round-trip tests use generated inputs, not hardcoded examples
80- Lossy transformations (e.g., float formatting with precision loss) document the acceptable deviation and test within tolerance
81 
82---
83 
84## Rule PBT-03: Invariant Properties
85 
86**Rule**: Functions with documented invariants MUST have property-based tests verifying those invariants hold across generated inputs. Common invariants include:
87- **Size preservation**: output collection has the same size as input (e.g., map, sort)
88- **Element preservation**: output contains exactly the same elements as input, possibly reordered (e.g., sort, shuffle)
89- **Ordering guarantees**: output satisfies an ordering constraint (e.g., sort produces non-decreasing order)
90- **Range constraints**: output values fall within a defined range (e.g., normalize produces values in [0, 1])
91- **Type preservation**: output type matches expected type for all valid inputs
92- **Business rule invariants**: domain-specific rules that must always hold (e.g., "account balance never goes negative after a valid transaction", "discount never exceeds item price")
93 
94**Verification**:
95- Each documented invariant has a corresponding property-based test
96- Invariant tests generate a wide range of inputs including boundary values
97- Business rule invariants identified in functional design are covered by PBT
98- Invariant tests do not duplicate exact assertions from example-based tests — they test the general rule, not specific cases
99 
100---
101 
102## Rule PBT-04: Idempotency Properties
103 
104**Rule**: Any operation that claims or requires idempotency MUST have a property-based test proving it. The test MUST verify that `f(f(x)) = f(x)` for all valid generated inputs. This applies to:
105- API endpoints documented as idempotent (PUT, DELETE)
106- Data normalization or sanitization functions
107- Cache population operations
108- Deduplication logic
109- Configuration application (applying config twice should not change state)
110- Message processing in at-least-once delivery systems
111 
112**Verification**:
113- Every operation documented as idempotent has a PBT asserting `f(f(x)) = f(x)`
114- Idempotency tests use domain-appropriate generators (not just primitives)
115- For stateful operations, the test verifies observable state equivalence after single vs repeated application
116 
117---
118 
119## Rule PBT-05: Oracle and Model-Based Testing
120 
121**Rule**: When a reference implementation, simplified model, or known-correct algorithm exists, property-based tests MUST compare the system under test against the oracle. This applies to:
122- Optimized algorithms replacing a known brute-force version
123- Refactored code replacing legacy implementations
124- Parallel/concurrent implementations compared against sequential versions
125- Custom implementations of well-known algorithms (sorting, searching, graph traversal)
126- New query engines compared against a reference database
127 
128The property-based test MUST generate random valid inputs and assert that the system under test produces equivalent results to the oracle for all generated inputs.
129 
130**Verification**:
131- When a reference implementation exists (or can be trivially written), an oracle PBT is present
132- Oracle tests generate diverse inputs covering normal, boundary, and adversarial cases
133- Equivalence is defined precisely (exact equality, structural equality, or documented tolerance)
134- If no oracle exists, this rule is marked N/A with rationale
135 
136---
137 
138## Rule PBT-06: Stateful Property Testing
139 
140**Rule**: Components that manage mutable state MUST be evaluated for stateful property testing. Stateful PBT generates random sequences of commands (operations) against the system and verifies that invariants hold after each step. This applies to:
141- In-memory caches and data stores
142- State machines and workflow engines
143- Queue and buffer implementations
144- Session management systems
145- Shopping carts, order pipelines, and similar stateful business objects
146- Any component where the result of an operation depends on prior operations
147 
148Stateful PBT MUST:
149- Define a simplified model (reference state) that mirrors the system under test
150- Generate random sequences of valid commands (add, remove, update, query, etc.)
151- Execute each command against both the real system and the model
152- Assert that observable state or query results match between system and model after each command
153- Test sequences of varying lengths, including empty sequences
154 
155**Verification**:
156- Stateful components identified in functional design have stateful PBT or document why it is not applicable
157- A simplified model is defined for comparison
158- Command generators produce valid operation sequences with realistic parameter distributions
159- Invariants are checked after each command in the sequence, not just at the end
160- If no stateful components exist, this rule is marked N/A
161 
162---
163 
164## Rule PBT-07: Generator Quality
165 
166**Rule**: Property-based tests MUST use domain-specific generators that produce realistic, structured inputs — not just primitive types. Poor generators (e.g., random strings for email fields, unbounded integers for age fields) produce meaningless test cases and miss real bugs.
167 
168Generator requirements:
169- **Domain types**: Custom generators MUST be created for domain objects (e.g., User, Order, Transaction) that respect business constraints (valid email format, positive amounts, valid date ranges)
170- **Constrained primitives**: Numeric generators MUST be constrained to realistic ranges where the domain requires it
171- **Structured data**: Generators for complex inputs (nested objects, lists of domain objects) MUST produce structurally valid data
172- **Edge case inclusion**: Generators SHOULD be configured to include boundary values (empty collections, zero, maximum values, Unicode strings) alongside normal values
173- **Reusability**: Domain generators SHOULD be defined as reusable test utilities, not duplicated across test files
174 
175**Verification**:
176- No PBT uses only raw primitive generators (e.g., `st.integers()` alone) for domain-typed parameters
177- Custom generators exist for domain objects used in PBT
178- Generators respect documented business constraints (e.g., positive amounts, valid formats)
179- Generator definitions are centralized and reusable where multiple tests share the same domain types
180 
181---
182 
183## Rule PBT-08: Shrinking and Reproducibility
184 
185**Rule**: All property-based tests MUST support shrinking and deterministic reproducibility.
186 
187- **Shrinking**: When a property fails, the PBT framework MUST automatically reduce the failing input to a minimal reproducing case. Tests MUST NOT disable or bypass the framework's shrinking mechanism unless there is a documented technical reason (e.g., shrinking is incompatible with external service calls in integration tests).
188- **Reproducibility**: Every PBT run MUST be reproducible via a seed value. The seed MUST be logged on failure so that the exact failing scenario can be replayed. CI configurations MUST either use a fixed seed for deterministic runs or log the random seed on every run for post-failure reproduction.
189- **CI integration**: PBT MUST be included in the project's CI pipeline. Flaky PBT failures (tests that pass on retry without code changes) MUST be investigated, not suppressed.
190 
191**Verification**:
192- PBT framework's shrinking is enabled (not overridden or disabled)
193- Test output on failure includes the seed value and the shrunk minimal failing input
194- CI configuration logs the seed for every PBT run or uses a fixed seed
195- No PBT is excluded from CI without documented justification
196- Flaky PBT failures are tracked and investigated, not silently retried
197 
198---
199 
200## Rule PBT-09: Framework Selection
201 
202**Rule**: The project MUST select and configure an appropriate property-based testing framework for its primary language(s). The framework MUST support:
203- Custom generators / strategies for domain types
204- Automatic shrinking of failing cases
205- Seed-based reproducibility
206- Integration with the project's existing test runner
207 
208Recommended frameworks by language (non-exhaustive):
209 
210| Language | Framework | Notes |
211|---|---|---|
212| Python | Hypothesis | Mature, excellent shrinking, Django integration |
213| JavaScript / TypeScript | fast-check | Integrates with Jest, Vitest, Mocha |
214| Java | jqwik | JUnit 5 integration, stateful testing support |
215| Kotlin | Kotest Property Testing | Kotest framework integration |
216| Scala | ScalaCheck | SBT integration, widely adopted |
217| Rust | proptest | Macro-based, good shrinking |
218| Go | rapid | Lightweight, idiomatic Go |
219| Haskell | QuickCheck | The original PBT framework |
220| C# / .NET | FsCheck | Works with xUnit, NUnit |
221| Erlang / Elixir | PropEr / StreamData | OTP-aware, stateful testing |
222 
223The selected framework MUST be documented in the tech stack decisions and included as a project dependency.
224 
225**Verification**:
226- A PBT framework is selected and documented in tech stack decisions
227- The framework is included in project dependencies (package.json, pom.xml, requirements.txt, etc.)
228- The framework supports custom generators, shrinking, and seed-based reproducibility
229- If the project uses multiple languages, each language with PBT-applicable code has a framework selected
230 
231---
232 
233## Rule PBT-10: Complementary Testing Strategy
234 
235**Rule**: Property-based tests MUST complement, not replace, example-based tests. The two approaches serve different purposes:
236 
237- **Example-based tests**: Document specific known scenarios, regression cases, and business-critical edge cases with explicit expected values. They serve as executable documentation of concrete behavior.
238- **Property-based tests**: Verify general invariants across a wide input space. They find unknown edge cases and validate that properties hold universally.
239 
240Requirements:
241- Critical business scenarios identified in user stories or requirements MUST have explicit example-based tests, even if a PBT covers the same property
242- PBT MUST NOT be the sole test for any business-critical path — at least one example-based test must pin the expected behavior for key scenarios
243- When a PBT discovers a failing case, the shrunk minimal example SHOULD be added as a permanent example-based regression test
244- Test documentation MUST clearly distinguish between example-based and property-based tests (separate test files, test classes, or clearly named test functions)
245 
246**Verification**:
247- Business-critical paths have both example-based and property-based tests
248- PBT is not used as the only test coverage for any critical feature
249- Test files or test classes clearly separate or label PBT vs example-based tests
250- Regression tests from PBT-discovered failures are captured as permanent example-based tests
251 
252---
253 
254## Enforcement Integration
255 
256These rules are cross-cutting constraints that apply to the following AI-DLC stages:
257 
258| Stage | Applicable Rules | Enforcement |
259|---|---|---|
260| Functional Design | PBT-01 | Property identification must appear in design artifacts |
261| NFR Requirements | PBT-09 | Framework selection must be included in tech stack decisions |
262| Code Generation (Planning) | PBT-01 through PBT-10 | Code generation plan must include PBT test steps for identified properties |
263| Code Generation (Generation) | PBT-02 through PBT-08, PBT-10 | Generated tests must include PBT alongside example-based tests |
264| Build and Test | PBT-08 | Test execution instructions must include PBT with seed logging and CI integration |
265 
266At each applicable stage:
267- Evaluate all PBT rule verification criteria against the artifacts produced
268- Include a "PBT Compliance" section in the stage completion summary listing each rule as compliant, non-compliant, or N/A
269- If any rule is non-compliant, this is a blocking PBT finding — follow the blocking finding behavior defined in the Overview
270- Include PBT rule references in design documentation and test instructions
271 
272---
273 
274## Appendix: Property Category Quick Reference
275 
276For developers and AI models identifying properties during Functional Design (PBT-01):
277 
278| Pattern Name | Formal Term | Test Shape | When to Use |
279|---|---|---|---|
280| There and back again | Invertible function | `f_inv(f(x)) == x` | Serialization, encoding, parsing |
281| Some things never change | Invariant | `measure(f(x)) == measure(x)` | Sort, map, filter, transform |
282| The more things change, the more they stay the same | Idempotence | `f(f(x)) == f(x)` | Normalization, dedup, cache writes |
283| Different paths, same destination | Commutativity | `f(g(x)) == g(f(x))` | Arithmetic, set operations, independent transforms |
284| Solve a smaller problem first | Structural induction | Property on `x` implies property on `x + element` | Recursive structures, lists, trees |
285| Hard to prove, easy to verify | Verification | `verify(solve(x)) == true` | Solvers, optimizers, search algorithms |
286| The test oracle | Reference comparison | `f(x) == oracle(x)` | Optimized vs brute-force, refactored vs legacy |
287 
288Source: Property category taxonomy adapted from Scott Wlaschin's "Choosing properties for property-based testing" ([fsharpforfunandprofit.com](https://fsharpforfunandprofit.com/posts/property-based-testing-2/)).
289