# Production Test Plan: Multi-User, Multi-Run Trading Engine ## Scope - Backend API, engine, broker, DB constraints/triggers - No UI tests ## Environment - `BASE_URL` for API - `DB_DSN` for PostgreSQL ## Test Case Matrix | ID | Category | Steps | Expected Result | DB Assertions | |---|---|---|---|---| | RL-01 | Lifecycle | Create RUNNING run; insert engine_state | Insert succeeds | engine_state row exists | | RL-02 | Lifecycle | STOPPED run; insert engine_state | Insert rejected | no new rows | | RL-03 | Lifecycle | ERROR run; insert engine_state | Insert rejected | no new rows | | RL-04 | Lifecycle | STOPPED -> RUNNING | Update rejected | status unchanged | | RL-05 | Lifecycle | Insert 2nd RUNNING run same user | Insert rejected | unique violation | | RL-06 | Lifecycle | Insert row with user_id/run_id mismatch | Insert rejected | FK violation | | RL-07 | Lifecycle | Delete strategy_run | Cascades delete | no orphans | | RL-08 | Lifecycle | Start run via API | status RUNNING | engine_status row | | RL-09 | Lifecycle | Stop run via API | status STOPPED | no new writes allowed | | OR-01 | Orders | place market BUY | FILLED | order+trade rows | | OR-02 | Orders | place order qty=0 | REJECTED | no trade | | OR-03 | Orders | place order qty<0 | REJECTED | no trade | | OR-04 | Orders | cash < cost | REJECTED | no trade | | OR-05 | Orders | limit order below price | PENDING | no trade | | OR-06 | Orders | stop run | pending orders canceled | status updated | | MT-01 | Market time | market closed SIP | no execution | no order/trade | | MT-02 | Market time | scheduled SIP | executes once | event_ledger row | | MT-03 | Market time | repeat same logical_time | no dup | unique constraint | | MT-04 | Market time | rebalance once | one event | event_ledger | | IR-01 | Idempotency | tick T twice | no duplicate rows | counts unchanged | | IR-02 | Idempotency | crash mid-tx | rollback | no partial rows | | IR-03 | Idempotency | retry same tick | exactly-once | counts unchanged | | RK-01 | Risk | insufficient cash | reject | no trade | | RK-02 | Risk | max position | reject | no trade | | RK-03 | Risk | forbidden symbol | reject | no trade | | LG-01 | Ledger | order->trade->position | consistent | FK + counts | | LG-02 | Ledger | equity = cash + mtm | matches | tolerance | | MU-01 | Multi-user | two users run | isolated | no cross rows | | MU-02 | Multi-run | same user two runs | isolated | run_id separation | | DR-01 | Determinism | replay same feed | identical outputs | diff=0 | | CC-01 | Concurrency | tick vs stop | no partial writes | consistent | | CC-02 | Concurrency | two ticks same time | dedupe | unique constraint | | FI-01 | Failure | injected error | rollback | no partial rows | ## Detailed Cases (Mandatory) ### Run Lifecycle & State Integrity - RL-02 STOPPED rejects writes - Steps: create STOPPED run; attempt inserts into engine_state, engine_status, paper_order, paper_trade, mtm_ledger, event_ledger, paper_equity_curve - Expected: each insert fails - DB assertions: no new rows for the run - RL-03 ERROR rejects writes - Steps: create ERROR run; attempt same inserts as RL-02 - Expected: each insert fails - DB assertions: no new rows for the run - RL-04 STOPPED cannot revive - Steps: create STOPPED run; UPDATE status=RUNNING - Expected: update rejected - DB assertions: status remains STOPPED - RL-05 One RUNNING run per user - Steps: insert RUNNING run; insert another RUNNING run for same user - Expected: unique violation - DB assertions: only one RUNNING row - RL-06 Composite FK enforcement - Steps: create run for user A and run for user B; attempt insert with user A + run B - Expected: FK violation - DB assertions: no inserted row - RL-07 Cascades - Steps: insert run and child rows; delete strategy_run - Expected: child rows removed - DB assertions: zero rows for run_id in children ### Order Execution Semantics - OR-01 Market order fill - Steps: RUNNING run; place market BUY via broker tick - Expected: order FILLED, trade created, position updated, equity updated - DB assertions: rows in paper_order, paper_trade, paper_position, paper_equity_curve - OR-10/11 Reject invalid qty - Steps: place order qty=0 and qty<0 - Expected: REJECTED - DB assertions: no trade rows for that run - OR-12 Insufficient cash - Steps: initial_cash too low; place buy above cash - Expected: REJECTED - DB assertions: no trade rows for that run ### Market Time & Session Logic - MT-01 Market closed SIP - Steps: call try_execute_sip with market_open=False - Expected: no order/trade - DB assertions: counts unchanged - MT-03 Idempotent logical_time - Steps: execute tick twice at same logical_time - Expected: no duplicate orders/trades/mtm/equity/events - DB assertions: unique logical_time counts <= 1 ### Engine Idempotency & Crash Recovery - IR-02 Crash mid-transaction - Steps: open transaction, write state, raise exception - Expected: rollback, no partial rows - DB assertions: counts unchanged - IR-03 Retry same tick - Steps: rerun tick for same logical_time - Expected: exactly-once side effects - DB assertions: no duplicates ### Multi-User / Multi-Run Isolation - MU-01 Two users, isolated runs - Steps: create two users + runs; execute tick for each - Expected: separate data - DB assertions: rows scoped by (user_id, run_id) - MU-02 Same user, two runs - Steps: run A tick; stop; run B tick - Expected: separate data - DB assertions: counts separate per run_id ### Deterministic Replay - DR-01 Replay determinism - Steps: run N ticks with fixed feed for run A and run B - Expected: identical ledgers (excluding surrogate IDs) - DB assertions: normalized rows equal ### Concurrency & Race Conditions - CC-01 Tick vs Stop race - Steps: lock run row in thread A; update status in thread B - Expected: consistent final status; no partial writes - DB assertions: no half-written rows - CC-02 Two ticks same logical_time - Steps: run two ticks concurrently with same logical_time - Expected: dedupe - DB assertions: SIP_EXECUTED count <= 1 ### Failure Injection - FI-01 Crash after trade before MTM - Steps: execute SIP (orders/trades), then log MTM in separate tx - Expected: no MTM before explicit insert; MTM inserts once when run - DB assertions: MTM logical_time count == 1 ## DB Assertions (Post-Run) - Orphan check: - `SELECT COUNT(*) FROM engine_state es LEFT JOIN strategy_run sr ON sr.user_id=es.user_id AND sr.run_id=es.run_id WHERE sr.run_id IS NULL;` - Dedupe check: - `SELECT user_id, run_id, logical_time, COUNT(*) FROM mtm_ledger GROUP BY user_id, run_id, logical_time HAVING COUNT(*) > 1;` - Lifecycle check: - `SELECT user_id, COUNT(*) FROM strategy_run WHERE status='RUNNING' GROUP BY user_id HAVING COUNT(*) > 1;` ## Test Execution Use `pytest -q` from repo root. See `README_test.md`.