155 lines
6.7 KiB
Markdown
155 lines
6.7 KiB
Markdown
# Production Test Plan: Multi-User, Multi-Run Trading Engine
|
|
|
|
## Scope
|
|
- Backend API, engine, broker, DB constraints/triggers
|
|
- No UI tests
|
|
|
|
## Environment
|
|
- `BASE_URL` for API
|
|
- `DB_DSN` for PostgreSQL
|
|
|
|
## Test Case Matrix
|
|
|
|
| ID | Category | Steps | Expected Result | DB Assertions |
|
|
|---|---|---|---|---|
|
|
| RL-01 | Lifecycle | Create RUNNING run; insert engine_state | Insert succeeds | engine_state row exists |
|
|
| RL-02 | Lifecycle | STOPPED run; insert engine_state | Insert rejected | no new rows |
|
|
| RL-03 | Lifecycle | ERROR run; insert engine_state | Insert rejected | no new rows |
|
|
| RL-04 | Lifecycle | STOPPED -> RUNNING | Update rejected | status unchanged |
|
|
| RL-05 | Lifecycle | Insert 2nd RUNNING run same user | Insert rejected | unique violation |
|
|
| RL-06 | Lifecycle | Insert row with user_id/run_id mismatch | Insert rejected | FK violation |
|
|
| RL-07 | Lifecycle | Delete strategy_run | Cascades delete | no orphans |
|
|
| RL-08 | Lifecycle | Start run via API | status RUNNING | engine_status row |
|
|
| RL-09 | Lifecycle | Stop run via API | status STOPPED | no new writes allowed |
|
|
| OR-01 | Orders | place market BUY | FILLED | order+trade rows |
|
|
| OR-02 | Orders | place order qty=0 | REJECTED | no trade |
|
|
| OR-03 | Orders | place order qty<0 | REJECTED | no trade |
|
|
| OR-04 | Orders | cash < cost | REJECTED | no trade |
|
|
| OR-05 | Orders | limit order below price | PENDING | no trade |
|
|
| OR-06 | Orders | stop run | pending orders canceled | status updated |
|
|
| MT-01 | Market time | market closed SIP | no execution | no order/trade |
|
|
| MT-02 | Market time | scheduled SIP | executes once | event_ledger row |
|
|
| MT-03 | Market time | repeat same logical_time | no dup | unique constraint |
|
|
| MT-04 | Market time | rebalance once | one event | event_ledger |
|
|
| IR-01 | Idempotency | tick T twice | no duplicate rows | counts unchanged |
|
|
| IR-02 | Idempotency | crash mid-tx | rollback | no partial rows |
|
|
| IR-03 | Idempotency | retry same tick | exactly-once | counts unchanged |
|
|
| RK-01 | Risk | insufficient cash | reject | no trade |
|
|
| RK-02 | Risk | max position | reject | no trade |
|
|
| RK-03 | Risk | forbidden symbol | reject | no trade |
|
|
| LG-01 | Ledger | order->trade->position | consistent | FK + counts |
|
|
| LG-02 | Ledger | equity = cash + mtm | matches | tolerance |
|
|
| MU-01 | Multi-user | two users run | isolated | no cross rows |
|
|
| MU-02 | Multi-run | same user two runs | isolated | run_id separation |
|
|
| DR-01 | Determinism | replay same feed | identical outputs | diff=0 |
|
|
| CC-01 | Concurrency | tick vs stop | no partial writes | consistent |
|
|
| CC-02 | Concurrency | two ticks same time | dedupe | unique constraint |
|
|
| FI-01 | Failure | injected error | rollback | no partial rows |
|
|
|
|
## Detailed Cases (Mandatory)
|
|
|
|
### Run Lifecycle & State Integrity
|
|
- RL-02 STOPPED rejects writes
|
|
- Steps: create STOPPED run; attempt inserts into engine_state, engine_status, paper_order, paper_trade, mtm_ledger, event_ledger, paper_equity_curve
|
|
- Expected: each insert fails
|
|
- DB assertions: no new rows for the run
|
|
- RL-03 ERROR rejects writes
|
|
- Steps: create ERROR run; attempt same inserts as RL-02
|
|
- Expected: each insert fails
|
|
- DB assertions: no new rows for the run
|
|
- RL-04 STOPPED cannot revive
|
|
- Steps: create STOPPED run; UPDATE status=RUNNING
|
|
- Expected: update rejected
|
|
- DB assertions: status remains STOPPED
|
|
- RL-05 One RUNNING run per user
|
|
- Steps: insert RUNNING run; insert another RUNNING run for same user
|
|
- Expected: unique violation
|
|
- DB assertions: only one RUNNING row
|
|
- RL-06 Composite FK enforcement
|
|
- Steps: create run for user A and run for user B; attempt insert with user A + run B
|
|
- Expected: FK violation
|
|
- DB assertions: no inserted row
|
|
- RL-07 Cascades
|
|
- Steps: insert run and child rows; delete strategy_run
|
|
- Expected: child rows removed
|
|
- DB assertions: zero rows for run_id in children
|
|
|
|
### Order Execution Semantics
|
|
- OR-01 Market order fill
|
|
- Steps: RUNNING run; place market BUY via broker tick
|
|
- Expected: order FILLED, trade created, position updated, equity updated
|
|
- DB assertions: rows in paper_order, paper_trade, paper_position, paper_equity_curve
|
|
- OR-10/11 Reject invalid qty
|
|
- Steps: place order qty=0 and qty<0
|
|
- Expected: REJECTED
|
|
- DB assertions: no trade rows for that run
|
|
- OR-12 Insufficient cash
|
|
- Steps: initial_cash too low; place buy above cash
|
|
- Expected: REJECTED
|
|
- DB assertions: no trade rows for that run
|
|
|
|
### Market Time & Session Logic
|
|
- MT-01 Market closed SIP
|
|
- Steps: call try_execute_sip with market_open=False
|
|
- Expected: no order/trade
|
|
- DB assertions: counts unchanged
|
|
- MT-03 Idempotent logical_time
|
|
- Steps: execute tick twice at same logical_time
|
|
- Expected: no duplicate orders/trades/mtm/equity/events
|
|
- DB assertions: unique logical_time counts <= 1
|
|
|
|
### Engine Idempotency & Crash Recovery
|
|
- IR-02 Crash mid-transaction
|
|
- Steps: open transaction, write state, raise exception
|
|
- Expected: rollback, no partial rows
|
|
- DB assertions: counts unchanged
|
|
- IR-03 Retry same tick
|
|
- Steps: rerun tick for same logical_time
|
|
- Expected: exactly-once side effects
|
|
- DB assertions: no duplicates
|
|
|
|
### Multi-User / Multi-Run Isolation
|
|
- MU-01 Two users, isolated runs
|
|
- Steps: create two users + runs; execute tick for each
|
|
- Expected: separate data
|
|
- DB assertions: rows scoped by (user_id, run_id)
|
|
- MU-02 Same user, two runs
|
|
- Steps: run A tick; stop; run B tick
|
|
- Expected: separate data
|
|
- DB assertions: counts separate per run_id
|
|
|
|
### Deterministic Replay
|
|
- DR-01 Replay determinism
|
|
- Steps: run N ticks with fixed feed for run A and run B
|
|
- Expected: identical ledgers (excluding surrogate IDs)
|
|
- DB assertions: normalized rows equal
|
|
|
|
### Concurrency & Race Conditions
|
|
- CC-01 Tick vs Stop race
|
|
- Steps: lock run row in thread A; update status in thread B
|
|
- Expected: consistent final status; no partial writes
|
|
- DB assertions: no half-written rows
|
|
- CC-02 Two ticks same logical_time
|
|
- Steps: run two ticks concurrently with same logical_time
|
|
- Expected: dedupe
|
|
- DB assertions: SIP_EXECUTED count <= 1
|
|
|
|
### Failure Injection
|
|
- FI-01 Crash after trade before MTM
|
|
- Steps: execute SIP (orders/trades), then log MTM in separate tx
|
|
- Expected: no MTM before explicit insert; MTM inserts once when run
|
|
- DB assertions: MTM logical_time count == 1
|
|
|
|
## DB Assertions (Post-Run)
|
|
|
|
- Orphan check:
|
|
- `SELECT COUNT(*) FROM engine_state es LEFT JOIN strategy_run sr ON sr.user_id=es.user_id AND sr.run_id=es.run_id WHERE sr.run_id IS NULL;`
|
|
- Dedupe check:
|
|
- `SELECT user_id, run_id, logical_time, COUNT(*) FROM mtm_ledger GROUP BY user_id, run_id, logical_time HAVING COUNT(*) > 1;`
|
|
- Lifecycle check:
|
|
- `SELECT user_id, COUNT(*) FROM strategy_run WHERE status='RUNNING' GROUP BY user_id HAVING COUNT(*) > 1;`
|
|
|
|
## Test Execution
|
|
|
|
Use `pytest -q` from repo root. See `README_test.md`.
|