PQAP Comprehensive Testing Plan
Document Version: 1.0 Created: 2025-12-25 Last Updated: 2025-12-25
Executive Summary
This document outlines a comprehensive testing strategy for the PQAP (Polymarket Quantitative Automation Platform) trading system. The plan addresses unit tests, integration tests, API endpoint tests, data consistency tests, and UI/template tests with a focus on preventing regressions and catching real bugs in critical data transformation flows.
Current Testing State
Existing Test Coverage
| Category | Files | Tests | Status |
|---|---|---|---|
| Unit - Strategies | tests/unit/test_strategies.py |
~40 | Good |
| Unit - Paper Trading | tests/unit/test_paper_trading.py |
~35 | Good |
| Unit - Database Explorer | tests/unit/test_database_explorer.py |
~18 | Good |
| Unit - Config | tests/unit/test_config.py |
~10 | Good |
| Integration - Trading Flow | tests/integration/test_trading_flow.py |
~15 | Good |
| Backtest | tests/backtest/test_backtest.py |
~20 | Good |
Critical Gaps Identified
- Data Transformation Tests - No tests for paper_trading_provider -> dashboard display mapping
- Dashboard API Tests - No endpoint-level tests for dashboard routes
- Template Rendering Tests - No tests for Jinja2 template variable bindings
- Cross-Page Consistency Tests - No verification of data consistency across pages
- Signal Data Tests - No tests for signal timestamp/type field handling
- Storage Layer Tests - Limited coverage of MetricsStorage methods
Testing Categories
1. Unit Tests
1.1 Data Transformation Tests (Critical)
Purpose: Verify correct field mapping between data sources and display layers.
Files to Test:
- src/admin/dashboard_api.py - DashboardAPI class
- src/execution/paper_trading.py - PaperTradingEngine output methods
- src/tracking/storage.py - MetricsStorage query methods
Test Scenarios:
# tests/unit/test_data_transformations.py
class TestPositionDataTransformation:
"""Test position data flows from paper trading to dashboard."""
def test_position_field_mapping(self):
"""Test that paper trading position fields map correctly to dashboard."""
# Paper trading returns: entry_price, current_price, unrealized_pnl
# Dashboard expects: avg_entry_price, current_price, unrealized_pnl
pass
def test_closed_positions_filtered(self):
"""Test that closed positions are excluded from display."""
pass
def test_zero_size_positions_filtered(self):
"""Test that zero-size positions are excluded."""
pass
def test_market_title_fallback(self):
"""Test market_title falls back to market_question then market_id."""
pass
class TestTradeDataTransformation:
"""Test trade data flows from paper trading to dashboard."""
def test_trade_field_mapping(self):
"""Test trade fields map correctly for template."""
# Paper trading: outcome_name, timestamp, value
# Dashboard: outcome, timestamp (formatted), value
pass
def test_trade_pnl_display(self):
"""Test P&L values display correctly (especially zero values)."""
pass
def test_trade_timestamp_formatting(self):
"""Test timestamp strings are handled correctly."""
pass
class TestSignalDataTransformation:
"""Test signal data flows from storage to dashboard."""
def test_signal_timestamp_field(self):
"""Test signals use 'timestamp' or 'created_at' correctly."""
# signals.html expects: signal.timestamp or signal.created_at
pass
def test_signal_type_display(self):
"""Test signal_type displays correctly (BUY/SELL/UNKNOWN)."""
pass
def test_signal_strength_normalization(self):
"""Test strength values (0-1 vs percentage) display correctly."""
pass
Fixtures Needed:
@pytest.fixture
def mock_paper_trading_data():
"""Mock complete paper trading provider response."""
return {
"summary": {
"total_value": 1050.00,
"starting_capital": 1000.00,
"available_capital": 800.00,
"position_value": 250.00,
"total_pnl": 50.00,
"unrealized_pnl": 25.00,
"realized_pnl": 25.00,
"return_pct": 5.0,
"open_positions": 2,
"total_trades": 10
},
"positions": [
{
"market_id": "market_123",
"market_question": "Will BTC exceed $100k?",
"outcome_id": "yes_123",
"outcome": "YES",
"side": "BUY",
"size": 100.0,
"entry_price": 0.55,
"current_price": 0.60,
"unrealized_pnl": 5.00,
"closed": False
}
],
"trades": [
{
"trade_id": "paper_1",
"strategy_id": "dual_arb_v1",
"market_id": "market_123",
"outcome_id": "yes_123",
"outcome_name": "YES",
"side": "BUY",
"size": 100.0,
"price": 0.55,
"value": 55.00,
"timestamp": "2025-12-25T10:30:00",
"pnl": 0.00
}
]
}
@pytest.fixture
def mock_signal_data():
"""Mock signal data from storage."""
return [
{
"signal_id": "sig_abc123",
"strategy_id": "dual_arb_v1",
"market_id": "market_123",
"signal_type": "buy",
"direction": "BUY",
"strength": "0.85",
"executed": 1,
"rejected_reason": None,
"created_at": "2025-12-25T10:30:00",
"timestamp": "2025-12-25T10:30:00"
}
]
1.2 Storage Layer Tests
Files to Test:
- src/tracking/storage.py - All MetricsStorage methods
Test Scenarios:
# tests/unit/test_storage.py
class TestMetricsStorageQueries:
"""Test MetricsStorage query methods."""
def test_get_trades_with_filters(self, temp_db_path):
"""Test trade filtering by strategy, side, date range."""
pass
def test_get_trades_pagination(self, temp_db_path):
"""Test trade pagination with limit and offset."""
pass
def test_get_trades_count_matches_query(self, temp_db_path):
"""Test count returns same total as unbounded query."""
pass
def test_get_trades_summary_aggregations(self, temp_db_path):
"""Test summary statistics are calculated correctly."""
pass
def test_get_recent_signals_hours_filter(self, temp_db_path):
"""Test signal time-based filtering."""
pass
def test_get_daily_pnl_date_handling(self, temp_db_path):
"""Test daily P&L handles date objects and strings."""
pass
def test_get_active_positions_filters_closed(self, temp_db_path):
"""Test active positions excludes size=0 positions."""
pass
class TestMetricsStorageSave:
"""Test MetricsStorage save methods."""
def test_save_trade_decimal_handling(self, temp_db_path):
"""Test Decimal values are serialized and retrieved correctly."""
pass
def test_save_signal_with_all_fields(self, temp_db_path):
"""Test signal with all optional fields saves correctly."""
pass
def test_upsert_strategy_state_creates_new(self, temp_db_path):
"""Test upsert creates new strategy state."""
pass
def test_upsert_strategy_state_updates_existing(self, temp_db_path):
"""Test upsert updates existing strategy state."""
pass
1.3 Strategy Tests Extensions
Files to Test:
- src/strategies/mean_reversion/strategy.py
- src/strategies/market_maker/strategy.py
- src/strategies/momentum/strategy.py
- src/strategies/value_betting/strategy.py
Test Scenarios:
# tests/unit/test_strategy_implementations.py
class TestMeanReversionStrategy:
"""Test mean reversion strategy edge cases."""
def test_insufficient_history_returns_none(self):
"""Strategy returns None with < required history."""
pass
def test_extreme_deviation_high_confidence(self):
"""High deviation produces high confidence signal."""
pass
def test_within_threshold_no_signal(self):
"""Price within threshold produces no signal."""
pass
class TestMarketMakerStrategy:
"""Test market maker strategy."""
def test_calculates_spread_correctly(self):
"""Spread calculation based on volatility."""
pass
def test_positions_both_sides(self):
"""Generates both buy and sell quotes."""
pass
# Similar for momentum, value_betting strategies
2. Integration Tests
2.1 Dashboard Data Flow Tests
Purpose: Test complete data flow from backend to dashboard pages.
Test Scenarios:
# tests/integration/test_dashboard_data_flow.py
class TestDashboardDataFlow:
"""Test end-to-end data flow to dashboard."""
@pytest.fixture
def dashboard_api(self, mock_controller, mock_storage, mock_config):
"""Create DashboardAPI with mocks."""
return DashboardAPI(
controller=mock_controller,
storage=mock_storage,
config=mock_config,
paper_trading_provider=lambda: mock_paper_trading_data,
signals_provider=lambda: mock_signal_data
)
async def test_dashboard_page_renders_positions(self, dashboard_api, aiohttp_client):
"""Test dashboard page includes all open positions."""
pass
async def test_trades_page_renders_both_tabs(self, dashboard_api, aiohttp_client):
"""Test trades page renders Positions and History tabs."""
pass
async def test_signals_page_renders_all_signals(self, dashboard_api, aiohttp_client):
"""Test signals page renders with correct field mapping."""
pass
async def test_cross_page_pnl_consistency(self, dashboard_api, aiohttp_client):
"""Test P&L values are consistent across dashboard, trades, pnl pages."""
pass
async def test_cross_page_position_count_consistency(self, dashboard_api, aiohttp_client):
"""Test position counts match across all pages."""
pass
2.2 Paper Trading to Dashboard Flow
Purpose: Verify paper trading state correctly reflects in dashboard.
Test Scenarios:
# tests/integration/test_paper_to_dashboard.py
class TestPaperTradingDashboardIntegration:
"""Test paper trading data flows correctly to dashboard."""
def test_execute_trade_reflected_in_dashboard(self):
"""After executing trade, dashboard shows new position."""
pass
def test_close_position_removes_from_dashboard(self):
"""After closing position, dashboard no longer shows it."""
pass
def test_price_update_reflected_in_unrealized_pnl(self):
"""Price update changes unrealized P&L on dashboard."""
pass
def test_portfolio_history_updates(self):
"""Portfolio history chart data updates after trades."""
pass
2.3 Strategy Execution Integration
Purpose: Test strategy signal generation to trade execution flow.
# tests/integration/test_strategy_execution.py
class TestStrategyToExecution:
"""Test strategy signal flows to execution."""
def test_signal_tracked_in_storage(self):
"""Generated signal is saved to storage."""
pass
def test_executed_signal_marked_as_executed(self):
"""After execution, signal marked executed in storage."""
pass
def test_rejected_signal_has_reason(self):
"""Rejected signals store rejection reason."""
pass
3. API/Endpoint Tests
3.1 Dashboard Route Tests
Files to Test:
- src/admin/dashboard_api.py - All route handlers
Test Scenarios:
# tests/api/test_dashboard_routes.py
class TestDashboardRoutes:
"""Test dashboard HTTP routes."""
@pytest.fixture
async def client(self, aiohttp_client, app_with_routes):
"""Create test client."""
return await aiohttp_client(app_with_routes)
async def test_dashboard_page_returns_200(self, client):
"""GET /dashboard returns 200."""
resp = await client.get('/dashboard')
assert resp.status == 200
async def test_strategies_page_returns_200(self, client):
"""GET /strategies returns 200."""
pass
async def test_trades_page_returns_200(self, client):
"""GET /trades returns 200."""
pass
async def test_trades_page_with_filters(self, client):
"""GET /trades with query params works."""
resp = await client.get('/trades?strategy=dual_arb_v1&side=BUY')
assert resp.status == 200
async def test_signals_page_returns_200(self, client):
"""GET /signals returns 200."""
pass
async def test_markets_page_returns_200(self, client):
"""GET /markets returns 200."""
pass
async def test_pnl_page_returns_200(self, client):
"""GET /pnl returns 200."""
pass
async def test_anomalies_page_returns_200(self, client):
"""GET /anomalies returns 200."""
pass
async def test_observability_page_returns_200(self, client):
"""GET /observability returns 200."""
pass
async def test_admin_logs_page_returns_200(self, client):
"""GET /admin/logs returns 200."""
pass
async def test_admin_config_page_returns_200(self, client):
"""GET /admin/config returns 200."""
pass
class TestDashboardAPIEndpoints:
"""Test dashboard JSON API endpoints."""
async def test_api_portfolio_returns_json(self, client):
"""GET /api/portfolio returns valid JSON."""
resp = await client.get('/api/portfolio')
assert resp.status == 200
data = await resp.json()
assert 'total_value' in data
async def test_api_positions_returns_json(self, client):
"""GET /api/positions returns positions array."""
resp = await client.get('/api/positions')
assert resp.status == 200
data = await resp.json()
assert 'positions' in data
async def test_api_trades_returns_json(self, client):
"""GET /api/trades returns trades with pagination."""
resp = await client.get('/api/trades')
assert resp.status == 200
data = await resp.json()
assert 'trades' in data
assert 'total' in data
assert 'page' in data
async def test_api_trades_export_returns_csv(self, client):
"""GET /api/trades/export returns CSV."""
resp = await client.get('/api/trades/export')
assert resp.status == 200
assert 'text/csv' in resp.content_type
async def test_api_observability_returns_json(self, client):
"""GET /api/observability returns health data."""
pass
async def test_api_logs_returns_json(self, client):
"""GET /api/logs returns parsed log entries."""
pass
async def test_api_test_telegram_post(self, client):
"""POST /admin/config/test-telegram works."""
pass
async def test_api_save_config_post(self, client):
"""POST /admin/config/save updates config."""
pass
4. Data Consistency Tests
4.1 Cross-Page Consistency
Purpose: Verify data displayed on different pages is consistent.
# tests/consistency/test_cross_page_data.py
class TestCrossPageConsistency:
"""Test data consistency across dashboard pages."""
async def test_total_pnl_consistent(self, client):
"""Total P&L same on dashboard, trades, pnl pages."""
dashboard = await client.get('/dashboard')
trades = await client.get('/trades')
pnl = await client.get('/pnl')
# Extract P&L values and compare
pass
async def test_open_positions_count_consistent(self, client):
"""Open position count same on dashboard and trades."""
pass
async def test_total_trades_count_consistent(self, client):
"""Total trades count same across pages."""
pass
async def test_strategy_metrics_consistent(self, client):
"""Strategy metrics same on strategies and dashboard."""
pass
4.2 API vs Page Consistency
# tests/consistency/test_api_page_consistency.py
class TestAPIPageConsistency:
"""Test API data matches rendered page data."""
async def test_portfolio_api_matches_dashboard(self, client):
"""API portfolio data matches dashboard display."""
api_resp = await client.get('/api/portfolio')
page_resp = await client.get('/dashboard')
# Compare values
pass
async def test_positions_api_matches_trades_page(self, client):
"""API positions match trades page positions tab."""
pass
async def test_trades_api_matches_trades_page(self, client):
"""API trades match trades page history tab."""
pass
5. UI/Template Tests
5.1 Jinja2 Template Rendering Tests
Purpose: Verify templates render correctly with various data states.
# tests/templates/test_template_rendering.py
class TestDashboardTemplate:
"""Test dashboard.html template rendering."""
def test_renders_with_empty_positions(self, template_renderer):
"""Dashboard renders correctly with no positions."""
pass
def test_renders_with_positions(self, template_renderer, mock_positions):
"""Dashboard renders all position fields correctly."""
pass
def test_pnl_color_classes(self, template_renderer):
"""Positive P&L gets 'positive' class, negative gets 'negative'."""
pass
def test_handles_missing_fields_gracefully(self, template_renderer):
"""Template handles None/missing fields without errors."""
pass
class TestTradesTemplate:
"""Test trades.html template rendering."""
def test_positions_tab_renders(self, template_renderer):
"""Positions tab content renders correctly."""
pass
def test_history_tab_renders(self, template_renderer):
"""Trade history tab renders correctly."""
pass
def test_pagination_controls_render(self, template_renderer):
"""Pagination shows correct page info."""
pass
def test_filters_preserve_values(self, template_renderer):
"""Filter inputs preserve selected values."""
pass
class TestSignalsTemplate:
"""Test signals.html template rendering."""
def test_timestamp_fallback(self, template_renderer):
"""Uses timestamp field, falls back to created_at."""
pass
def test_signal_type_badge(self, template_renderer):
"""Signal type displays with correct badge class."""
pass
def test_strength_percentage_display(self, template_renderer):
"""Strength 0-1 values display as percentages."""
pass
def test_executed_badge_styling(self, template_renderer):
"""Executed signals get 'yes' badge, others get 'no'."""
pass
def test_pagination_client_side(self, template_renderer):
"""Client-side pagination hides excess rows."""
pass
5.2 Template Filter Tests
# tests/templates/test_template_filters.py
class TestTemplateFilters:
"""Test custom Jinja2 filters."""
def test_format_number_thousands(self):
"""format_number adds thousand separators."""
assert format_number(1234567) == "1,234,567"
def test_format_number_decimals(self):
"""format_number respects decimal places."""
assert format_number(1234.567, 2) == "1,234.57"
def test_format_datetime_valid(self):
"""format_datetime handles datetime objects."""
pass
def test_format_datetime_string(self):
"""format_datetime handles ISO strings."""
pass
def test_format_datetime_none(self):
"""format_datetime returns 'Never' for None."""
pass
Test Data Fixtures
Core Fixtures
# tests/fixtures/trading_data.py
@pytest.fixture
def sample_paper_positions():
"""Sample paper trading positions."""
return [
{
"market_id": "market_btc_100k",
"market_question": "Will BTC exceed $100,000 by end of 2025?",
"outcome_id": "yes_btc_100k",
"outcome": "YES",
"side": "BUY",
"size": 100.0,
"entry_price": 0.45,
"current_price": 0.52,
"unrealized_pnl": 7.00,
"realized_pnl": 0.0,
"closed": False,
"entry_time": "2025-12-20T10:00:00"
},
{
"market_id": "market_eth_5k",
"market_question": "Will ETH exceed $5,000?",
"outcome_id": "no_eth_5k",
"outcome": "NO",
"side": "BUY",
"size": 50.0,
"entry_price": 0.60,
"current_price": 0.55,
"unrealized_pnl": -2.50,
"realized_pnl": 0.0,
"closed": False,
"entry_time": "2025-12-21T14:30:00"
}
]
@pytest.fixture
def sample_paper_trades():
"""Sample paper trading trade history."""
return [
{
"trade_id": "paper_001",
"strategy_id": "dual_arb_v1",
"market_id": "market_btc_100k",
"outcome_id": "yes_btc_100k",
"outcome_name": "YES",
"side": "BUY",
"size": 100.0,
"price": 0.45,
"value": 45.00,
"timestamp": "2025-12-20T10:00:00",
"pnl": 0.0
},
{
"trade_id": "paper_002",
"strategy_id": "market_maker_v1",
"market_id": "market_eth_5k",
"outcome_id": "no_eth_5k",
"outcome_name": "NO",
"side": "BUY",
"size": 50.0,
"price": 0.60,
"value": 30.00,
"timestamp": "2025-12-21T14:30:00",
"pnl": 0.0
}
]
@pytest.fixture
def sample_signals():
"""Sample signals data."""
return [
{
"signal_id": "sig_abc123def456",
"strategy_id": "dual_arb_v1",
"market_id": "market_btc_100k",
"signal_type": "buy",
"direction": "BUY",
"strength": "0.85",
"expected_profit": "0.02",
"executed": 1,
"rejected_reason": None,
"created_at": "2025-12-20T09:59:55",
"timestamp": "2025-12-20T09:59:55"
},
{
"signal_id": "sig_xyz789uvw012",
"strategy_id": "mean_reversion_v1",
"market_id": "market_eth_5k",
"signal_type": "sell",
"direction": "SELL",
"strength": "0.72",
"expected_profit": None,
"executed": 0,
"rejected_reason": "Insufficient capital",
"created_at": "2025-12-21T14:29:30",
"timestamp": "2025-12-21T14:29:30"
}
]
@pytest.fixture
def sample_strategies_status():
"""Sample strategy status data."""
return [
{
"strategy_id": "dual_arb_v1",
"name": "Dual-Outcome Arbitrage",
"version": "1.0",
"enabled": True,
"auto_disabled": False,
"allocated_capital": 2000.0,
"performance": {
"total_pnl": 125.50,
"total_trades": 45,
"wins": 32,
"losses": 13,
"win_rate": 71.1
},
"signals_24h": {
"total": 15,
"executed": 8
}
}
]
Implementation Priority
Critical (Week 1)
| Task | Description | Complexity | Dependencies |
|---|---|---|---|
| T1 | Data Transformation Unit Tests | Medium | None |
| T2 | Position Field Mapping Tests | Low | T1 |
| T3 | Signal Timestamp/Type Tests | Low | T1 |
| T4 | Dashboard Route Status Tests | Low | None |
High (Week 2)
| Task | Description | Complexity | Dependencies |
|---|---|---|---|
| T5 | Storage Layer Query Tests | Medium | None |
| T6 | Cross-Page Consistency Tests | Medium | T4 |
| T7 | API JSON Endpoint Tests | Low | T4 |
| T8 | Trade Data Flow Integration | High | T1, T5 |
Medium (Week 3)
| Task | Description | Complexity | Dependencies |
|---|---|---|---|
| T9 | Template Rendering Tests | Medium | None |
| T10 | Template Filter Tests | Low | None |
| T11 | Paper Trading -> Dashboard Integration | High | T1, T8 |
| T12 | Strategy Implementation Tests | Medium | None |
Lower Priority (Week 4+)
| Task | Description | Complexity | Dependencies |
|---|---|---|---|
| T13 | Config Save/Load Tests | Low | None |
| T14 | Telegram Alert Tests | Low | None |
| T15 | Backtest Engine Extensions | Medium | None |
| T16 | Anomaly Detection Tests | Medium | None |
Parallel Implementation Tasks
The following tasks can be executed in parallel by different agents:
Parallel Group 1: Core Unit Tests
- Agent A: Data Transformation Tests (T1, T2, T3)
- Agent B: Storage Layer Tests (T5)
- Agent C: Template Filter Tests (T10)
Parallel Group 2: API and Route Tests
- Agent A: Dashboard Route Tests (T4)
- Agent B: API Endpoint Tests (T7)
- Agent C: Strategy Unit Tests (T12)
Parallel Group 3: Integration Tests
- Agent A: Cross-Page Consistency Tests (T6)
- Agent B: Paper Trading Integration Tests (T11)
- Agent C: Template Rendering Tests (T9)
Parallel Group 4: Extended Coverage
- Agent A: Trade Data Flow Tests (T8)
- Agent B: Config and Alert Tests (T13, T14)
- Agent C: Backtest and Anomaly Tests (T15, T16)
Test Commands
# Run all tests
pytest tests/ -v
# Run specific test category
pytest tests/unit/ -v
pytest tests/integration/ -v
pytest tests/api/ -v
pytest tests/consistency/ -v
pytest tests/templates/ -v
# Run with coverage
pytest tests/ --cov=src --cov-report=html
# Run specific test file
pytest tests/unit/test_data_transformations.py -v
# Run tests matching pattern
pytest -k "test_position" -v
# Run with parallel execution
pytest tests/ -n auto
Success Metrics
- Coverage Target: 80%+ code coverage for critical modules
- Regression Prevention: All data transformation paths tested
- CI Integration: Tests run on every commit
- Documentation: Test docstrings explain purpose
- Speed: Full test suite completes in < 60 seconds
Appendix: Key File References
| Module | Path | Priority |
|---|---|---|
| Paper Trading Engine | src/execution/paper_trading.py |
Critical |
| Dashboard API | src/admin/dashboard_api.py |
Critical |
| Metrics Storage | src/tracking/storage.py |
Critical |
| Strategy Base | src/strategies/base.py |
High |
| Template Renderer | src/admin/template_renderer.py |
High |
| Strategy Controller | src/admin/controller.py |
High |
| Database Explorer | src/admin/database_explorer.py |
Medium |
| Backtest Engine | src/backtest/engine.py |
Medium |
| Anomaly Detector | src/analytics/anomaly_detector.py |
Low |