MSX API

Data pipeline for the Muscat Stock Exchange (MSX). Scrapes the symbol universe, backfills daily and intraday history per symbol into a local SQLite database, runs data quality checks, and simulates a momentum trading strategy against the stored data.

Disclaimer

MSX has no official public API. This project calls two undocumented endpoints on www.msx.om, identified via network inspection of the public site. They are unauthenticated and back the site's own market-watch and charting widgets.

• Endpoints can change or disappear without notice.
• No server-side date-range filtering exists on the chart-data endpoint. Every pull fetches a symbol's full series; update filters new-vs-seen records client-side.
• Field semantics are inferred from observed payloads, not a spec.
• Keep the inter-request delay in place. Do not parallelize requests. Do not poll more often than needed.

Personal/research tool. Not a vendor-guaranteed data feed.

Endpoints

Endpoint	Method	Body	Returns
/api.aspx/MarketToday	POST	{}	Single-item list of index-level fields (MSX30, Change, Turnover, Volume, Trades). Numeric fields are comma-thousands strings.
/company-chart-data.aspx?s={SYMBOL}	GET	—	Flat array mixing daily closes (open/high/low null) and minute-level intraday ticks (carry Year/Month/Day/Hour/Minute).
/companies.aspx/List	POST	{}	One record per listed instrument: Symbol plus Market/Sector/SubSector/Listed codes.

companies.aspx/List returns every instrument MSX tracks (equities, bonds, mutual funds, preferred shares, rights issues; ~290 records). symbols.py narrows this to an equities-only universe (currently 105 symbols):

• Listed == "1" (active status).
• Subtract symbols from POST /mutual-funds.aspx/List and POST /bonds.aspx/List.
• Exclude names containing "RIGHT" or "PREF".

105 is above the commonly-cited ~77 headline-equity count. No field isolates a smaller set cleanly. Treat 105 as an approximation. Rationale documented next to SYMBOL_LISTED_STATUS_FILTER in config.py.

Project layout

msx_pipeline/
  __init__.py
  client.py     HTTP client: retries, timeouts, delay, logging
  symbols.py    symbol discovery + JSON cache (equities-only filter)
  storage.py    SQLite schema + upsert logic
  ingest.py     orchestration: symbols -> fetch -> store
  quality.py    data quality checks + report rendering
  strategy.py   momentum signal generation (ranking)
  portfolio.py  position sizing, rebalancing, transaction cost + slippage model
  simulator.py  day-by-day backtest engine + incremental live/paper mode
  metrics.py    performance metrics (CAGR, Sharpe, drawdown, fill-time, etc.)
  export.py     self-contained JSON/CSV/xlsx export of a simulation run
  cli.py        argparse CLI
  config.py     constants (URLs, delay, DB path, thresholds, strategy params)
requirements.txt
README.md

Setup

git clone https://github.com/MajdKZ1/MSX-API.git
cd msx_pipeline
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No credentials or API keys. Endpoints are unauthenticated; none should be added.

Quickstart

1. Discover symbols and backfill history

python -m msx_pipeline.cli discover-symbols
python -m msx_pipeline.cli backfill

Creates data/msx.sqlite3 (gitignored). Default 1.5s inter-request delay; a full backfill of ~105 symbols takes a few minutes of request time, plus payload parse time, since long-listed names carry thousands of records back to 2007.

python -m msx_pipeline.cli validate

2. Run a backtest

python -m msx_pipeline.cli backtest --start 2024-01-01 --end 2026-07-23

Replays the momentum strategy over the date range and prints a performance report (CAGR, Sharpe, max drawdown, alpha vs. benchmark, entry fill-time distribution).

Flag	config.py constant	Meaning	Default
--lookback	MOMENTUM_LOOKBACK_DAYS	N-trading-day momentum window	20
--top-k	MOMENTUM_TOP_K	symbols held, equal-weighted	10
--capital	STARTING_CAPITAL	starting capital, OMR	5,000,000
--cost-rate	TRANSACTION_COST_RATE	transaction cost, fraction of notional	0.0015
--position-limit	POSITION_LIMIT_PCT	max portfolio fraction in one symbol	0.15
--adv-cap	ADV_PARTICIPATION_CAP	max trade size as fraction of trailing ADV	0.10
--rebalance-frequency	REBALANCE_FREQUENCY	weekly or monthly	weekly

Two additional parameters are set only in config.py (not CLI flags, since they shape benchmark construction rather than the strategy) but are recorded per-run for reproducibility: BENCHMARK_CONSTITUENT_COUNT (default 30), BENCHMARK_TURNOVER_LOOKBACK_DAYS (default 60).

python -m msx_pipeline.cli backtest --start 2024-01-01 --end 2026-07-23 \
    --run-id my-experiment --top-k 8 --rebalance-frequency monthly

Omit --run-id and one is auto-generated (backtest-YYYYMMDDTHHMMSS).

3. List runs

python -m msx_pipeline.cli list-runs

Prints every simulation_runs row, most recent first: run_id, mode, status, date range, full parameter set.

4. Advance a live run

python -m msx_pipeline.cli update
python -m msx_pipeline.cli run-daily --run-id my-experiment

update fetches new data since the last pull. run-daily advances the named run by one day, loading prior cash/positions from that run's latest snapshot. Run this pair once per MSX trading day (Sunday-Thursday; MSX closed Friday/Saturday), e.g. via cron after market close. Calling run-daily before update has new data is a no-op (logged).

5. Export

python -m msx_pipeline.cli export --run-id my-experiment --format json --out my-experiment.json
python -m msx_pipeline.cli export --run-id my-experiment --format csv  --out my-experiment-csv/
python -m msx_pipeline.cli export --run-id my-experiment --format xlsx --out my-experiment.xlsx

Pulls everything tied to a run_id into one self-contained file or folder. No DB access needed downstream.

Global flags

-v / --verbose: debug-level logging, any subcommand.

Ingestion-only (discover-symbols, backfill, update):

• --delay SECONDS: override inter-request delay (default 1.5s; also MSX_REQUEST_DELAY).
• --rediscover: re-scrape the symbol list live instead of using the cache.

Configuration

All tunables live in msx_pipeline/config.py, overridable via environment variables:

Variable	Purpose	Default
MSX_REQUEST_DELAY	seconds between requests	1.5
MSX_DATA_DIR	directory for DB + symbol cache	<repo root>/data
MSX_DB_PATH	SQLite file path	<data dir>/msx.sqlite3
MSX_SYMBOLS_CACHE	symbol cache JSON path	<data dir>/symbols_cache.json

All three resolve relative to config.PROJECT_ROOT, not the process's working directory. A relative path set via env var is joined onto PROJECT_ROOT; set an absolute path to place the DB elsewhere. data/ and exports/ are gitignored.

Strategy/simulation parameters default to the constants in this section and can be overridden per-invocation without editing config.py.

Database schema

• symbols: symbol (PK), first_seen_date, last_updated
• daily_history: symbol (FK), date, close, volume, turnover, open, high, low. Unique on (symbol, date).
• intraday_ticks: symbol (FK), timestamp, close, volume, turnover. Unique on (symbol, timestamp).

Records from company-chart-data.aspx route automatically: non-null Hour/Minute fields go to intraday_ticks, otherwise daily_history. All writes are upserts; backfill and update are idempotent and safe on a cron.

simulation_runs

run_id (PK), mode (backtest/live), created_at, start_date, end_date, every SimulationParams field (momentum_lookback_days, top_k, starting_capital, transaction_cost_rate, adv_participation_cap, position_limit_pct, rebalance_frequency), plus adv_lookback_days, benchmark_constituent_count, benchmark_turnover_lookback_days (snapshotted from config.py at creation time), and status.

trades

run_id (FK), date, symbol, side, requested_shares, filled_shares, price, gross_value, transaction_cost, unfilled_shares, partially_filled. One row per simulated order.

portfolio_snapshots

run_id (FK), date, cash, positions_value, total_value, benchmark_value, positions_json ({"SYMBOL": shares, ...}), num_positions. Unique on (run_id, date). One row per trading day per run.

Databases created before rebalance_frequency and the benchmark_*/adv_lookback_days columns existed are migrated automatically on the first storage.get_connection() call (idempotent ALTER TABLE ... ADD COLUMN, backfilled with the then-current defaults).

Data quality checks (validate)

• Gaps: missing MSX trading days (Sun-Thu) between consecutive daily records beyond MAX_ALLOWED_GAP_DAYS.
• Zero-liquidity stretches: runs of near-zero volume days at or above ZERO_LIQUIDITY_RUN_THRESHOLD. Flagged for exclusion from anomaly detection, not treated as data errors.
• Turnover consistency: turnover ≈ volume × close within TURNOVER_TOLERANCE (default 5%). Persistent large deviations indicate a units mismatch.
• Duplicates / out-of-order dates: sanity check; UNIQUE(symbol, date) should already prevent true duplicates.

Report prints to stdout; optionally written to file with --output.

Strategy

v1: cross-sectional momentum, long-only. For each symbol, compute the N-trading-day return (default 20). Rank descending. Hold the top K (default 10), equal-weighted. Rebalance on the first trading day of each new ISO week or calendar month (--rebalance-frequency {weekly,monthly}), not a fixed weekday/day-of-month, so a holiday shifting the period's first day doesn't skip a rebalance. No shorting, no other signals.

Fund simulation model

• Starting capital: 5,000,000 OMR.

• Transaction cost: 0.15% of trade notional per trade. This is the rate given in the original spec, not a verified MSX brokerage fee. No published fee schedule was found; adjust TRANSACTION_COST_RATE in config.py if an authoritative figure surfaces.

• Slippage: modeled as a volume-participation cap, not a price-impact formula. A trade cannot exceed 10% of the symbol's trailing 20-day average daily volume (ADV_PARTICIPATION_CAP / ADV_LOOKBACK_DAYS). Excess size is logged as partially_filled=1 with the gap in trades.unfilled_shares. BUY orders are additionally capped so cash never goes negative.

• Position limit: max 15% of portfolio value in one symbol (POSITION_LIMIT_PCT). At defaults (K=10, limit=15%) equal weighting (10% each) never triggers the cap; relevant only below K≈7.

• Benchmark: no historical MSX30 time series exists through any known endpoint. MarketToday is a live snapshot only; performance.aspx/PerformanceChartData returns a single period-summary record, not a daily series. The benchmark is a reconstructed proxy: equal-weighted daily returns of the BENCHMARK_CONSTITUENT_COUNT (default 30) symbols with the highest trailing average turnover, reconstituted daily, compounded from the same starting capital.

  Equal-weighting all 105 equities produces a wildly non-representative benchmark, since illiquid micro-caps (e.g. SHPS, HECI) regularly move 50-160% in a day on a few hundred OMR of turnover. Restricting to the most liquid names better approximates MSX30's own liquidity-driven constituent selection.

  The turnover ranking uses a rolling BENCHMARK_TURNOVER_LOOKBACK_DAYS-day (default 60) trailing average, lagged by one day, recomputed daily. This is not a whole-period average; a whole-period average would introduce look-ahead bias. Still a proxy, not the official index. Treat comparisons as directional.

Entry fill-time metric

For each position entry (a symbol going from zero to nonzero holding), measures calendar days until mark-to-market weight reaches target weight (within POSITION_FULL_TARGET_TOLERANCE, default 1%). Three outcomes:

• completed: reached target weight; contributes a days-to-fill value.
• incomplete: sold back to zero before reaching target.
• still_open: still building at end of backtest window (right-censored).

metrics.compute_entry_episodes(conn, run_id) returns raw per-entry records. metrics.compute_entry_fill_times(conn, run_id) aggregates to mean/median/min/max. Computed by walking portfolio_snapshots, no trade replay needed.

Export formats

--format json (default) writes:

{
  "run": { /* full simulation_runs row */ },
  "metrics": {
    "strategy": { /* PerformanceMetrics: cagr, sharpe_ratio, max_drawdown, ... */ },
    "benchmark": { /* same fields, for benchmark series */ },
    "alpha_cagr": -0.3293
  },
  "portfolio_snapshots": [
    { "date": "2024-01-01", "cash": ..., "positions_value": ..., "total_value": ...,
      "benchmark_value": ..., "num_positions": 10, "positions": {"BWPC": 10683.08, ...} }
  ],
  "trades": [
    { "date": ..., "symbol": ..., "side": "BUY"|"SELL", "requested_shares": ...,
      "filled_shares": ..., "price": ..., "gross_value": ..., "transaction_cost": ...,
      "unfilled_shares": ..., "partially_filled": true|false }
  ],
  "fill_times": [
    { "symbol": ..., "signal_date": ..., "status": "completed"|"incomplete"|"still_open",
      "completion_date": "2024-07-08" | null, "days_to_fill": 14 | null }
  ],
  "benchmark": [
    { "date": "2024-01-01", "benchmark_value": 5052927.60 }
  ]
}

portfolio_snapshots and benchmark both carry benchmark_value deliberately, so benchmark works standalone.

--format csv writes separate files in --out/: run_metadata.csv, metrics.csv (one row per series, alpha_cagr repeated on both), snapshots.csv (positions flattened to a positions_json string column), trades.csv, fill_times.csv, benchmark.csv. All read with pandas.read_csv; positions_json needs json.loads per cell.

--format xlsx writes the same six tables as sheets (Snapshots, Trades, FillTimes, Metrics, Benchmark, RunMetadata) via openpyxl, header row, auto-sized columns. positions flattened to JSON text as in the CSV export. inf/nan float values are cast to string form ("inf") before writing, since openpyxl otherwise silently writes them as blank cells.

Diagnostic results

Two diagnostic backtests (2024-01-01 to 2026-07-23) isolating the strategy's underperformance against its benchmark. Baseline: weekly rebalance, K=10, ADV cap active. CAGR 6.00%, alpha -32.93%, 39.4% of entries reached target weight.

1. ADV cap disabled (--adv-cap inf, diagnostic only, assumes unlimited instant fills at the quoted close): CAGR 44.95%, alpha +6.01%, entry completion rate 93.2%. The momentum signal has positive edge over the benchmark when execution is unconstrained; the underperformance is not a signal-quality problem.

2. Monthly rebalancing (--rebalance-frequency monthly, ADV cap active): completion rate improved to 45.5% but did not close the gap to benchmark. CAGR fell to 4.09%, alpha widened to -34.84% (Sharpe improved to 1.33, max drawdown improved to -2.47%, off a smaller total return). MSX's liquidity constraint is severe enough that rebalance cadence alone does not fix it.

Both runs are retained as simulation_runs rows (JSON dumps in exports/).

Known v1 limitations

• Momentum ranking and position valuation use different price views by design: only symbols with a print today are tradeable, but existing holdings are marked to market using the last known price (forward-filled) so a thinly-traded position isn't valued at zero on a no-trade day.
• 23 rows in daily_history have close == 0, likely halted/suspended-trading days reported as 0 rather than omitted (spread across 2007-2012). Treated as missing data throughout simulator.py; a literal 0 would otherwise produce an infinite one-day return on resumption.
• ADV-based partial fills mean the strategy can hold more than top_k symbols at once. Thin names take multiple weekly rebalances to fully enter or exit, so residual positions accumulate beyond the current top-K target. A real consequence of MSX's liquidity profile, not a bug.