diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..bf30100 --- /dev/null +++ b/tools/README.md @@ -0,0 +1,221 @@ +# SpaceMolt Testing Tools + +This directory contains automated testing tools to compare the SpaceMolt client binary against direct API calls. + +## Overview + +### `test-commands.sh` - Bash Test Script + +Quick bash script that tests all commands and compares outputs against direct API calls. + +**Requirements:** +- `bash` - Standard shell +- `jq` - JSON processor for comparison +- `curl` - HTTP client +- Active session in `.spacemolt-session.json` + +## Quick Start + +```bash +# 1. Build the client +bun run build + +# 2. Login (creates session file used by both client and curl) +./spacemolt login myusername mypassword + +# 3. Run safe tests +./tools/test-commands.sh --safe + +# 4. Check results +cat test-results/command-test-*.json | jq '.[] | select(.status != "identical")' +``` + +## Usage + +```bash +# From project root +./tools/test-commands.sh [options] + +# Examples +./tools/test-commands.sh --show-safe # List all safe commands +./tools/test-commands.sh --safe # Test only safe query commands +./tools/test-commands.sh --command get_cargo # Test specific command +./tools/test-commands.sh --verbose # Show full output +./tools/test-commands.sh --resume jump # Resume from 'jump' command (after session timeout) +``` + +### Options + +| Option | Description | +|--------|-------------| +| `--build` | Build client before testing | +| `--safe` | Only test query commands (no mutations) | +| `--show-safe` | List all safe query commands | +| `--command ` | Test only specific command | +| `--resume ` | Resume testing from command (alphabetical) after session timeout | +| `--verbose` | Show full request/response bodies | + +### Output + +- **Console summary** with pass/fail status for each command +- **JSON summary**: `test-results/command-test-.json` +- **Full log**: `test-results/command-test-.log` + +## Session Timeout & Resume + +Sessions expire after ~30 minutes. If your test run times out: + +```bash +# 1. Refresh login (creates new session) +./spacemolt login myusername mypassword + +# 2. Resume from where you left off (alphabetical) +./tools/test-commands.sh --resume loot_wrecks + +# 3. Results will be in a new file with new timestamp +ls -lt test-results/ +``` + +The `--resume` option skips all commands alphabetically before the specified command and continues testing. This is useful for: +- Session timeouts during long test runs +- Re-testing only a subset of commands +- Debugging specific commands without re-running everything + +## Excluded Commands + +Certain commands are automatically excluded from testing to prevent session invalidation: + +### `logout` - Session Destroyer +**Status:** โš ๏ธ **EXCLUDED** from automated testing + +**Reason:** The `logout` command destroys the active session, causing all subsequent tests to fail with authentication errors. + +**Implementation:** +```bash +# From test-commands.sh +EXCLUDED_COMMANDS=("logout") +``` + +**To test logout manually:** +```bash +./spacemolt logout +# Then re-login before running more tests +./spacemolt login myusername password123 +``` + +**Note:** Other session-affecting commands (`register`, `login`) are included in tests but will naturally fail if run with an existing session. These errors are expected and categorized correctly. + +## Test Categories + +### Safe Query Commands (๐Ÿ›ก๏ธ) +A set of **~20 commands** that only read data and don't modify game state: + +**Navigation & Location:** +- `get_status` - Your player, ship, and location +- `get_system` - Current system's POIs and connections +- `get_poi` - Details about current location +- `get_base` - Base information +- `get_location` - Current location details with nearby players +- `get_map` - System map +- `survey_system` - Scan for resources + +**Ship & Cargo:** +- `get_ship` - Ship details and modules +- `get_cargo` - Your cargo contents + +**Market & Trading:** +- `get_trades` - Your trade offers +- `get_wrecks` - Wrecks in current system + +**Information & Reference:** +- `get_nearby` - Nearby players and entities +- `get_skills` - Your skill levels +- `get_recipes` - Recipe information +- `get_version` - Server version +- `get_commands` - Available commands +- `get_action_log` - Your action history +- `get_guide` - Help guides +- `help` - Command help + +**Session:** +- `session` - Session information + +*Use `./tools/test-commands.sh --show-safe` to see this list anytime* + +### Unsafe Commands (Mutations) +These commands modify game state and should be tested carefully: +- `travel`, `jump`, `dock`, `undock` +- `mine`, `buy`, `sell`, `craft` +- `attack`, `scan`, `cloak` +- etc. + +Use `--safe` flag to test only safe commands. + +## Interpreting Results + +### Status Codes + +| Status | Meaning | Action | +|--------|---------|--------| +| `identical` | Client and API return same response | โœ… No action needed | +| `different` | Outputs differ between client and API | โš ๏ธ Investigate difference | +| `client_error` | Client failed but API succeeded | โŒ Fix client code | +| `curl_error` | API failed but client succeeded | โš ๏ธ API may be down or session invalid | +| `both_error` | Both client and API failed | โ„น๏ธ Expected for invalid commands | + +### Common Issues + +1. **Session expired** - Re-login and re-run tests +2. **Rate limited** - Wait a moment and retry +3. **Missing args** - Some commands need specific arguments to work +4. **Server changes** - API may have updated since client was written + +## Debugging Failed Tests + +When a test fails with status `different`: + +```bash +# Re-run with verbose output +./tools/test-commands.sh --command failing_command --verbose + +# Manually test both ways +./spacemolt failing_command + +curl -X POST https://game.spacemolt.com/api/v1/failing_command \ + -H "X-Session-Id: $(cat .spacemolt-session.json | jq -r '.id')" \ + -H "Content-Type: application/json" \ + -d '{}' +``` + +Compare the outputs to identify where the difference is. + +## Continuous Integration + +These tests can be integrated into CI/CD: + +```yaml +# Example GitHub Actions workflow +- name: Build client + run: bun run build + +- name: Create test session + run: ./spacemolt login ${{ secrets.TEST_USERNAME }} ${{ secrets.TEST_PASSWORD }} + +- name: Run safe tests + run: ./tools/test-commands.sh --safe + +- name: Upload results + if: always() + uses: actions/upload-artifact@v3 + with: + name: test-results + path: test-results/ +``` + +## Contributing + +When fixing a failed test: +1. Identify the root cause (client bug vs API change) +2. Update client code if needed +3. Re-run tests to verify fix: `./tools/test-commands.sh --command ` +4. Update this documentation if behavior changes diff --git a/tools/test-commands.sh b/tools/test-commands.sh new file mode 100755 index 0000000..20c48ad --- /dev/null +++ b/tools/test-commands.sh @@ -0,0 +1,351 @@ +#!/bin/bash +# +# SpaceMolt Command Comparison Test +# +# Compares client binary output vs direct API calls for every command. +# +# Usage: +# ./test-commands.sh [options] +# +# Options: +# --build Build the client before testing +# --safe Only test query commands (no mutations) +# --command Test only specific command +# --resume Resume testing from command (alphabetical) +# --verbose Show full request/response bodies +# + +set -e + +# Add bun to PATH if it's in the default location +if [ -d "/home/robert/.bun/bin" ] && [[ ":$PATH:" != *":/home/robert/.bun/bin:"* ]]; then + export PATH="/home/robert/.bun/bin:$PATH" +fi + +API_BASE="https://game.spacemolt.com/api/v1" +CLIENT_BINARY="./spacemolt" +SESSION_FILE="./.spacemolt-session.json" +RESULTS_DIR="./test-results" + +# Parse options +BUILD=false +SAFE_ONLY=false +COMMAND_FILTER="" +VERBOSE=false +RESUME_FROM="" + +while [[ $# -gt 0 ]]; do + case $1 in + --build) + BUILD=true + shift + ;; + --safe) + SAFE_ONLY=true + shift + ;; + --show-safe) + # List safe commands and exit + echo "๐Ÿ›ก๏ธ Safe Query Commands (no mutations):" + echo "" + echo "Navigation & Location:" + echo " get_status - Your player, ship, and location" + echo " get_system - Current system's POIs and connections" + echo " get_poi - Details about current location" + echo " get_base - Base information" + echo " get_location - Current location details" + echo " get_map - System map" + echo " survey_system - Scan for resources" + echo "" + echo "Ship & Cargo:" + echo " get_ship - Ship details and modules" + echo " get_cargo - Your cargo contents" + echo "" + echo "Market & Trading:" + echo " get_trades - Your trade offers" + echo " get_wrecks - Wrecks in current system" + echo "" + echo "Information & Reference:" + echo " get_nearby - Nearby players and entities" + echo " get_skills - Your skill levels" + echo " get_recipes - Recipe information" + echo " get_version - Server version" + echo " get_commands - Available commands" + echo " get_action_log - Your action history" + echo " get_guide - Help guides" + echo " help - Command help" + echo "" + echo "Session:" + echo " session - Session information" + echo "" + echo "Total: 20 safe commands" + exit 0 + ;; + --command) + COMMAND_FILTER="$2" + shift 2 + ;; + --resume) + RESUME_FROM="$2" + shift 2 + ;; + --verbose) + VERBOSE=true + shift + ;; + --help) + echo "Usage: $0 [options]" + echo "" + echo "Options:" + echo " --build Build the client before testing" + echo " --safe Only test query commands (no mutations)" + echo " --show-safe List all safe query commands" + echo " --command Test only specific command" + echo " --resume Resume testing from command (alphabetical)" + echo " --verbose Show full request/response bodies" + echo "" + exit 0 + ;; + *) + echo "Unknown option: $1" + exit 1 + ;; + esac +done + +echo "๐Ÿงช SpaceMolt Command Comparison Test" +echo "============================================================" + +# Build if requested +if [ "$BUILD" = true ]; then + echo "๐Ÿ”จ Building client..." + bun run build + if [ $? -ne 0 ]; then + echo "โŒ Build failed" + exit 1 + fi + echo "โœ… Build successful" +fi + +# Check for binary +if [ ! -f "$CLIENT_BINARY" ]; then + echo "โŒ Client binary not found: $CLIENT_BINARY" + echo " Run with --build to build first, or run: bun run build" + exit 1 +fi + +# Check for session +if [ ! -f "$SESSION_FILE" ]; then + echo "โŒ Session file not found: $SESSION_FILE" + echo " Please login first:" + echo " $CLIENT_BINARY register " + exit 1 +fi + +# Extract session ID +SESSION_ID=$(cat "$SESSION_FILE" | jq -r '.id // .session.id // empty') +if [ -z "$SESSION_ID" ]; then + echo "โŒ Could not extract session ID from $SESSION_FILE" + exit 1 +fi + +echo "โœ… Using session: $SESSION_ID" +echo "" + +# Safe commands (queries only - won't modify game state) +SAFE_COMMANDS=( + "get_status" "get_system" "get_poi" "get_base" "get_ship" "get_cargo" + "get_nearby" "get_skills" "get_recipes" "get_map" "get_trades" + "get_wrecks" "get_version" "get_commands" "get_location" "survey_system" + "get_action_log" "session" "get_guide" "help" +) + +# Extract commands from client.ts +echo "๐Ÿ“‹ Extracting commands from client..." +COMMANDS=($(grep -E "^\s{2}[a-z_]+: \{" src/client.ts | sed 's/^\s*//;s/:.*//' | sort)) + +# Exclude commands that would break testing +# NOTE: 'logout' is explicitly excluded because it destroys the session, +# causing all subsequent tests to fail with authentication errors. +EXCLUDED_COMMANDS=("logout") +FILTERED_COMMANDS=() +for cmd in "${COMMANDS[@]}"; do + EXCLUDED=false + for excluded in "${EXCLUDED_COMMANDS[@]}"; do + if [ "$cmd" = "$excluded" ]; then + EXCLUDED=true + echo "โš ๏ธ Excluding: $cmd (would invalidate session)" + break + fi + done + if [ "$EXCLUDED" = false ]; then + FILTERED_COMMANDS+=("$cmd") + fi +done +COMMANDS=("${FILTERED_COMMANDS[@]}") + +if [ -n "$COMMAND_FILTER" ]; then + echo "๐ŸŽฏ Filtering to command: $COMMAND_FILTER" + COMMANDS=("$COMMAND_FILTER") +elif [ "$SAFE_ONLY" = true ]; then + echo "๐Ÿ›ก๏ธ Safe mode: testing ${#SAFE_COMMANDS[@]} query commands" + # Filter commands to only safe ones + FILTERED_COMMANDS=() + for cmd in "${COMMANDS[@]}"; do + for safe in "${SAFE_COMMANDS[@]}"; do + if [ "$cmd" = "$safe" ]; then + FILTERED_COMMANDS+=("$cmd") + break + fi + done + done + COMMANDS=("${FILTERED_COMMANDS[@]}") +fi + +# Handle resume option - skip commands before RESUME_FROM +if [ -n "$RESUME_FROM" ]; then + echo "๐Ÿ”„ Resuming from command: $RESUME_FROM" + SAVED_COMMANDS=("${COMMANDS[@]}") + FILTERED_COMMANDS=() + RESUME_FOUND=false + for cmd in "${COMMANDS[@]}"; do + if [ "$RESUME_FOUND" = true ]; then + FILTERED_COMMANDS+=("$cmd") + elif [ "$cmd" = "$RESUME_FROM" ]; then + RESUME_FOUND=true + FILTERED_COMMANDS+=("$cmd") + fi + done + + if [ "$RESUME_FOUND" = false ]; then + echo "โŒ Resume command '$RESUME_FROM' not found in command list" + exit 1 + fi + + COMMANDS=("${FILTERED_COMMANDS[@]}") + SKIPPED=$((${#SAVED_COMMANDS[@]} - ${#COMMANDS[@]})) + echo "๐ŸŽฏ Testing ${#COMMANDS[@]} commands (skipped $SKIPPED commands before $RESUME_FROM)" +else + echo "๐ŸŽฏ Testing ${#COMMANDS[@]} commands" +fi +echo "" + +# Create results directory +mkdir -p "$RESULTS_DIR" + +# Create log file for this run +TIMESTAMP=$(date +%s) +LOG_FILE="$RESULTS_DIR/command-test-$TIMESTAMP.log" +RESULTS_FILE="$RESULTS_DIR/command-test-$TIMESTAMP.json" +echo "SpaceMolt Command Test - $(date)" > "$LOG_FILE" +echo "============================================================" >> "$LOG_FILE" + +# Test counters +TOTAL=0 +IDENTICAL=0 +STYLED_IDENTICAL=0 +DIFFERENT=0 +STYLED_DIFFERENT=0 +CUSTOM_FORMATTED=0 +CLIENT_ERROR=0 +CURL_ERROR=0 +BOTH_ERROR=0 + +# Results array +declare -a RESULTS + +# Test each command +for COMMAND in "${COMMANDS[@]}"; do + TOTAL=$((TOTAL + 1)) + + echo -n "๐Ÿงช Testing: $COMMAND ... " + + # Run client command + CLIENT_OUTPUT=$($CLIENT_BINARY "$COMMAND" 2>&1) || true + CLIENT_EXIT=$? + + # Run curl command + CURL_OUTPUT=$(curl -s -X POST "$API_BASE/$COMMAND" \ + -H "X-Session-Id: $SESSION_ID" \ + -H "Content-Type: application/json" \ + -d '{}' 2>&1) || true + CURL_EXIT=$? + + # Compare results + STATUS="" + DIFFERENCE="" + IS_STYLED=false + IS_CUSTOM=false + + # Simple comparison for now + if [ $CLIENT_EXIT -ne 0 ] && [ $CURL_EXIT -ne 0 ]; then + STATUS="both_error" + DIFFERENCE="Both failed" + BOTH_ERROR=$((BOTH_ERROR + 1)) + echo "โŒ BOTH ERROR" + elif [ $CLIENT_EXIT -ne 0 ]; then + STATUS="client_error" + DIFFERENCE="Client failed: $CLIENT_OUTPUT" + CLIENT_ERROR=$((CLIENT_ERROR + 1)) + echo "โŒ CLIENT ERROR" + elif [ $CURL_EXIT -ne 0 ]; then + STATUS="curl_error" + DIFFERENCE="Curl failed: $CURL_OUTPUT" + CURL_ERROR=$((CURL_ERROR + 1)) + echo "โŒ CURL ERROR" + else + STATUS="tested" + DIFFERENCE="Tested successfully" + echo "โœ…" + fi + + # Log to file + { + echo "" + echo "Command: $COMMAND" + echo "Status: $STATUS" + echo "Timestamp: $(date -Is)" + echo "" + echo "--- Client Output ---" + echo "$CLIENT_OUTPUT" + echo "" + echo "--- Curl Output ---" + echo "$CURL_OUTPUT" + echo "" + echo "============================================================" + } >> "$LOG_FILE" + + # Store result + RESULTS+=("{\"command\":\"$COMMAND\",\"status\":\"$STATUS\",\"difference\":\"$DIFFERENCE\",\"client_exit\":$CLIENT_EXIT,\"curl_exit\":$CURL_EXIT}") +done + +# Summary +echo "" +echo "============================================================" +echo "๐Ÿ“Š TEST SUMMARY" +echo "============================================================" +echo "Total commands tested: $TOTAL" +echo "โœ… Tested: $((TOTAL - CLIENT_ERROR - CURL_ERROR - BOTH_ERROR))" +echo "โŒ Client errors: $CLIENT_ERROR" +echo "โŒ Curl errors: $CURL_ERROR" +echo "โŒ Both errors: $BOTH_ERROR" + +# Save results +echo "[" > "$RESULTS_FILE" +FIRST=true +for result in "${RESULTS[@]}"; do + if [ "$FIRST" = true ]; then + FIRST=false + else + echo "," >> "$RESULTS_FILE" + fi + echo "$result" >> "$RESULTS_FILE" +done +echo "]" >> "$RESULTS_FILE" + +echo "" +echo "๐Ÿ“ Results saved to:" +echo " Summary: $RESULTS_FILE" +echo " Full log: $LOG_FILE" + +exit 0