FEAT: Propagate error messages from simulation methods to database

[mberz]

Proposed changes

• Add support for tacking different StatusCodes and error messages to the CloudExecutor and simulation task
• Handle and propagate errors in the simulation task
  • Explicitly raised errors by simulation methods are propagated with specific error message
  • Unexpected errors are propagated with generic error message
  • Errors outside the docker container are forwarded with a minimal matching message if available.
• Set up models and schema for error messages.

requires choras-org/frontend-v2#185

[Copilot]

Pull request overview

This PR adds end-to-end propagation of simulation error messages (including non-zero container exit codes and structured JSON errors) into persisted Simulation/SimulationRun records so the frontend can display meaningful failure reasons.

Changes:

• Detect non-zero simulation container exit codes and attempt to extract structured error messages from the result JSON.
• Persist errorMessage on Simulation and SimulationRun, and expose it via Marshmallow schemas.
• Extend CloudExecutor’s completion stub to carry an exit code and return more informative logs.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
app/services/simulation_service.py	Adds exit-code checking, structured error extraction, and persists error messages on failures.
app/services/executors/cloud_executor.py	Tracks simulation failure via JSON "error" and surfaces an exit code through _CompletedJob.
app/schemas/simulation_schema.py	Exposes errorMessage in API responses for simulations and runs.
app/models/SimulationRun.py	Adds errorMessage column to persist run-level failures.
app/models/Simulation.py	Adds errorMessage column to persist simulation-level failures.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

app/services/simulation_service.py:571

• The "unexpected" and "failed to initialize" error paths include the raw exception string in the message persisted to the DB / returned to the client, which can leak internal details. Also, initialization failures are surfaced as HTTP 400 even though they are server-side errors (should be 500).

            # Unexpected errors - log full details but show generic message
            error_details = traceback.format_exc()
            logger.error(f"Unexpected simulation error:\n{error_details}")

            error_msg = f"An unexpected error occurred: {str(ex)}"

app/services/simulation_service.py:436

• Cancellation from CloudExecutor uses exit code 130, but this block treats any non-zero StatusCode as a RuntimeError. That bypasses the cancel-flag handling later in the function and will mark cancelled runs as Status.Error instead of Status.Cancelled.

            exit_code = container_result["StatusCode"]
            if exit_code != 0:
                # Try to read a structured error written by the simulation container
                # (or moved into place by the cloud executor on remote failure).
                # If the container exits without writing {"error": {"message": ...}}

app/services/simulation_service.py:565

• New behavior is introduced here (propagating explicit RuntimeError messages into simulation_run.errorMessage/simulation.errorMessage and handling non-zero container exit codes). The existing unit/integration tests for run_solver / executors should be updated or extended to cover: (1) non-zero StatusCode returning a structured JSON error message, (2) cancellation exit code 130 mapping to Status.Cancelled, and (3) the expected shape of container.wait() results (dict with StatusCode).

        except RuntimeError as ex:
            # These are errors explicitly raised in the simulation-method
            # including a meaningful error message.
            # Propagate error messages to the database (frontend).
            error_msg = str(ex)
            logger.error(f"Simulation error: {error_msg}")
            simulation_run.status = Status.Error
            simulation_run.errorMessage = error_msg
            simulation.status = Status.Error
            simulation.errorMessage = error_msg
            session.commit()

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (7)

app/services/simulation_service.py:576

• This is not a generic frontend message because str(ex) can include filesystem paths, database details, or other internal data. The PR explicitly requires unexpected failures to use a generic message; keep the traceback in the log but do not persist exception text to the API-facing field.

            error_msg = f"An unexpected error occurred: {str(ex)}"

app/models/Simulation.py:26

• This model change has no accompanying Alembic migration. db.create_all() only creates missing tables and does not add columns to existing simulations tables, so upgraded deployments will fail when ORM queries reference errorMessage. Add a migration that adds this nullable column.

    errorMessage = db.Column(db.String, nullable=True)

app/services/executors/cloud_executor.py:830

• When output download or cleanup fails, _poll_until_complete returns (False, 0), and this return preserves status code 0. run_solver therefore treats a failed output collection as successful and can mark the simulation completed. Convert this case to a non-zero job status while retaining the cleanup-specific log message.

        elif not success:
            logs_output = "Cloud job completed but cleanup failed"
        else:
            logs_output = "Cloud job completed successfully"

        return _CompletedJob(exit_code=exit_code, logs_output=logs_output)

app/models/SimulationRun.py:22

• This column also needs to be added to existing simulationRuns tables by an Alembic migration. Updating the ORM model alone leaves deployed databases unchanged, causing queries and writes through this model to fail after rollout.

    errorMessage = db.Column(db.String, nullable=True)

app/services/simulation_service.py:558

• This catches every built-in RuntimeError, not only a user-facing simulation-method error. For example, the export failures at lines 512/533 and the local mount failure in local_executor.py:44 also reach this handler and expose their internal text. Raise and catch a dedicated exception only for the structured container error, allowing unrelated runtime failures to use the generic handler.

        except RuntimeError as ex:

app/services/simulation_service.py:568

• Once this value is set, a retry never clears it: start_solver_task resets the Simulation status at lines 261-269, and the success path sets Completed, but neither resets errorMessage. A successful rerun will therefore still serialize the previous failure. Clear the simulation error when starting a new run (and/or when marking it successful).

            simulation.status = Status.Error
            simulation.errorMessage = error_msg

app/services/simulation_service.py:586

• error_msg is only logged here, so failures outside the inner block never propagate an error status/message. For example, if the linked Simulation cannot be loaded, the handler leaves the run at its last committed state. When a SimulationRun has been loaded, update and commit its error state here; update Simulation as well when available.

    except Exception as ex:
        session.rollback()
        error_msg = f"Failed to initialize simulation: {str(ex)}"
        logger.error(error_msg)

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

app/services/executors/cloud_executor.py:573

• A completed remote simulation is still reported with exit code 0 when output download or cleanup fails. _collect_outputs_and_cleanup() returns False for SFTP download failures as well as cleanup failures, so run_solver() will mark the run Completed even though its artifacts may be missing. Return a non-zero exit code when success is false.

                return (success, 0)  # Exit code 0 for success

app/services/simulation_service.py:598

• This client-facing message includes the raw exception text, despite the PR requirement that unexpected failures use a generic message. Exception strings can expose internal paths, connection details, or other implementation data; keep the traceback in server logs but persist only a fixed generic message, and update the corresponding test expectation.

            error_msg = f"An unexpected error occurred: {str(ex)}"

app/models/SimulationRun.py:22

• These new persisted columns have no database migration. create_all() only creates missing tables and does not add columns to existing ones, while the production entrypoint does not run database creation at all; deployed databases will therefore fail when these attributes are queried or written. Add an upgrade that adds nullable errorMessage columns to both simulations and simulationRuns.

    errorMessage = db.Column(db.String, nullable=True)

app/models/Simulation.py:26

• This value is never cleared when start_solver_task() reruns an existing Simulation: that path resets the status to Created and creates a fresh SimulationRun, but the Simulation retains its previous errorMessage. After a successful retry the API can therefore serialize an old failure message alongside Completed status. Clear simulation.errorMessage when starting a new run.

    errorMessage = db.Column(db.String, nullable=True)