diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 437d7be..a39eef5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -194,6 +194,15 @@ jobs: MYSQL_TEST_PASSWORD: chat2db-ci-root MYSQL_TEST_REQUIRED: "1" run: cargo test -p chat2db-core --test native_mysql_product --locked + - name: Verify native MySQL Console without Java + env: + MYSQL_TEST_HOST: 127.0.0.1 + MYSQL_TEST_PORT: "3306" + MYSQL_TEST_USER: root + MYSQL_TEST_PASSWORD: chat2db-ci-root + run: >- + cargo test -p chat2db-core --test native_mysql_console_docker --locked + -- --ignored - uses: actions/setup-java@c1e323688fd81a25caa38c78aa6df2d33d3e20d9 # v4.7.1 with: distribution: temurin diff --git a/Cargo.lock b/Cargo.lock index 34a1754..65a9b07 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -776,11 +776,13 @@ name = "chat2db-web" version = "0.1.0" dependencies = [ "axum", + "base64 0.22.1", "chat2db-contract", "chat2db-core", "chat2db-java-bridge", "chat2db-local", "chat2db-storage", + "chrono", "futures-util", "http-body-util", "serde", @@ -803,8 +805,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" dependencies = [ "iana-time-zone", + "js-sys", "num-traits", "serde", + "wasm-bindgen", "windows-link 0.2.1", ] diff --git a/Cargo.toml b/Cargo.toml index 6b1b24e..8ec4305 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -29,6 +29,7 @@ async-trait = "0.1" axum = "0.8.9" base64 = "0.22.1" bytes = "1" +chrono = "0.4.45" clap = { version = "4.5", features = ["derive"] } directories = "6.0" eventsource-stream = "0.2.3" diff --git a/Makefile b/Makefile index 81ca509..89336ab 100644 --- a/Makefile +++ b/Makefile @@ -70,6 +70,11 @@ native-mysql-integration: MYSQL_TEST_PASSWORD="$(MYSQL_TEST_PASSWORD)" \ MYSQL_TEST_REQUIRED="1" \ cargo test -p chat2db-core --test native_mysql_product --locked + @MYSQL_TEST_HOST="$(MYSQL_TEST_HOST)" \ + MYSQL_TEST_PORT="$(MYSQL_TEST_PORT)" \ + MYSQL_TEST_USER="$(MYSQL_TEST_USER)" \ + MYSQL_TEST_PASSWORD="$(MYSQL_TEST_PASSWORD)" \ + cargo test -p chat2db-core --test native_mysql_console_docker --locked -- --ignored community-product-mysql-integration: java community-h2-classpath mysql-driver-pack @test -n "$(MYSQL_TEST_USER)" || (echo "MYSQL_TEST_USER is required" >&2; exit 1) diff --git a/README.md b/README.md index 0d06628..7c0d23a 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ Source-available implementation of the Chat2DB Community hybrid runtime. Chat2DB Rust owns the product runtime in Rust, uses a native Rust path for the -current MySQL browser and SELECT slice, and retains the broader public Chat2DB +current MySQL browser and Console data plane, and retains the broader public Chat2DB Community database compatibility layer behind a supervised Java process. The repository is under active development and is not yet a stable end-user release. @@ -27,7 +27,7 @@ git submodule update --init --recursive ## Current state The repository has completed Stages 1 through 6, the first thirteen -independently buildable Stage 7 slices, and the first end-user Community +independently buildable Stage 7 slices, and a native end-user Community Console compatibility slice: - canonical Rust API contracts; @@ -99,11 +99,14 @@ Console compatibility slice: - durable SQLite-backed Community Console create/get/list/update/delete, including SQL text, datasource/database/schema binding, saved status, and open-tab state across process restarts; and -- Community Console SELECT execution through upstream `mysql_async 0.37.0`, a - MySQL read-only transaction, and the existing bounded Core retained-result - path without starting Java: Web uses the historical synchronous result shape, - while desktop maps operation lifecycle, rows, failure, and cancellation to - the original JCEF event bus; +- Community Console execution through upstream `mysql_async 0.37.0` without + starting Java: unparameterized reads, DDL/DML, semicolon and `DELIMITER` + scripts, multiple result sets, explicit transactions, error-continue policy, + preserved-single dispatch, `EXPLAIN`, normal/all-row paging, datasource + read-only enforcement, cancellation, a shared 64 MiB result budget, bounded + large-cell tokens/downloads, and durable per-statement history; Web uses the + historical synchronous result shape while desktop emits the original JCEF + statement/result/row/update-count lifecycle; - a shared Web/Tauri legacy dispatcher: Axum maps the original `/api` routes, while desktop preserves the original JCEF correlation envelope through one `legacy_request` Tauri command; and @@ -135,7 +138,10 @@ The legacy boundary matches paged table searches against names or comments, ignores `searchKey` on Community's complete-list endpoints, validates metadata `pageSize` in `1..=100000`, returns binding failures in the HTTP 200 JSON envelope, and preserves `defaultValue: null` separately from an empty-string -default. +default. The native Console integration additionally passes against MySQL 8.4 +for DDL/DML, `DELIMITER` procedures, multi-results, transactions, error +continuation, cancellation, a 6 MiB `LONGTEXT`, `single`, `EXPLAIN`, +`pageSizeAll`, and datasource read-only protection while Java remains dormant. Stage 6 is complete. Web and desktop own the product runtime and publish its owner-only local endpoint; CLI and MCP attach to that host and never contact @@ -172,19 +178,19 @@ Tauri, and the shared frontend. Stage 7M adds `community.dql-builder.v1` at tag build a row-limited SELECT without opening JDBC, then Rust validates that SQL and executes it through the existing forced-read-only query and retained-result path. The fixed 149-JAR classpath keeps H2 and MySQL; PostgreSQL and other -dialects do not block the MySQL preview. MySQL writes, Agent, CLI, and MCP -conformance remain outside this read-only milestone. The current MySQL -connection, object metadata, preview, and supported Console SELECT routes -dispatch to `mysql_async` before Java lease acquisition; Community parser, -formatter, completion, and builders remain Java-backed. - -The first Console compatibility slice adds SQLite migration 3 for saved -Consoles and the historical `/api/operation/saved/*` plus -`/api/rdb/dml/execute` routes. Web waits on the Core operation and returns the -existing Community grid result. Desktop starts the same Core query, emits the -original `sql_execution_event` sequence through Tauri, and supports -`sql-cancel`. This slice intentionally supports query/SELECT execution only; -arbitrary DDL, DML, and multi-statement Console scripts are not implemented. +dialects do not block the MySQL preview. Agent, CLI, and MCP MySQL conformance +remain outside this milestone. The current MySQL connection, object metadata, +preview, and Console data plane dispatch to `mysql_async` before Java lease +acquisition; Community parser, formatter, completion, and builders remain +Java-backed. + +The Console compatibility path uses SQLite migrations 3 and 4 for saved +Consoles and durable execution history. Historical `/api/operation/saved/*`, +`/api/operation/log/*`, `/api/rdb/dml/execute`, `/execute_ddl`, and large-cell +routes share the same native Core execution. Desktop `sql-execute` and +`sql-cancel` keep active cancellation handles and emit row payloads exactly once +through Tauri. Native bind parameters and remaining edge-case Community result +shapes are not implemented. The Stage 5 and Stage 7G through Stage 7M custom React workbench was an intermediate implementation and is no longer the product frontend. Commit diff --git a/apps/chat2db-desktop/src/lib.rs b/apps/chat2db-desktop/src/lib.rs index a3070d2..a103014 100644 --- a/apps/chat2db-desktop/src/lib.rs +++ b/apps/chat2db-desktop/src/lib.rs @@ -39,7 +39,8 @@ use chat2db_contract::{ UpdateDatasourceRequest, UpdateProviderProfileRequest, ValidateCommunitySqlRequest, }; use chat2db_core::{ - AppError, Application, RuntimeConfig, RuntimeHost, load_fixed_community_classpath, + AppError, Application, MysqlConsoleCancellation, RuntimeConfig, RuntimeHost, + load_fixed_community_classpath, }; use chat2db_java_bridge::{BridgeError, EngineCommand, EngineConfig}; use chat2db_local::{LocalError, LocalServer}; @@ -114,10 +115,53 @@ struct DesktopState { application: Application, local_server: Mutex>, runtime_host: Mutex>, + legacy_sql_cancellations: LegacySqlCancellationRegistry, subscriptions: SubscriptionRegistry, + next_legacy_execution_id: AtomicU64, next_subscription_id: AtomicU64, } +#[derive(Default)] +struct LegacySqlCancellationRegistry { + cancellations: Mutex>, +} + +impl LegacySqlCancellationRegistry { + async fn insert(&self, execution_id: String, cancellation: MysqlConsoleCancellation) { + self.cancellations + .lock() + .await + .insert(execution_id, cancellation); + } + + async fn remove(&self, execution_id: &str) { + self.cancellations.lock().await.remove(execution_id); + } + + async fn cancel(&self, execution_id: &str, reason: Option) -> bool { + let cancellation = self.cancellations.lock().await.get(execution_id).cloned(); + cancellation.is_some_and(|cancellation| cancellation.cancel(reason)) + } + + async fn cancel_all(&self) { + let cancellations = self + .cancellations + .lock() + .await + .drain() + .map(|(_, cancellation)| cancellation) + .collect::>(); + for cancellation in cancellations { + let _ = cancellation.cancel(Some("The desktop runtime is shutting down".to_owned())); + } + } + + #[cfg(test)] + async fn active_count(&self) -> usize { + self.cancellations.lock().await.len() + } +} + #[derive(Default)] struct SubscriptionRegistry { controls: Mutex>, @@ -199,12 +243,15 @@ impl DesktopState { application, local_server: Mutex::new(Some(local_server)), runtime_host: Mutex::new(Some(runtime_host)), + legacy_sql_cancellations: LegacySqlCancellationRegistry::default(), subscriptions: SubscriptionRegistry::default(), + next_legacy_execution_id: AtomicU64::new(1), next_subscription_id: AtomicU64::new(1), }) } async fn shutdown(&self) -> Result<(), DesktopError> { + self.legacy_sql_cancellations.cancel_all().await; self.subscriptions.release_all().await; let local_server = self.local_server.lock().await.take(); let local_result = match local_server { @@ -570,15 +617,15 @@ async fn legacy_request( window: WebviewWindow, request: String, ) -> Result { - if let Some(response) = legacy_client_command_for(&state.application, &window, &request).await? - { + if let Some(response) = legacy_client_command_for(state.inner(), &window, &request).await? { return Ok(response); } legacy_request_for(&state.application, &request).await } +#[allow(clippy::too_many_lines)] async fn legacy_client_command_for( - application: &Application, + state: &Arc, window: &WebviewWindow, request: &str, ) -> Result, String> { @@ -601,11 +648,51 @@ async fn legacy_client_command_for( let sql_request = decode_client_message::( request.get("message"), )?; - let accepted = chat2db_web::legacy::start_sql_execution(application, &sql_request) + if chat2db_web::legacy::uses_native_mysql_console(&state.application, &sql_request) .await - .map_err(|error| legacy_failure_message(&error))?; + .map_err(|error| legacy_failure_message(&error))? + { + let execution_id = state + .next_legacy_execution_id + .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |current| { + current.checked_add(1) + }) + .map(|id| format!("mysql-console-{id}")) + .map_err(|_| "No MySQL Console execution ids remain".to_owned())?; + let cancellation = MysqlConsoleCancellation::new(); + state + .legacy_sql_cancellations + .insert(execution_id.clone(), cancellation.clone()) + .await; + let task_state = Arc::clone(state); + let task_window = window.clone(); + let task_execution_id = execution_id.clone(); + tauri::async_runtime::spawn(async move { + forward_native_mysql_sql_execution( + Arc::clone(&task_state), + task_window, + request_uuid, + task_execution_id.clone(), + sql_request, + cancellation, + ) + .await; + task_state + .legacy_sql_cancellations + .remove(&task_execution_id) + .await; + }); + return Ok(Some(client_command_response(&serde_json::json!({ + "executionId": execution_id, + })))); + } + + let accepted = + chat2db_web::legacy::start_sql_execution(&state.application, &sql_request) + .await + .map_err(|error| legacy_failure_message(&error))?; let execution_id = accepted.operation_id.clone(); - let task_application = application.clone(); + let task_application = state.application.clone(); let task_window = window.clone(); let task_execution_id = execution_id.clone(); tauri::async_runtime::spawn(async move { @@ -629,8 +716,23 @@ async fn legacy_client_command_for( .and_then(serde_json::Value::as_str) .filter(|value| !value.trim().is_empty()) .ok_or_else(|| "sql-cancel requires a non-empty executionId".to_owned())?; - let cancelled = application.cancel_operation(execution_id).await.disposition - == CancelDisposition::Accepted; + let cancelled = if state + .legacy_sql_cancellations + .cancel( + execution_id, + Some("The SQL execution was cancelled".to_owned()), + ) + .await + { + true + } else { + state + .application + .cancel_operation(execution_id) + .await + .disposition + == CancelDisposition::Accepted + }; Ok(Some(client_command_response(&serde_json::json!(cancelled)))) } _ => Ok(None), @@ -656,6 +758,184 @@ fn legacy_failure_message(error: &chat2db_web::legacy::LegacyFailure) -> String format!("{}: {}", error.code, error.message) } +#[allow(clippy::too_many_lines)] +async fn forward_native_mysql_sql_execution( + state: Arc, + window: WebviewWindow, + request_uuid: String, + execution_id: String, + request: chat2db_web::legacy::LegacySqlExecuteRequest, + cancellation: MysqlConsoleCancellation, +) { + let started_at = Instant::now(); + let mut sequence = 0_u64; + if emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "started", + None, + None, + &serde_json::json!({ "executionId": execution_id }), + ) + .is_err() + { + let _ = cancellation.cancel(Some("The SQL event receiver closed".to_owned())); + return; + } + + let results = chat2db_web::legacy::execute_mysql_sql( + &state.application, + &request, + cancellation.clone(), + &execution_id, + "SQL_EDITOR_JCEF", + ) + .await; + if cancellation.is_cancelled() { + let _ = emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "cancelled", + None, + None, + &serde_json::json!({ + "executionId": execution_id, + "message": "The SQL execution was cancelled", + }), + ); + return; + } + let results = match results { + Ok(results) if !results.is_empty() => results, + Ok(_) => { + emit_legacy_terminal_error( + &window, + &request_uuid, + &execution_id, + &mut sequence, + &ApiError::new( + "sql_execution_incomplete", + "The SQL execution ended without a result", + ), + ); + return; + } + Err(error) => { + emit_legacy_terminal_error( + &window, + &request_uuid, + &execution_id, + &mut sequence, + &ApiError::new(error.code, error.message), + ); + return; + } + }; + + let mut active_statement = None; + let mut active_sql = String::new(); + let mut active_duration = 0_u64; + let mut fallback_statement_sequence = 0_u32; + let mut fallback_result_sequence = 0_u32; + for result in results { + let statement_sequence = result.statement_sequence.unwrap_or_else(|| { + fallback_statement_sequence = fallback_statement_sequence.saturating_add(1); + fallback_statement_sequence + }); + if active_statement != Some(statement_sequence) { + if let Some(previous_statement) = active_statement { + let _ = emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "statementFinished", + Some(previous_statement), + None, + &serde_json::json!({ + "sql": active_sql, + "duration": active_duration, + }), + ); + } + active_statement = Some(statement_sequence); + active_sql.clone_from(&result.sql); + active_duration = 0; + fallback_result_sequence = 0; + if emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "statementStarted", + Some(statement_sequence), + None, + &serde_json::json!({ + "sql": result.sql, + "originalSql": result.original_sql, + "sequence": statement_sequence, + }), + ) + .is_err() + { + let _ = cancellation.cancel(Some("The SQL event receiver closed".to_owned())); + return; + } + } + fallback_result_sequence = fallback_result_sequence.saturating_add(1); + let result_sequence = result.result_set_id.unwrap_or(fallback_result_sequence); + active_duration = active_duration.saturating_add(result.duration); + if emit_legacy_sql_result_events( + &window, + &request_uuid, + &execution_id, + &mut sequence, + statement_sequence, + result_sequence, + result, + ) + .is_err() + { + let _ = cancellation.cancel(Some("The SQL event receiver closed".to_owned())); + return; + } + } + + if let Some(statement_sequence) = active_statement { + let _ = emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "statementFinished", + Some(statement_sequence), + None, + &serde_json::json!({ + "sql": active_sql, + "duration": active_duration, + }), + ); + } + let duration = u64::try_from(started_at.elapsed().as_millis()).unwrap_or(u64::MAX); + let _ = emit_legacy_sql_event( + &window, + &request_uuid, + &execution_id, + &mut sequence, + "finished", + None, + None, + &serde_json::json!({ + "executionId": execution_id, + "duration": duration, + }), + ); +} + #[allow(clippy::too_many_lines)] async fn forward_legacy_sql_execution( application: Application, @@ -770,6 +1050,8 @@ async fn forward_legacy_sql_execution( &request_uuid, &execution_id, &mut sequence, + 1, + 1, result, ) .is_err() @@ -833,31 +1115,43 @@ fn emit_legacy_sql_result_events( request_uuid: &str, execution_id: &str, sequence: &mut u64, + statement_sequence: u32, + result_sequence: u32, result: chat2db_web::legacy::LegacyManageResult, ) -> Result<(), tauri::Error> { - let mut started = serde_json::to_value(&result).unwrap_or_else(|_| serde_json::json!({})); - if let Some(started) = started.as_object_mut() { - started.insert("dataList".to_owned(), serde_json::json!([])); + let tabular = result.result_set_id.is_some() || !result.header_list.is_empty(); + let mut rows = serde_json::to_value(result).unwrap_or_else(|_| serde_json::json!({})); + if !tabular { + return emit_legacy_sql_event( + window, + request_uuid, + execution_id, + sequence, + "updateCount", + Some(statement_sequence), + Some(result_sequence), + &rows, + ); } + let rowless = legacy_sql_rowless_payload(&mut rows); emit_legacy_sql_event( window, request_uuid, execution_id, sequence, "resultStarted", - Some(1), - Some(1), - &started, + Some(statement_sequence), + Some(result_sequence), + &rowless, )?; - let rows = serde_json::to_value(&result).unwrap_or_else(|_| serde_json::json!({})); emit_legacy_sql_event( window, request_uuid, execution_id, sequence, "rows", - Some(1), - Some(1), + Some(statement_sequence), + Some(result_sequence), &rows, )?; emit_legacy_sql_event( @@ -866,12 +1160,24 @@ fn emit_legacy_sql_result_events( execution_id, sequence, "resultFinished", - Some(1), - Some(1), - &serde_json::to_value(result).unwrap_or_else(|_| serde_json::json!({})), + Some(statement_sequence), + Some(result_sequence), + &rowless, ) } +fn legacy_sql_rowless_payload(rows: &mut serde_json::Value) -> serde_json::Value { + let Some(rows_object) = rows.as_object_mut() else { + return serde_json::json!({}); + }; + let data_list = rows_object + .insert("dataList".to_owned(), serde_json::json!([])) + .unwrap_or_else(|| serde_json::json!([])); + let rowless = serde_json::Value::Object(rows_object.clone()); + rows_object.insert("dataList".to_owned(), data_list); + rowless +} + fn emit_legacy_terminal_error( window: &WebviewWindow, request_uuid: &str, @@ -1883,16 +2189,17 @@ mod tests { OperationEvent, OperationEventEnvelope, OperationStreamMessage, StartCommunityTablePreviewRequest, ValidateCommunitySqlRequest, }; - use chat2db_core::{AppError, Application}; + use chat2db_core::{AppError, Application, MysqlConsoleCancellation}; use tokio::sync::oneshot; use super::{ BUNDLED_COMMUNITY_CLASSPATH, BUNDLED_DRIVER_PACKS, BUNDLED_JAVA_BIN, - BUNDLED_JAVA_ENGINE_JAR, BundledRuntimeResources, DesktopError, RuntimeResourceOverrides, - SubscriptionRegistry, agent_stream_message, build_community_dml_for, - build_community_namespace_sql_for, client_command_response, complete_community_sql_for, - decode_client_message, format_community_sql_for, legacy_request_for, - legacy_sql_push_message, operation_stream_message, parse_after_sequence, + BUNDLED_JAVA_ENGINE_JAR, BundledRuntimeResources, DesktopError, + LegacySqlCancellationRegistry, RuntimeResourceOverrides, SubscriptionRegistry, + agent_stream_message, build_community_dml_for, build_community_namespace_sql_for, + client_command_response, complete_community_sql_for, decode_client_message, + format_community_sql_for, legacy_request_for, legacy_sql_push_message, + legacy_sql_rowless_payload, operation_stream_message, parse_after_sequence, resolve_runtime_resource_paths, start_community_table_preview_for, validate_community_sql_for, validate_java_engine_jar, validate_optional_os_env, }; @@ -1967,6 +2274,48 @@ mod tests { assert_eq!(message["message"]["message"]["success"], true); } + #[test] + fn community_sql_result_rows_are_only_present_in_the_rows_event() { + let mut rows = serde_json::json!({ + "success": true, + "headerList": [{ "name": "id" }], + "dataList": [[{ "value": "1" }], [{ "value": "2" }]], + "resultSetId": 1, + }); + + let rowless = legacy_sql_rowless_payload(&mut rows); + + assert_eq!(rows["dataList"].as_array().map(Vec::len), Some(2)); + assert_eq!(rowless["dataList"], serde_json::json!([])); + assert_eq!(rowless["headerList"], rows["headerList"]); + assert_eq!(rowless["resultSetId"], 1); + } + + #[tokio::test] + async fn native_mysql_cancellation_registry_owns_execution_lifecycle() { + let registry = LegacySqlCancellationRegistry::default(); + let cancellation = MysqlConsoleCancellation::new(); + registry + .insert("mysql-console-1".to_owned(), cancellation.clone()) + .await; + + assert_eq!(registry.active_count().await, 1); + assert!( + registry + .cancel("mysql-console-1", Some("cancelled by test".to_owned())) + .await + ); + assert!(cancellation.is_cancelled()); + assert!( + !registry + .cancel("mysql-console-1", Some("second cancellation".to_owned())) + .await + ); + + registry.remove("mysql-console-1").await; + assert_eq!(registry.active_count().await, 0); + } + #[tokio::test] async fn legacy_request_preserves_the_community_jcef_response_shape() { let response = legacy_request_for( diff --git a/apps/chat2db-web/Cargo.toml b/apps/chat2db-web/Cargo.toml index 1af8b43..f192297 100644 --- a/apps/chat2db-web/Cargo.toml +++ b/apps/chat2db-web/Cargo.toml @@ -11,11 +11,13 @@ repository.workspace = true [dependencies] axum.workspace = true +base64.workspace = true chat2db-contract = { path = "../../crates/chat2db-contract" } chat2db-core = { path = "../../crates/chat2db-core" } chat2db-java-bridge = { path = "../../crates/chat2db-java-bridge" } chat2db-local = { path = "../../crates/chat2db-local" } chat2db-storage = { path = "../../crates/chat2db-storage" } +chrono.workspace = true futures-util.workspace = true serde.workspace = true serde_json.workspace = true diff --git a/apps/chat2db-web/src/legacy.rs b/apps/chat2db-web/src/legacy.rs index 75d208d..abf1dd9 100644 --- a/apps/chat2db-web/src/legacy.rs +++ b/apps/chat2db-web/src/legacy.rs @@ -5,18 +5,22 @@ //! response translations without duplicating datasource or JDBC behavior. use std::{ - collections::HashSet, - time::{Duration, Instant}, + collections::{BTreeMap, HashSet}, + fs, + path::PathBuf, + time::{Duration, Instant, SystemTime, UNIX_EPOCH}, }; use axum::{ Json, Router, + body::Body, extract::{Query, State}, - http::StatusCode, + http::{StatusCode, header}, middleware, response::{IntoResponse, Response}, routing::{get, post}, }; +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use chat2db_contract::{ ApiError, ColumnNullability, CommunityFunction, CommunityProcedure, CommunityTable, CommunityTableColumn, CommunityTableIndex, CommunityTableIndexColumn, CommunityTrigger, @@ -29,11 +33,15 @@ use chat2db_contract::{ QueryLimits, ResultColumn, ResultMetadata, ResultPageRequest, StartCommunityTablePreviewRequest, StartQueryRequest, UpdateDatasourceRequest, }; -use chat2db_core::{AppError, Application}; +use chat2db_core::{ + AppError, Application, LargeValueChunk, LargeValueEncoding, LargeValuePreview, LargeValueType, + MysqlConsoleCancellation, MysqlConsoleRequest, MysqlConsoleResult, +}; use chat2db_storage::{ - CreateSavedConsole, SavedConsoleListQuery, SavedConsoleRecord, Storage, StorageError, - UpdateSavedConsole, + CreateOperationLog, CreateSavedConsole, OperationLogListQuery, OperationLogRecord, + SavedConsoleListQuery, SavedConsoleRecord, Storage, StorageError, UpdateSavedConsole, }; +use chrono::{Local, TimeZone as _}; use serde::{Deserialize, Deserializer, Serialize, de::DeserializeOwned}; const DEFAULT_PAGE_NO: u32 = 1; @@ -43,9 +51,12 @@ const DEFAULT_SQL_PAGE_SIZE: u32 = 200; const MAX_METADATA_PAGE_SIZE: u32 = 100_000; const MAX_PREVIEW_ROWS: u32 = 1_000; const MAX_SQL_ROWS: u32 = 10_000; +const LARGE_VALUE_CHUNK_SIZE: u32 = 256 * 1024; +const LARGE_VALUE_FALLBACK_PREVIEW_BYTES: usize = 64 * 1024; const RESULT_PAGE_MAX_BYTES: u64 = 8 * 1024 * 1024; const PREVIEW_TIMEOUT: Duration = Duration::from_secs(30); const SQL_EXECUTION_TIMEOUT: Duration = Duration::from_secs(60); +const SQL_CANCELLATION_GRACE: Duration = Duration::from_secs(10); /// Community's historical response wrapper. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -432,10 +443,16 @@ pub struct LegacyResultHeader { pub column_name: String, pub column_type: String, pub table_name: Option, + pub database_name: Option, + pub schema_name: Option, pub primary_key: bool, + pub comment: Option, + pub default_value: Option, + pub auto_increment: Option, pub nullable: bool, pub column_size: Option, pub decimal_digits: Option, + pub editor_type: Option, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -443,10 +460,68 @@ pub struct LegacyResultHeader { pub struct LegacyResultCell { pub value: Option, pub large_value: bool, + pub large_value_id: Option, pub value_type: String, pub sql_type: i32, pub column_type: String, + pub size_bytes: Option, + pub size_chars: Option, + pub loaded_bytes: Option, + pub loaded_chars: Option, pub truncated: bool, + pub unsupported_reason: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyLargeCellValueRequest { + pub large_value_id: String, + #[serde(default)] + pub offset: u64, + #[serde(default = "default_large_value_chunk_size")] + pub limit: u32, + #[serde(default = "default_large_value_format")] + pub format: String, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyLargeCellDownloadRequest { + pub large_value_id: String, + #[serde(default = "default_large_value_download_format")] + pub format: String, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyLargeCellChunk { + pub value: String, + pub offset: u64, + pub next_offset: u64, + pub eof: bool, + pub size_bytes: u64, + pub size_chars: Option, + pub encoding: String, + pub content_type: String, + pub display_mode: LargeValueType, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyExecutionMetrics { + pub started_at_epoch_ms: u64, + pub finished_at_epoch_ms: u64, + pub total_duration_ms: u64, + pub execute_duration_ms: u64, + pub fetch_duration_ms: u64, + pub fetched_row_count: u64, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyExecutionContext { + pub database_name: Option, + pub schema_name: Option, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -471,6 +546,11 @@ pub struct LegacyManageResult { pub has_next_page: bool, pub execute_sql_params: LegacySqlExecuteRequest, pub extra: serde_json::Value, + pub comment: Option, + pub result_set_id: Option, + pub statement_sequence: Option, + pub execution_metrics: Option, + pub execution_context: Option, } #[derive(Debug, Clone, Deserialize)] @@ -599,6 +679,88 @@ pub struct LegacySavedConsoleResponse { pub operation_type: String, } +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyOperationLogCreateRequest { + #[serde(default)] + pub id: Option, + #[serde(default, deserialize_with = "deserialize_string_or_default")] + pub name: String, + #[serde(default)] + pub data_source_id: Option, + #[serde(default)] + pub data_source_name: Option, + #[serde(default)] + pub connectable: Option, + #[serde(default)] + pub database_name: Option, + #[serde(rename = "type", default)] + pub database_type: Option, + #[serde(default, deserialize_with = "deserialize_string_or_default")] + pub ddl: String, + #[serde(default)] + pub more: bool, + #[serde(default)] + pub status: Option, + #[serde(default)] + pub operation_rows: Option, + #[serde(default)] + pub use_time: Option, + #[serde(default)] + pub extend_info: Option, + #[serde(default)] + pub schema_name: Option, + #[serde(default)] + pub organization_id: Option, + #[serde(default)] + pub user_name: Option, +} + +#[derive(Debug, Clone, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyOperationLogListQuery { + #[serde(default = "default_page_no")] + pub page_no: u32, + #[serde(default = "default_page_size")] + pub page_size: u32, + #[serde(default, deserialize_with = "deserialize_string_or_default")] + pub search_key: String, + #[serde(default)] + pub data_source_id: Option, + #[serde(default)] + pub database_name: Option, + #[serde(default)] + pub schema_name: Option, + #[serde(default)] + pub organization_id: Option, + #[serde(default)] + pub operation_type: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LegacyOperationLogResponse { + pub id: i64, + pub gmt_create: String, + pub gmt_modified: String, + pub name: String, + pub data_source_id: Option, + pub data_source_name: Option, + pub connectable: Option, + pub database_name: Option, + #[serde(rename = "type")] + pub database_type: Option, + pub ddl: String, + pub more: bool, + pub status: Option, + pub operation_rows: Option, + pub use_time: Option, + pub extend_info: Option, + pub schema_name: Option, + pub organization_id: Option, + pub user_name: Option, +} + #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "camelCase")] pub struct LegacyIdQuery { @@ -744,9 +906,17 @@ pub struct LegacySqlExecuteRequest { #[serde(default = "default_sql_page_size")] pub page_size: u32, #[serde(default)] + pub page_size_all: bool, + #[serde(default)] pub console_id: Option, #[serde(default)] + pub apply_id: Option, + #[serde(default)] pub result_set_id: Option, + #[serde(default)] + pub error_continue: Option, + #[serde(default)] + pub explain: bool, } /// Transport-neutral request used by the retained desktop command-line bridge. @@ -790,6 +960,18 @@ fn default_sql_page_size() -> u32 { DEFAULT_SQL_PAGE_SIZE } +const fn default_large_value_chunk_size() -> u32 { + LARGE_VALUE_CHUNK_SIZE +} + +fn default_large_value_format() -> String { + "base64".to_owned() +} + +fn default_large_value_download_format() -> String { + "raw".to_owned() +} + impl From<&LegacyTablePreviewRequest> for LegacySqlExecuteRequest { fn from(request: &LegacyTablePreviewRequest) -> Self { Self { @@ -803,8 +985,12 @@ impl From<&LegacyTablePreviewRequest> for LegacySqlExecuteRequest { single: true, page_no: request.page_no, page_size: request.page_size, + page_size_all: false, console_id: None, + apply_id: None, result_set_id: None, + error_continue: None, + explain: false, } } } @@ -1089,6 +1275,123 @@ pub(crate) async fn delete_saved_console(application: &Application, id: i64) -> .map(|_| ()) } +pub(crate) async fn create_operation_log( + application: &Application, + request: &LegacyOperationLogCreateRequest, +) -> LegacyResult { + if request.ddl.trim().is_empty() { + return Err(LegacyFailure::invalid( + "invalid_operation_log", + "ddl is required", + )); + } + let operation_rows = request + .operation_rows + .map(i64::try_from) + .transpose() + .map_err(|_| { + LegacyFailure::invalid( + "invalid_operation_log", + "operationRows is outside the supported range", + ) + })?; + let use_time = request + .use_time + .map(i64::try_from) + .transpose() + .map_err(|_| { + LegacyFailure::invalid( + "invalid_operation_log", + "useTime is outside the supported range", + ) + })?; + let data_source_id = request + .data_source_id + .as_ref() + .map(LegacyIdentifier::as_string); + let storage = legacy_storage(application)?; + let input = CreateOperationLog { + name: non_blank(&request.name), + connectable: request + .connectable + .or_else(|| data_source_id.as_ref().map(|id| !id.trim().is_empty())), + data_source_id, + data_source_name: request.data_source_name.clone(), + database_name: request.database_name.clone(), + database_type: request.database_type.clone(), + ddl: request.ddl.clone(), + status: request + .status + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .unwrap_or("SUCCESS") + .to_owned(), + operation_rows, + use_time, + extend_info: request.extend_info.clone(), + schema_name: request.schema_name.clone(), + organization_id: request.organization_id, + user_name: request.user_name.clone(), + more: request.more, + operation_type: "SQL_EXECUTE".to_owned(), + }; + legacy_storage_call(move || storage.create_operation_log(input)) + .await + .map(|record| record.id) +} + +pub(crate) async fn get_operation_log( + application: &Application, + id: i64, +) -> LegacyResult { + let storage = legacy_storage(application)?; + let record = legacy_storage_call(move || storage.get_operation_log(id)) + .await? + .ok_or_else(|| { + LegacyFailure::invalid( + "operation_log_not_found", + "The operation log does not exist", + ) + })?; + Ok(operation_log_response(record, false)) +} + +pub(crate) async fn list_operation_logs( + application: &Application, + query: &LegacyOperationLogListQuery, +) -> LegacyResult> { + let storage = legacy_storage(application)?; + let storage_query = OperationLogListQuery { + data_source_id: query + .data_source_id + .as_ref() + .map(LegacyIdentifier::as_string), + database_name: query.database_name.clone(), + schema_name: query.schema_name.clone(), + operation_type: query.operation_type.clone(), + search_key: non_blank(&query.search_key), + page_no: query.page_no, + page_size: query.page_size, + }; + legacy_storage_call(move || storage.list_operation_logs(&storage_query)) + .await + .map(|page| { + let total = usize::try_from(page.total).unwrap_or(usize::MAX); + LegacyPage { + data: page + .records + .into_iter() + .map(|record| operation_log_response(record, true)) + .collect(), + page_no: page.page_no, + page_size: page.page_size, + total, + has_next_page: u64::from(page.page_no) * u64::from(page.page_size) < page.total, + } + }) +} + /// Builds the flat namespace tree used when no custom grouping exists. pub(crate) async fn namespace_tree( application: &Application, @@ -1530,6 +1833,7 @@ pub(crate) async fn preview_table( || page.metadata.truncated_by_max_rows || page.metadata.truncated_by_max_result_bytes; let headers: Vec = page.columns.iter().map(result_header).collect(); + let large_value_owner = application.create_large_value_owner(); let data_list = page .rows .into_iter() @@ -1537,7 +1841,7 @@ pub(crate) async fn preview_table( row.values .into_iter() .zip(page.columns.iter()) - .map(|(value, column)| result_cell(value, column)) + .map(|(value, column)| result_cell(application, &large_value_owner, value, column)) .collect() }) .collect(); @@ -1561,6 +1865,15 @@ pub(crate) async fn preview_table( has_next_page, execute_sql_params: LegacySqlExecuteRequest::from(request), extra: serde_json::json!({}), + comment: None, + result_set_id: None, + statement_sequence: Some(1), + execution_metrics: None, + execution_context: Some(LegacyExecutionContext { + database_name: (!request.database_name.is_empty()) + .then(|| request.database_name.clone()), + schema_name: (!request.schema_name.is_empty()).then(|| request.schema_name.clone()), + }), }]) } @@ -1575,20 +1888,7 @@ pub async fn start_sql_execution( application: &Application, request: &LegacySqlExecuteRequest, ) -> LegacyResult { - let (_, row_limit) = sql_page_window(request)?; - let datasource_id = request.data_source_id.as_string(); - if datasource_id.trim().is_empty() { - return Err(LegacyFailure::invalid( - "invalid_sql_execute_request", - "dataSourceId is required", - )); - } - if request.sql.trim().is_empty() { - return Err(LegacyFailure::invalid( - "invalid_sql_execute_request", - "sql is required", - )); - } + let (datasource_id, row_limit) = validate_sql_execute_request(request)?; application .start_query(StartQueryRequest { datasource_id, @@ -1670,6 +1970,7 @@ pub async fn read_sql_result( page.metadata.row_count.clone() }; let header_list = page.columns.iter().map(result_header).collect(); + let large_value_owner = application.create_large_value_owner(); let data_list = page .rows .into_iter() @@ -1677,7 +1978,7 @@ pub async fn read_sql_result( row.values .into_iter() .zip(page.columns.iter()) - .map(|(value, column)| result_cell(value, column)) + .map(|(value, column)| result_cell(application, &large_value_owner, value, column)) .collect() }) .collect(); @@ -1701,6 +2002,15 @@ pub async fn read_sql_result( has_next_page, execute_sql_params: request.clone(), extra: serde_json::json!({}), + comment: None, + result_set_id: request.result_set_id, + statement_sequence: Some(1), + execution_metrics: None, + execution_context: Some(LegacyExecutionContext { + database_name: (!request.database_name.is_empty()) + .then(|| request.database_name.clone()), + schema_name: (!request.schema_name.is_empty()).then(|| request.schema_name.clone()), + }), }) } @@ -1715,6 +2025,18 @@ pub async fn execute_sql( application: &Application, request: &LegacySqlExecuteRequest, ) -> LegacyResult> { + let _ = validate_sql_execute_request(request)?; + if uses_native_mysql_console(application, request).await? { + let execution_id = application.create_large_value_owner(); + return execute_mysql_sql( + application, + request, + MysqlConsoleCancellation::new(), + &execution_id, + "SQL_EDITOR_HTTP", + ) + .await; + } let started_at = Instant::now(); let accepted = start_sql_execution(application, request).await?; let terminal = tokio::time::timeout( @@ -1742,6 +2064,550 @@ pub async fn execute_sql( } } +/// Executes the Community single-result DDL contract. +/// +/// # Errors +/// +/// Returns request, datasource, execution, or missing-result failures. +pub async fn execute_ddl( + application: &Application, + request: &LegacySqlExecuteRequest, +) -> LegacyResult { + execute_sql(application, request) + .await? + .into_iter() + .next() + .ok_or_else(|| { + LegacyFailure::invalid( + "sql_execution_incomplete", + "The SQL execution ended without a result", + ) + }) +} + +/// Executes the native `MySQL` Console contract with a caller-owned cancellation +/// source. Desktop keeps this source by execution id while HTTP uses it for the +/// synchronous timeout boundary. +/// +/// # Errors +/// +/// Returns validation or datasource failures that occur before execution. +pub async fn execute_mysql_sql( + application: &Application, + request: &LegacySqlExecuteRequest, + cancellation: MysqlConsoleCancellation, + execution_id: &str, + history_source: &str, +) -> LegacyResult> { + let (datasource_id, _) = validate_sql_execute_request(request)?; + + let execution = application.execute_mysql_console( + MysqlConsoleRequest { + datasource_id, + database_name: request.database_name.clone(), + sql: request.sql.clone(), + page_no: request.page_no, + page_size: request.page_size, + result_set_id: request.result_set_id, + single: request.single, + page_size_all: request.page_size_all, + explain: request.explain, + error_continue: request.error_continue.unwrap_or(true), + }, + cancellation.clone(), + ); + let large_value_owner = application.create_large_value_owner(); + tokio::pin!(execution); + let timed_out = tokio::select! { + result = &mut execution => Some(result), + () = tokio::time::sleep(SQL_EXECUTION_TIMEOUT) => None, + }; + let results = match timed_out { + Some(Ok(results)) => results + .into_iter() + .map(|result| mysql_console_result(application, &large_value_owner, request, result)) + .collect(), + Some(Err(error)) => { + let error = LegacyFailure::from(error); + vec![sql_failure_result(request, &error, 0)] + } + None => { + let _ = cancellation.cancel(Some("The SQL execution timed out".to_owned())); + if tokio::time::timeout(SQL_CANCELLATION_GRACE, &mut execution) + .await + .is_err() + { + tracing::warn!( + "MySQL Console execution did not finish during the timeout cleanup window" + ); + } + vec![sql_failure_result( + request, + &LegacyFailure::invalid( + "sql_execution_timeout", + "The SQL execution did not finish in time", + ), + u64::try_from(SQL_EXECUTION_TIMEOUT.as_millis()).unwrap_or(u64::MAX), + )] + } + }; + record_mysql_console_history_best_effort( + application, + request, + &results, + execution_id, + history_source, + ) + .await; + Ok(results) +} + +async fn record_mysql_console_history_best_effort( + application: &Application, + request: &LegacySqlExecuteRequest, + results: &[LegacyManageResult], + execution_id: &str, + history_source: &str, +) { + let Some(storage) = application.storage().cloned() else { + return; + }; + let mut statements = BTreeMap::>::new(); + for (index, result) in results.iter().enumerate() { + let fallback = u32::try_from(index).unwrap_or(u32::MAX).saturating_add(1); + statements + .entry(result.statement_sequence.unwrap_or(fallback)) + .or_default() + .push(result); + } + for statement_results in statements.into_values() { + let Some(first) = statement_results.first().copied() else { + continue; + }; + let operation_rows = statement_results + .iter() + .map(|result| result.update_count) + .fold(0_u64, u64::saturating_add); + let use_time = statement_results + .iter() + .map(|result| result.duration) + .fold(0_u64, u64::saturating_add); + let message = statement_results + .iter() + .find(|result| !result.success && !result.message.trim().is_empty()) + .map_or("", |result| result.message.as_str()); + let extend_info = serde_json::to_string(&serde_json::json!({ + "source": history_source, + "sqlType": first.sql_type, + "executionId": execution_id, + "statementSequence": first.statement_sequence.unwrap_or(1), + "message": message, + })) + .ok(); + let input = CreateOperationLog { + name: None, + data_source_id: Some(request.data_source_id.as_string()), + data_source_name: non_blank(&request.data_source_name), + connectable: Some(true), + database_name: non_blank(&request.database_name), + database_type: Some(default_if_blank(&request.database_type, "MYSQL")), + ddl: first.original_sql.clone(), + status: mysql_console_history_status(statement_results.as_slice()).to_owned(), + operation_rows: i64::try_from(operation_rows).ok(), + use_time: i64::try_from(use_time).ok(), + extend_info, + schema_name: non_blank(&request.schema_name), + organization_id: None, + user_name: None, + more: first.original_sql.chars().count() > 200, + operation_type: "SQL_EXECUTE".to_owned(), + }; + let result = tokio::task::spawn_blocking({ + let storage = storage.clone(); + move || storage.create_operation_log(input) + }) + .await; + match result { + Ok(Ok(_)) => {} + Ok(Err(error)) => tracing::warn!(%error, "MySQL Console history write failed"), + Err(error) => tracing::warn!(%error, "MySQL Console history task failed"), + } + } +} + +fn mysql_console_history_status(results: &[&LegacyManageResult]) -> &'static str { + if results.iter().any(|result| { + result + .extra + .get("messages") + .and_then(serde_json::Value::as_array) + .into_iter() + .flatten() + .filter_map(|message| message.get("errorCode")) + .filter_map(serde_json::Value::as_str) + .any(|code| matches!(code, "mysql_console_cancelled" | "sql_execution_cancelled")) + }) { + "cancelled" + } else if results.iter().all(|result| result.success) { + "success" + } else { + "fail" + } +} + +/// Reads one bounded retained large-cell chunk in the requested display format. +/// +/// # Errors +/// +/// Returns validation, expired-token, range, decoding, or unsupported-format failures. +pub fn read_large_cell_value( + application: &Application, + request: &LegacyLargeCellValueRequest, +) -> LegacyResult { + if request.large_value_id.trim().is_empty() { + return Err(LegacyFailure::invalid( + "invalid_large_cell_value_request", + "largeValueId is required", + )); + } + let encoded = matches!( + request.format.trim().to_ascii_lowercase().as_str(), + "base64" | "hex" + ); + let chunk = if encoded { + application.read_large_value_encoded_chunk( + &request.large_value_id, + request.offset, + request.limit, + ) + } else { + application.read_large_value_chunk(&request.large_value_id, request.offset, request.limit) + } + .map_err(LegacyFailure::from)?; + format_large_value_chunk(chunk, &request.format) +} + +/// Writes one complete retained large-cell value to a unique temporary download path. +/// +/// # Errors +/// +/// Returns validation, expired-token, decoding, task, directory, or file-write failures. +pub async fn download_large_cell_value_to_path( + application: &Application, + request: &LegacyLargeCellDownloadRequest, +) -> LegacyResult { + let application = application.clone(); + let request = request.clone(); + tokio::task::spawn_blocking(move || { + let download = prepare_large_cell_download(&application, &request)?; + let directory = std::env::temp_dir().join("chat2db").join("downloads"); + fs::create_dir_all(&directory).map_err(|_| LegacyFailure { + code: "large_cell_download_failed".to_owned(), + message: "The large cell download directory could not be created".to_owned(), + })?; + let path = unique_large_cell_download_path( + &directory, + &request.large_value_id, + download.extension, + ); + fs::write(&path, download.bytes).map_err(|_| LegacyFailure { + code: "large_cell_download_failed".to_owned(), + message: "The large cell value could not be written to disk".to_owned(), + })?; + Ok(path.to_string_lossy().into_owned()) + }) + .await + .map_err(|_| LegacyFailure { + code: "large_cell_download_failed".to_owned(), + message: "The large cell download task did not finish".to_owned(), + })? +} + +struct PreparedLargeCellDownload { + bytes: Vec, + content_type: &'static str, + extension: &'static str, +} + +fn prepare_large_cell_download( + application: &Application, + request: &LegacyLargeCellDownloadRequest, +) -> LegacyResult { + if request.large_value_id.trim().is_empty() { + return Err(LegacyFailure::invalid( + "invalid_large_cell_value_request", + "largeValueId is required", + )); + } + let (raw, value_type) = read_complete_large_value(application, &request.large_value_id)?; + match request.format.trim().to_ascii_lowercase().as_str() { + "" | "raw" => Ok(PreparedLargeCellDownload { + bytes: raw, + content_type: if value_type == LargeValueType::Text { + "text/plain; charset=utf-8" + } else { + "application/octet-stream" + }, + extension: if value_type == LargeValueType::Text { + "txt" + } else { + "bin" + }, + }), + "text" => Ok(PreparedLargeCellDownload { + bytes: String::from_utf8_lossy(&raw).into_owned().into_bytes(), + content_type: "text/plain; charset=utf-8", + extension: "txt", + }), + "hex" => Ok(PreparedLargeCellDownload { + bytes: encode_hex(&raw).into_bytes(), + content_type: "text/plain; charset=utf-8", + extension: "hex", + }), + _ => Err(LegacyFailure::invalid( + "invalid_large_cell_value_request", + "format must be raw, text, or hex", + )), + } +} + +fn read_complete_large_value( + application: &Application, + large_value_id: &str, +) -> LegacyResult<(Vec, LargeValueType)> { + let mut offset = 0_u64; + let mut output = Vec::new(); + let mut value_type = None; + loop { + let chunk = application + .read_large_value_chunk(large_value_id, offset, LARGE_VALUE_CHUNK_SIZE) + .map_err(LegacyFailure::from)?; + value_type.get_or_insert(chunk.display_mode); + output.extend(decode_large_value_chunk(&chunk)?); + if chunk.eof { + return Ok((output, value_type.unwrap_or(LargeValueType::Binary))); + } + if chunk.next_offset <= offset { + return Err(LegacyFailure { + code: "large_cell_download_failed".to_owned(), + message: "The large cell value did not advance while downloading".to_owned(), + }); + } + offset = chunk.next_offset; + } +} + +fn format_large_value_chunk( + chunk: LargeValueChunk, + format: &str, +) -> LegacyResult { + let normalized = format.trim().to_ascii_lowercase(); + let (value, encoding) = match normalized.as_str() { + "" | "auto" => ( + chunk.value.clone(), + match chunk.encoding { + LargeValueEncoding::Utf8 => "utf-8", + LargeValueEncoding::Base64 => "base64", + }, + ), + "base64" => ( + BASE64_STANDARD.encode(decode_large_value_chunk(&chunk)?), + "base64", + ), + "text" => ( + String::from_utf8_lossy(&decode_large_value_chunk(&chunk)?).into_owned(), + "utf-8", + ), + "hex" => (encode_hex(&decode_large_value_chunk(&chunk)?), "hex"), + _ => { + return Err(LegacyFailure::invalid( + "invalid_large_cell_value_request", + "format must be text, hex, base64, or auto", + )); + } + }; + Ok(LegacyLargeCellChunk { + value, + offset: chunk.offset, + next_offset: chunk.next_offset, + eof: chunk.eof, + size_bytes: chunk.size_bytes, + size_chars: chunk.size_chars, + encoding: encoding.to_owned(), + content_type: chunk.content_type, + display_mode: chunk.display_mode, + }) +} + +fn decode_large_value_chunk(chunk: &LargeValueChunk) -> LegacyResult> { + match chunk.encoding { + LargeValueEncoding::Utf8 => Ok(chunk.value.as_bytes().to_vec()), + LargeValueEncoding::Base64 => { + BASE64_STANDARD + .decode(chunk.value.as_bytes()) + .map_err(|_| LegacyFailure { + code: "large_cell_value_invalid".to_owned(), + message: "The retained binary value could not be decoded".to_owned(), + }) + } + } +} + +fn encode_hex(bytes: &[u8]) -> String { + const HEX: &[u8; 16] = b"0123456789ABCDEF"; + let mut output = String::with_capacity(bytes.len().saturating_mul(2)); + for byte in bytes { + output.push(char::from(HEX[usize::from(byte >> 4)])); + output.push(char::from(HEX[usize::from(byte & 0x0f)])); + } + output +} + +fn unique_large_cell_download_path( + directory: &std::path::Path, + large_value_id: &str, + extension: &str, +) -> PathBuf { + let token_fragment = large_value_id + .chars() + .filter(char::is_ascii_alphanumeric) + .take(12) + .collect::(); + directory.join(format!( + "chat2db-cell-{}-{token_fragment}.{extension}", + unix_epoch_millis() + )) +} + +/// Reports whether the request resolves to the native `MySQL` driver. +/// +/// # Errors +/// +/// Returns a datasource lookup failure when the requested datasource is absent. +pub async fn uses_native_mysql_console( + application: &Application, + request: &LegacySqlExecuteRequest, +) -> LegacyResult { + if request.database_type.eq_ignore_ascii_case("mysql") { + return Ok(true); + } + let datasource = application + .get_datasource(&request.data_source_id.as_string()) + .await?; + Ok(datasource.driver_id.eq_ignore_ascii_case("mysql") + || datasource.driver_id.to_ascii_lowercase().contains("mysql")) +} + +fn mysql_console_result( + application: &Application, + large_value_owner: &str, + request: &LegacySqlExecuteRequest, + result: MysqlConsoleResult, +) -> LegacyManageResult { + let MysqlConsoleResult { + statement_sequence, + result_set_id, + sql, + success, + message, + update_count, + columns, + rows, + row_count, + has_more, + duration_ms, + error, + } = result; + let header_list = columns.iter().map(result_header).collect(); + let data_list = rows + .into_iter() + .map(|row| { + row.values + .into_iter() + .zip(columns.iter()) + .map(|(value, column)| result_cell(application, large_value_owner, value, column)) + .collect() + }) + .collect(); + let sql_type = legacy_sql_type(&sql).to_owned(); + let extra = error.map_or_else( + || serde_json::json!({}), + |error| { + serde_json::json!({ + "messages": [{ + "level": "ERROR", + "message": error.message, + "source": "database", + "errorCode": error.code, + "resultSetId": result_set_id, + }] + }) + }, + ); + let finished_at_epoch_ms = unix_epoch_millis(); + LegacyManageResult { + data_list, + header_list, + description: if success { + "Query executed successfully".to_owned() + } else { + String::new() + }, + message, + sql: sql.clone(), + original_sql: sql, + success, + duration: duration_ms, + update_count, + can_edit: false, + table_name: request.table_name.clone(), + refresh_targets: refresh_targets(request, &sql_type, success), + sql_type, + page_no: request.page_no, + page_size: request.page_size, + fuzzy_total: row_count.to_string(), + has_next_page: has_more, + execute_sql_params: request.clone(), + extra, + comment: None, + result_set_id, + statement_sequence: Some(statement_sequence), + execution_metrics: Some(LegacyExecutionMetrics { + started_at_epoch_ms: finished_at_epoch_ms.saturating_sub(duration_ms), + finished_at_epoch_ms, + total_duration_ms: duration_ms, + execute_duration_ms: duration_ms, + fetch_duration_ms: 0, + fetched_row_count: row_count, + }), + execution_context: Some(LegacyExecutionContext { + database_name: non_blank(&request.database_name), + schema_name: non_blank(&request.schema_name), + }), + } +} + +fn refresh_targets( + request: &LegacySqlExecuteRequest, + sql_type: &str, + success: bool, +) -> Vec { + if !success + || !matches!( + sql_type, + "INSERT" | "UPDATE" | "DELETE" | "CREATE" | "ALTER" | "DROP" | "TRUNCATE" + ) + { + return Vec::new(); + } + vec![serde_json::json!({ + "dataSourceId": request.data_source_id.as_string(), + "databaseName": request.database_name, + "schemaName": request.schema_name, + "tableName": request.table_name, + })] +} + fn sql_page_window(request: &LegacySqlExecuteRequest) -> LegacyResult<(u32, u32)> { if request.page_no == 0 || request.page_size == 0 { return Err(LegacyFailure::invalid( @@ -1774,6 +2640,24 @@ fn sql_page_window(request: &LegacySqlExecuteRequest) -> LegacyResult<(u32, u32) Ok((offset, row_limit)) } +fn validate_sql_execute_request(request: &LegacySqlExecuteRequest) -> LegacyResult<(String, u32)> { + let (_, row_limit) = sql_page_window(request)?; + let datasource_id = request.data_source_id.as_string(); + if datasource_id.trim().is_empty() { + return Err(LegacyFailure::invalid( + "invalid_sql_execute_request", + "dataSourceId is required", + )); + } + if request.sql.trim().is_empty() { + return Err(LegacyFailure::invalid( + "invalid_sql_execute_request", + "sql is required", + )); + } + Ok((datasource_id, row_limit)) +} + /// Builds the Community result-grid error item used for accepted operations /// that fail asynchronously. Keeping this public prevents desktop IPC from /// inventing a second error projection. @@ -1811,6 +2695,15 @@ pub fn sql_failure_result( "errorCode": error.code, }] }), + comment: None, + result_set_id: request.result_set_id, + statement_sequence: Some(1), + execution_metrics: None, + execution_context: Some(LegacyExecutionContext { + database_name: (!request.database_name.is_empty()) + .then(|| request.database_name.clone()), + schema_name: (!request.schema_name.is_empty()).then(|| request.schema_name.clone()), + }), } } @@ -1818,25 +2711,102 @@ fn elapsed_millis(started_at: Instant) -> u64 { u64::try_from(started_at.elapsed().as_millis()).unwrap_or(u64::MAX) } +fn unix_epoch_millis() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .ok() + .and_then(|duration| u64::try_from(duration.as_millis()).ok()) + .unwrap_or(0) +} + fn legacy_sql_type(sql: &str) -> &'static str { let keyword = sql .trim_start() .split(|character: char| !character.is_ascii_alphabetic()) .next() .unwrap_or_default(); - if keyword.eq_ignore_ascii_case("select") - || keyword.eq_ignore_ascii_case("with") - || keyword.eq_ignore_ascii_case("show") - || keyword.eq_ignore_ascii_case("describe") - || keyword.eq_ignore_ascii_case("desc") - || keyword.eq_ignore_ascii_case("explain") + if ["select", "with", "show", "describe", "desc", "explain"] + .iter() + .any(|candidate| keyword.eq_ignore_ascii_case(candidate)) { - "SELECT" + return "SELECT"; + } + [ + ("insert", "INSERT"), + ("replace", "REPLACE_INTO"), + ("update", "UPDATE"), + ("delete", "DELETE"), + ("create", "CREATE"), + ("alter", "ALTER"), + ("drop", "DROP"), + ("truncate", "TRUNCATE"), + ("call", "CALL"), + ("use", "USE_DATABASE"), + ("set", "SET"), + ("start", "START_TRANSACTION"), + ("begin", "BEGIN_WORK"), + ("commit", "COMMIT"), + ("rollback", "ROLLBACK"), + ] + .into_iter() + .find_map(|(candidate, sql_type)| keyword.eq_ignore_ascii_case(candidate).then_some(sql_type)) + .unwrap_or("OTHER") +} + +fn operation_log_response(record: OperationLogRecord, preview: bool) -> LegacyOperationLogResponse { + let (ddl, more) = if preview { + operation_log_preview(&record.ddl, record.more) } else { - "OTHER" + (record.ddl, record.more) + }; + LegacyOperationLogResponse { + id: record.id, + gmt_create: format_operation_log_time(record.created_at_ms), + gmt_modified: format_operation_log_time(record.updated_at_ms), + name: record.name.unwrap_or_default(), + data_source_id: record.data_source_id, + data_source_name: record.data_source_name, + connectable: record.connectable, + database_name: record.database_name, + database_type: record.database_type, + ddl, + more, + status: Some(record.status), + operation_rows: record + .operation_rows + .and_then(|value| value.try_into().ok()), + use_time: record.use_time.and_then(|value| value.try_into().ok()), + extend_info: record.extend_info, + schema_name: record.schema_name, + organization_id: record.organization_id, + user_name: record.user_name, } } +fn operation_log_preview(ddl: &str, persisted_more: bool) -> (String, bool) { + const PREVIEW_CHARS: usize = 200; + let mut characters = ddl.chars(); + let mut preview = characters.by_ref().take(PREVIEW_CHARS).collect::(); + let truncated = characters.next().is_some(); + if truncated { + preview.push_str("..."); + } + (preview, persisted_more || truncated) +} + +fn format_operation_log_time(timestamp_ms: i64) -> String { + Local + .timestamp_millis_opt(timestamp_ms) + .single() + .map(|timestamp| timestamp.format("%Y-%m-%d %H:%M:%S").to_string()) + .unwrap_or_default() +} + +fn non_blank(value: &str) -> Option { + let value = value.trim(); + (!value.is_empty()).then(|| value.to_owned()) +} + fn saved_console_response(record: SavedConsoleRecord) -> LegacySavedConsoleResponse { let connectable = record .data_source_id @@ -1938,6 +2908,10 @@ fn storage_failure(error: StorageError) -> LegacyFailure { code: "invalid_saved_console".to_owned(), message: message.to_owned(), }, + StorageError::InvalidOperationLog(message) => LegacyFailure { + code: "invalid_operation_log".to_owned(), + message: message.to_owned(), + }, other => LegacyFailure::from(AppError::from(other)), } } @@ -2134,40 +3108,178 @@ fn result_header(column: &ResultColumn) -> LegacyResultHeader { column_name: column.name.clone(), column_type: column.jdbc_type_name.clone(), table_name: column.table_name.clone(), + database_name: column.catalog_name.clone(), + schema_name: column.schema_name.clone(), primary_key: false, + comment: None, + default_value: None, + auto_increment: None, nullable: column.nullability != ColumnNullability::NoNulls, column_size: column.precision.or(column.display_size), decimal_digits: column.scale, + editor_type: None, + } +} + +fn result_cell( + application: &Application, + large_value_owner: &str, + value: JdbcValue, + column: &ResultColumn, +) -> LegacyResultCell { + match value { + JdbcValue::Text { value } => { + retained_text_cell(application, large_value_owner, value, "TEXT", column) + } + JdbcValue::Json { value } => { + retained_text_cell(application, large_value_owner, value, "JSON", column) + } + JdbcValue::Binary { value } => match BASE64_STANDARD.decode(value.as_bytes()) { + Ok(value) => retained_binary_cell(application, large_value_owner, value, column), + Err(_) => unsupported_cell( + Some(value), + "BINARY", + column, + None, + None, + "The binary cell payload could not be decoded".to_owned(), + ), + }, + value => scalar_result_cell(value, column), } } -fn result_cell(value: JdbcValue, column: &ResultColumn) -> LegacyResultCell { - let (value, value_type) = match value { - JdbcValue::Null => (None, "UNKNOWN"), - JdbcValue::Boolean { value } => (Some(value.to_string()), "UNKNOWN"), +fn retained_text_cell( + application: &Application, + owner_id: &str, + value: String, + value_type: &str, + column: &ResultColumn, +) -> LegacyResultCell { + let size_bytes = value.len() as u64; + let size_chars = value.chars().count() as u64; + let fallback = bounded_utf8_preview(&value, LARGE_VALUE_FALLBACK_PREVIEW_BYTES); + match application.retain_large_text(owner_id, value) { + Ok(preview) => retained_preview_cell(preview, value_type, column), + Err(error) => unsupported_cell( + Some(fallback), + value_type, + column, + Some(size_bytes), + Some(size_chars), + error.api_error().message, + ), + } +} + +fn retained_binary_cell( + application: &Application, + owner_id: &str, + value: Vec, + column: &ResultColumn, +) -> LegacyResultCell { + let size_bytes = value.len() as u64; + let fallback = + BASE64_STANDARD.encode(&value[..value.len().min(LARGE_VALUE_FALLBACK_PREVIEW_BYTES)]); + match application.retain_large_binary(owner_id, value) { + Ok(preview) => retained_preview_cell(preview, "BINARY", column), + Err(error) => unsupported_cell( + Some(fallback), + "BINARY", + column, + Some(size_bytes), + None, + error.api_error().message, + ), + } +} + +fn retained_preview_cell( + preview: LargeValuePreview, + value_type: &str, + column: &ResultColumn, +) -> LegacyResultCell { + LegacyResultCell { + value: Some(preview.value), + large_value: preview.large_value, + large_value_id: preview.large_value_id, + value_type: value_type.to_owned(), + sql_type: column.jdbc_type, + column_type: column.jdbc_type_name.clone(), + size_bytes: Some(preview.size_bytes), + size_chars: preview.size_chars, + loaded_bytes: Some(preview.loaded_bytes), + loaded_chars: preview.loaded_chars, + truncated: preview.truncated, + unsupported_reason: None, + } +} + +fn unsupported_cell( + value: Option, + value_type: &str, + column: &ResultColumn, + size_bytes: Option, + size_chars: Option, + reason: String, +) -> LegacyResultCell { + LegacyResultCell { + loaded_bytes: value.as_ref().map(|value| value.len() as u64), + loaded_chars: value.as_ref().map(|value| value.chars().count() as u64), + value, + large_value: true, + large_value_id: None, + value_type: value_type.to_owned(), + sql_type: column.jdbc_type, + column_type: column.jdbc_type_name.clone(), + size_bytes, + size_chars, + truncated: true, + unsupported_reason: Some(reason), + } +} + +fn scalar_result_cell(value: JdbcValue, column: &ResultColumn) -> LegacyResultCell { + let value = match value { + JdbcValue::Null => None, + JdbcValue::Boolean { value } => Some(value.to_string()), JdbcValue::SignedInteger { value } | JdbcValue::UnsignedInteger { value } | JdbcValue::Float32 { value } | JdbcValue::Float64 { value } | JdbcValue::Decimal { value } - | JdbcValue::Text { value } | JdbcValue::Date { value } | JdbcValue::Time { value } | JdbcValue::Timestamp { value } | JdbcValue::TimestampWithTimeZone { value } - | JdbcValue::Uuid { value } => (Some(value), "UNKNOWN"), - JdbcValue::Binary { value } => (Some(value), "BINARY"), - JdbcValue::Json { value } => (Some(value), "JSON"), - JdbcValue::Opaque { display_value, .. } => (Some(display_value), "UNKNOWN"), + | JdbcValue::Uuid { value } => Some(value), + JdbcValue::Opaque { display_value, .. } => Some(display_value), + JdbcValue::Text { .. } | JdbcValue::Binary { .. } | JdbcValue::Json { .. } => { + unreachable!("large values are handled before scalar conversion") + } }; LegacyResultCell { value, large_value: false, - value_type: value_type.to_owned(), + large_value_id: None, + value_type: "UNKNOWN".to_owned(), sql_type: column.jdbc_type, column_type: column.jdbc_type_name.clone(), + size_bytes: None, + size_chars: None, + loaded_bytes: None, + loaded_chars: None, truncated: false, + unsupported_reason: None, + } +} + +fn bounded_utf8_preview(value: &str, max_bytes: usize) -> String { + let mut end = value.len().min(max_bytes); + while end > 0 && !value.is_char_boundary(end) { + end -= 1; } + value[..end].to_owned() } const fn legacy_data_type(value_type: JdbcValueType) -> &'static str { @@ -2504,6 +3616,25 @@ pub async fn dispatch( }, Err(error) => Err(error), }, + ("post", "/api/operation/log/create") => { + match decode::(request.message) { + Ok(body) => serialized(create_operation_log(application, &body).await), + Err(error) => Err(error), + } + } + ("get", "/api/operation/log/list") => { + match decode::(request.message) { + Ok(query) => serialized(list_operation_logs(application, &query).await), + Err(error) => Err(error), + } + } + ("get", "/api/operation/log") => match decode::(request.message) { + Ok(query) => match legacy_console_id(&query.id) { + Ok(id) => serialized(get_operation_log(application, id).await), + Err(error) => Err(error), + }, + Err(error) => Err(error), + }, ("get", "/api/namespaces/tree_list") => serialized(namespace_tree(application).await), ("get", "/api/rdb/database/list") => match decode::(request.message) { Ok(query) => serialized(list_databases(application, &query).await), @@ -2598,6 +3729,24 @@ pub async fn dispatch( Err(error) => Err(error), } } + ("post" | "put", "/api/rdb/dml/execute_ddl") => { + match decode::(request.message) { + Ok(body) => serialized(execute_ddl(application, &body).await), + Err(error) => Err(error), + } + } + ("post", "/api/rdb/cell/value") => { + match decode::(request.message) { + Ok(body) => serialized(read_large_cell_value(application, &body)), + Err(error) => Err(error), + } + } + ("post", "/api/rdb/cell/download_path") => { + match decode::(request.message) { + Ok(body) => serialized(download_large_cell_value_to_path(application, &body).await), + Err(error) => Err(error), + } + } (_, known_path) if LEGACY_PATHS.contains(&known_path) => Err(LegacyFailure::invalid( "method_not_allowed", "The request method is not supported for this route", @@ -2627,6 +3776,9 @@ const LEGACY_PATHS: &[&str] = &[ "/api/operation/saved/list", "/api/operation/saved", "/api/operation/saved/update", + "/api/operation/log/create", + "/api/operation/log/list", + "/api/operation/log", "/api/namespaces/tree_list", "/api/rdb/database/list", "/api/rdb/schema/list", @@ -2648,7 +3800,11 @@ const LEGACY_PATHS: &[&str] = &[ "/api/rdb/trigger/list", "/api/rdb/trigger/detail", "/api/rdb/dml/execute", + "/api/rdb/dml/execute_ddl", "/api/rdb/dml/execute_table", + "/api/rdb/cell/value", + "/api/rdb/cell/download", + "/api/rdb/cell/download_path", ]; const LEGACY_COUNTED_PATHS: &[&str] = &[ @@ -2754,6 +3910,12 @@ pub(crate) fn routes() -> Router { "/api/operation/saved/update", post(update_saved_console_handler).put(update_saved_console_handler), ) + .route( + "/api/operation/log/create", + post(create_operation_log_handler), + ) + .route("/api/operation/log/list", get(list_operation_logs_handler)) + .route("/api/operation/log", get(get_operation_log_handler)) .route("/api/namespaces/tree_list", get(namespace_tree_handler)) .route("/api/rdb/database/list", get(database_list_handler)) .route("/api/rdb/schema/list", get(schema_list_handler)) @@ -2778,10 +3940,20 @@ pub(crate) fn routes() -> Router { "/api/rdb/dml/execute", post(sql_execute_handler).put(sql_execute_handler), ) + .route( + "/api/rdb/dml/execute_ddl", + post(sql_execute_ddl_handler).put(sql_execute_ddl_handler), + ) .route( "/api/rdb/dml/execute_table", post(table_preview_handler).put(table_preview_handler), ) + .route("/api/rdb/cell/value", post(large_cell_value_handler)) + .route("/api/rdb/cell/download", post(large_cell_download_handler)) + .route( + "/api/rdb/cell/download_path", + post(large_cell_download_path_handler), + ) .layer(middleware::map_response(legacy_bad_request_envelope)) } @@ -2913,6 +4085,30 @@ async fn delete_saved_console_handler( }) } +async fn create_operation_log_handler( + State(application): State, + Json(request): Json, +) -> Json> { + envelope(create_operation_log(&application, &request).await) +} + +async fn list_operation_logs_handler( + State(application): State, + Query(query): Query, +) -> Json>> { + envelope(list_operation_logs(&application, &query).await) +} + +async fn get_operation_log_handler( + State(application): State, + Query(query): Query, +) -> Json> { + envelope(match legacy_console_id(&query.id) { + Ok(id) => get_operation_log(&application, id).await, + Err(error) => Err(error), + }) +} + async fn namespace_tree_handler( State(application): State, ) -> Json>> { @@ -3066,6 +4262,56 @@ async fn sql_execute_handler( envelope(execute_sql(&application, &request).await) } +async fn sql_execute_ddl_handler( + State(application): State, + Json(request): Json, +) -> Json> { + envelope(execute_ddl(&application, &request).await) +} + +async fn large_cell_value_handler( + State(application): State, + Json(request): Json, +) -> Json> { + envelope(read_large_cell_value(&application, &request)) +} + +async fn large_cell_download_path_handler( + State(application): State, + Json(request): Json, +) -> Json> { + envelope(download_large_cell_value_to_path(&application, &request).await) +} + +async fn large_cell_download_handler( + State(application): State, + Json(request): Json, +) -> Response { + let task = + tokio::task::spawn_blocking(move || prepare_large_cell_download(&application, &request)) + .await; + let download = match task { + Ok(Ok(download)) => download, + Ok(Err(error)) => return Json(LegacyEnvelope::<()>::failure(error)).into_response(), + Err(_) => { + return Json(LegacyEnvelope::<()>::failure(LegacyFailure { + code: "large_cell_download_failed".to_owned(), + message: "The large cell download task did not finish".to_owned(), + })) + .into_response(); + } + }; + Response::builder() + .status(StatusCode::OK) + .header(header::CONTENT_TYPE, download.content_type) + .header( + header::CONTENT_DISPOSITION, + format!("attachment; filename=chat2db-cell.{}", download.extension), + ) + .body(Body::from(download.bytes)) + .unwrap_or_else(|_| StatusCode::INTERNAL_SERVER_ERROR.into_response()) +} + #[cfg(test)] mod tests { use axum::{ @@ -3152,6 +4398,107 @@ mod tests { format!("{path}?dataSourceId=1&databaseName=inventory&schemaName={object_name}") } + #[tokio::test] + async fn retained_large_text_round_trips_through_chunk_and_download_contracts() { + let application = Application::new(); + let owner = application.create_large_value_owner(); + let value = "数据🙂".repeat(24_000); + let preview = application + .retain_large_text(&owner, value.clone()) + .expect("large text must be retained"); + assert!(preview.truncated); + let large_value_id = preview + .large_value_id + .expect("truncated text must receive a scoped token"); + + let chunk = read_large_cell_value( + &application, + &LegacyLargeCellValueRequest { + large_value_id: large_value_id.clone(), + offset: 0, + limit: 128, + format: "base64".to_owned(), + }, + ) + .expect("large text chunk must load"); + let decoded = BASE64_STANDARD + .decode(chunk.value) + .expect("chunk must use frontend-compatible base64"); + assert_eq!(decoded, value.as_bytes()[..128]); + assert_eq!(chunk.offset, 0); + assert_eq!(chunk.next_offset, 128); + assert_eq!(chunk.encoding, "base64"); + assert_eq!(chunk.display_mode, LargeValueType::Text); + + let next = read_large_cell_value( + &application, + &LegacyLargeCellValueRequest { + large_value_id: large_value_id.clone(), + offset: chunk.next_offset, + limit: 128, + format: "base64".to_owned(), + }, + ) + .expect("second large text chunk must load"); + let next_decoded = BASE64_STANDARD + .decode(next.value) + .expect("second chunk must use frontend-compatible base64"); + assert_eq!(next_decoded, value.as_bytes()[128..256]); + assert_eq!(next.offset, 128); + assert_eq!(next.next_offset, 256); + + let path = download_large_cell_value_to_path( + &application, + &LegacyLargeCellDownloadRequest { + large_value_id, + format: "text".to_owned(), + }, + ) + .await + .expect("large text must download to a local temporary file"); + assert_eq!( + std::fs::read_to_string(&path).expect("download must be readable"), + value + ); + std::fs::remove_file(path).expect("temporary test download must be removed"); + } + + #[test] + fn mysql_console_history_distinguishes_cancellation_from_failure() { + let request = serde_json::from_value(serde_json::json!({ + "dataSourceId": "datasource-1", + "databaseType": "MYSQL", + "sql": "SELECT SLEEP(30)", + "pageNo": 1, + "pageSize": 200 + })) + .expect("legacy SQL request must deserialize"); + let cancelled = sql_failure_result( + &request, + &LegacyFailure { + code: "mysql_console_cancelled".to_owned(), + message: "The SQL execution was cancelled".to_owned(), + }, + 12, + ); + assert_eq!(mysql_console_history_status(&[&cancelled]), "cancelled"); + + let failed = sql_failure_result( + &request, + &LegacyFailure { + code: "database.query_failed".to_owned(), + message: "Query failed".to_owned(), + }, + 12, + ); + assert_eq!(mysql_console_history_status(&[&failed]), "fail"); + + let mut succeeded = failed; + succeeded.success = true; + succeeded.extra = serde_json::json!({}); + assert_eq!(mysql_console_history_status(&[&succeeded]), "success"); + } + #[test] fn counted_metadata_envelope_keeps_total_at_the_top_level() { let body = counted_envelope_value(Ok(serde_json::json!([ diff --git a/apps/chat2db-web/src/lib.rs b/apps/chat2db-web/src/lib.rs index 5bc7faa..0ab9886 100644 --- a/apps/chat2db-web/src/lib.rs +++ b/apps/chat2db-web/src/lib.rs @@ -531,7 +531,7 @@ mod tests { } #[tokio::test] - async fn legacy_sql_execute_rejects_empty_sql_and_preserves_core_start_errors() { + async fn legacy_sql_execute_rejects_empty_sql_and_reports_missing_native_datasource() { let empty_response = router(Application::new()) .oneshot(json_request( Method::POST, @@ -553,7 +553,7 @@ mod tests { let directory = TempDir::new().expect("temp directory"); let storage = Storage::open(directory.path(), Arc::new(TestVault)).expect("test storage must open"); - let unavailable_response = router(Application::with_storage(storage)) + let unavailable_response = router(Application::with_storage(storage.clone())) .oneshot(json_request( Method::POST, "/api/rdb/dml/execute", @@ -568,8 +568,19 @@ mod tests { .await .expect("router must respond"); let unavailable: serde_json::Value = response_json(unavailable_response).await; - assert_eq!(unavailable["success"], false); - assert_eq!(unavailable["errorCode"], "database_engine_unavailable"); + assert_eq!(unavailable["success"], true); + assert_eq!(unavailable["data"][0]["success"], false); + assert_eq!( + unavailable["data"][0]["extra"]["messages"][0]["errorCode"], + "datasource_not_found" + ); + + let history = storage + .list_operation_logs(&chat2db_storage::OperationLogListQuery::default()) + .expect("failed native executions must be recorded best-effort"); + assert_eq!(history.total, 1); + assert_eq!(history.records[0].ddl, "SELECT 1"); + assert_eq!(history.records[0].status, "fail"); } #[test] diff --git a/crates/chat2db-core/src/error.rs b/crates/chat2db-core/src/error.rs index 87799c5..9c46be7 100644 --- a/crates/chat2db-core/src/error.rs +++ b/crates/chat2db-core/src/error.rs @@ -276,6 +276,9 @@ impl From for AppError { StorageError::InvalidSavedConsole(message) => { Self::invalid("invalid_saved_console", message) } + StorageError::InvalidOperationLog(message) => { + Self::invalid("invalid_operation_log", message) + } StorageError::ProviderNotFound(id) => Self::not_found( "provider_not_found", format!("Provider profile {id} does not exist"), diff --git a/crates/chat2db-core/src/large_value.rs b/crates/chat2db-core/src/large_value.rs new file mode 100644 index 0000000..113a636 --- /dev/null +++ b/crates/chat2db-core/src/large_value.rs @@ -0,0 +1,695 @@ +use std::{ + collections::{HashMap, VecDeque}, + sync::{Mutex, MutexGuard}, + time::{Duration, Instant}, +}; + +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; +use serde::{Deserialize, Serialize}; +use thiserror::Error; +use uuid::Uuid; + +const DEFAULT_PREVIEW_BYTES: usize = 64 * 1024; +const DEFAULT_MAX_CHUNK_SIZE: u32 = 256 * 1024; +const DEFAULT_MAX_VALUE_BYTES: u64 = 32 * 1024 * 1024; +const DEFAULT_MAX_TOTAL_BYTES: u64 = 128 * 1024 * 1024; +const DEFAULT_MAX_ENTRIES: usize = 512; +const DEFAULT_TTL: Duration = Duration::from_secs(10 * 60); +const SCOPED_TOKEN_SEPARATOR: char = '.'; + +pub(crate) fn new_owner_id() -> String { + Uuid::new_v4().simple().to_string() +} + +pub(crate) fn scope_preview(owner_id: &str, mut preview: LargeValuePreview) -> LargeValuePreview { + if let Some(token) = preview.large_value_id.take() { + preview.large_value_id = Some(format!("{owner_id}{SCOPED_TOKEN_SEPARATOR}{token}")); + } + preview +} + +pub(crate) fn scoped_token(value: &str) -> Result<(&str, &str), LargeValueError> { + let (owner_id, token) = value + .split_once(SCOPED_TOKEN_SEPARATOR) + .ok_or(LargeValueError::InvalidToken)?; + Uuid::parse_str(owner_id).map_err(|_| LargeValueError::InvalidToken)?; + parse_token(token)?; + Ok((owner_id, token)) +} + +/// Resource limits for retained Console large values. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LargeValueStoreConfig { + /// Maximum raw bytes included in an inline preview. + pub preview_bytes: usize, + /// Maximum character count for text chunks and byte count for binary chunks. + pub max_chunk_size: u32, + /// Maximum raw size accepted for one retained value. + pub max_value_bytes: u64, + /// Maximum raw size retained across all values. + pub max_total_bytes: u64, + /// Maximum number of retained values. + pub max_entries: usize, + /// Fixed lifetime of a token. Reads do not extend it. + pub ttl: Duration, +} + +impl Default for LargeValueStoreConfig { + fn default() -> Self { + Self { + preview_bytes: DEFAULT_PREVIEW_BYTES, + max_chunk_size: DEFAULT_MAX_CHUNK_SIZE, + max_value_bytes: DEFAULT_MAX_VALUE_BYTES, + max_total_bytes: DEFAULT_MAX_TOTAL_BYTES, + max_entries: DEFAULT_MAX_ENTRIES, + ttl: DEFAULT_TTL, + } + } +} + +/// The portable type of a retained large value. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "SCREAMING_SNAKE_CASE")] +pub enum LargeValueType { + Text, + Binary, +} + +/// Encoding used by a preview or chunk payload. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum LargeValueEncoding { + #[serde(rename = "utf-8")] + Utf8, + #[serde(rename = "base64")] + Base64, +} + +/// Bounded cell value returned with a Console result row. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LargeValuePreview { + pub value: String, + pub large_value: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub large_value_id: Option, + pub value_type: LargeValueType, + pub size_bytes: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub size_chars: Option, + pub loaded_bytes: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub loaded_chars: Option, + pub truncated: bool, + pub encoding: LargeValueEncoding, +} + +/// One bounded read from a retained value. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct LargeValueChunk { + pub value: String, + /// Character offset for text and byte offset for binary. + pub offset: u64, + /// Character offset for text and byte offset for binary. + pub next_offset: u64, + pub eof: bool, + pub size_bytes: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub size_chars: Option, + pub encoding: LargeValueEncoding, + pub content_type: String, + pub display_mode: LargeValueType, +} + +/// Current retained-value resource usage. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct LargeValueStoreStats { + pub entries: usize, + pub total_bytes: u64, +} + +/// A closed error contract for token and chunk operations. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)] +pub enum LargeValueError { + #[error("large value owner id is required")] + InvalidOwner, + #[error("large value token is malformed")] + InvalidToken, + #[error("large value token was not found")] + NotFound, + #[error("large value token has expired")] + Expired, + #[error("large value token belongs to another owner")] + OwnerMismatch, + #[error("large value chunk limit must be greater than zero")] + InvalidLimit, + #[error("large value offset {offset} exceeds length {length}")] + InvalidRange { offset: u64, length: u64 }, + #[error( + "large value capacity exceeded: requested {requested_bytes} bytes, value limit {max_value_bytes}, total limit {max_total_bytes}, entry limit {max_entries}" + )] + CapacityExceeded { + requested_bytes: u64, + max_value_bytes: u64, + max_total_bytes: u64, + max_entries: usize, + }, +} + +/// In-memory large-value retention for native Console results. +/// +/// Tokens are random UUID v4 values, have a fixed TTL, and are scoped to an +/// execution owner. When capacity is needed, the oldest retained value is +/// evicted first. +#[derive(Debug)] +pub struct LargeValueStore { + config: LargeValueStoreConfig, + state: Mutex, +} + +impl LargeValueStore { + /// Creates an empty store with explicit retention limits. + #[must_use] + pub fn new(config: LargeValueStoreConfig) -> Self { + Self { + config, + state: Mutex::new(StoreState::default()), + } + } + + /// Builds a bounded UTF-8 preview and retains the full value when needed. + /// + /// # Errors + /// + /// Returns [`LargeValueError::InvalidOwner`] for an empty owner, or + /// [`LargeValueError::CapacityExceeded`] when the full value cannot fit. + pub fn retain_text( + &self, + owner_id: &str, + value: String, + ) -> Result { + self.retain(owner_id, StoredValue::Text(value)) + } + + /// Builds a bounded base64 preview and retains the full value when needed. + /// + /// # Errors + /// + /// Returns [`LargeValueError::InvalidOwner`] for an empty owner, or + /// [`LargeValueError::CapacityExceeded`] when the full value cannot fit. + pub fn retain_binary( + &self, + owner_id: &str, + value: Vec, + ) -> Result { + self.retain(owner_id, StoredValue::Binary(value)) + } + + /// Reads a chunk after validating token syntax, expiry, and ownership. + /// + /// # Errors + /// + /// Returns a closed [`LargeValueError`] variant for invalid owners, tokens, + /// ranges, limits, expiry, missing entries, or owner mismatches. + pub fn read_chunk( + &self, + owner_id: &str, + token: &str, + offset: u64, + limit: u32, + ) -> Result { + self.read_chunk_inner(owner_id, token, offset, limit, false) + } + + /// Reads a base64 chunk whose offsets and limit are measured in raw bytes. + /// + /// This matches the Community large-cell transfer contract for encoded + /// text and binary values. + /// + /// # Errors + /// + /// Returns a closed [`LargeValueError`] variant for invalid owners, tokens, + /// ranges, limits, expiry, missing entries, or owner mismatches. + pub fn read_encoded_chunk( + &self, + owner_id: &str, + token: &str, + offset: u64, + limit: u32, + ) -> Result { + self.read_chunk_inner(owner_id, token, offset, limit, true) + } + + fn read_chunk_inner( + &self, + owner_id: &str, + token: &str, + offset: u64, + limit: u32, + encoded: bool, + ) -> Result { + validate_owner(owner_id)?; + if limit == 0 || self.config.max_chunk_size == 0 { + return Err(LargeValueError::InvalidLimit); + } + let token = parse_token(token)?; + let now = Instant::now(); + let mut state = self.lock_state(); + if state + .entries + .get(&token) + .is_some_and(|entry| now >= entry.expires_at) + { + state.remove(&token); + return Err(LargeValueError::Expired); + } + state.cleanup_expired(now); + let entry = state.entries.get(&token).ok_or(LargeValueError::NotFound)?; + if entry.owner_id != owner_id { + return Err(LargeValueError::OwnerMismatch); + } + let limit = limit.min(self.config.max_chunk_size); + if encoded { + entry + .value + .encoded_chunk(offset, limit, entry.size_bytes, entry.size_chars) + } else { + entry + .value + .chunk(offset, limit, entry.size_bytes, entry.size_chars) + } + } + + /// Removes one retained value after validating its owner. + /// + /// # Errors + /// + /// Returns a closed [`LargeValueError`] variant for invalid owners, tokens, + /// expiry, missing entries, or owner mismatches. + pub fn remove_token(&self, owner_id: &str, token: &str) -> Result<(), LargeValueError> { + validate_owner(owner_id)?; + let token = parse_token(token)?; + let now = Instant::now(); + let mut state = self.lock_state(); + let entry = state.entries.get(&token).ok_or(LargeValueError::NotFound)?; + if now >= entry.expires_at { + state.remove(&token); + return Err(LargeValueError::Expired); + } + if entry.owner_id != owner_id { + return Err(LargeValueError::OwnerMismatch); + } + state.remove(&token); + Ok(()) + } + + /// Removes every retained value for an execution or result owner. + #[must_use] + pub fn remove_owner(&self, owner_id: &str) -> usize { + let mut state = self.lock_state(); + state.cleanup_expired(Instant::now()); + let tokens = state + .entries + .iter() + .filter_map(|(token, entry)| (entry.owner_id == owner_id).then_some(*token)) + .collect::>(); + let removed = tokens.len(); + for token in tokens { + state.remove(&token); + } + removed + } + + /// Deletes expired entries and returns the number released. + #[must_use] + pub fn cleanup_expired(&self) -> usize { + self.lock_state().cleanup_expired(Instant::now()) + } + + /// Returns usage after removing expired entries. + #[must_use] + pub fn stats(&self) -> LargeValueStoreStats { + let mut state = self.lock_state(); + state.cleanup_expired(Instant::now()); + LargeValueStoreStats { + entries: state.entries.len(), + total_bytes: state.total_bytes, + } + } + + fn retain( + &self, + owner_id: &str, + value: StoredValue, + ) -> Result { + validate_owner(owner_id)?; + let preview = value.preview(self.config.preview_bytes); + if !preview.truncated { + return Ok(preview.into_public(None)); + } + let size_bytes = value.size_bytes(); + if size_bytes > self.config.max_value_bytes + || size_bytes > self.config.max_total_bytes + || self.config.max_entries == 0 + { + return Err(self.capacity_error(size_bytes)); + } + + let now = Instant::now(); + let mut state = self.lock_state(); + state.cleanup_expired(now); + while state.entries.len() >= self.config.max_entries + || state.total_bytes.saturating_add(size_bytes) > self.config.max_total_bytes + { + if !state.evict_oldest() { + break; + } + } + if state.entries.len() >= self.config.max_entries + || state.total_bytes.saturating_add(size_bytes) > self.config.max_total_bytes + { + return Err(self.capacity_error(size_bytes)); + } + + let token = loop { + let candidate = Uuid::new_v4(); + if !state.entries.contains_key(&candidate) { + break candidate; + } + }; + let size_chars = value.size_chars(); + let expires_at = now.checked_add(self.config.ttl).unwrap_or(now); + state.total_bytes = state.total_bytes.saturating_add(size_bytes); + state.order.push_back(token); + state.entries.insert( + token, + StoredEntry { + owner_id: owner_id.to_owned(), + value, + size_bytes, + size_chars, + expires_at, + }, + ); + Ok(preview.into_public(Some(token.simple().to_string()))) + } + + fn capacity_error(&self, requested_bytes: u64) -> LargeValueError { + LargeValueError::CapacityExceeded { + requested_bytes, + max_value_bytes: self.config.max_value_bytes, + max_total_bytes: self.config.max_total_bytes, + max_entries: self.config.max_entries, + } + } + + fn lock_state(&self) -> MutexGuard<'_, StoreState> { + self.state + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + } +} + +impl Default for LargeValueStore { + fn default() -> Self { + Self::new(LargeValueStoreConfig::default()) + } +} + +#[derive(Debug, Default)] +struct StoreState { + entries: HashMap, + order: VecDeque, + total_bytes: u64, +} + +impl StoreState { + fn remove(&mut self, token: &Uuid) -> bool { + let Some(entry) = self.entries.remove(token) else { + return false; + }; + self.total_bytes = self.total_bytes.saturating_sub(entry.size_bytes); + self.order.retain(|queued| queued != token); + true + } + + fn evict_oldest(&mut self) -> bool { + while let Some(token) = self.order.pop_front() { + if let Some(entry) = self.entries.remove(&token) { + self.total_bytes = self.total_bytes.saturating_sub(entry.size_bytes); + return true; + } + } + false + } + + fn cleanup_expired(&mut self, now: Instant) -> usize { + let expired = self + .entries + .iter() + .filter_map(|(token, entry)| (now >= entry.expires_at).then_some(*token)) + .collect::>(); + let removed = expired.len(); + for token in expired { + if let Some(entry) = self.entries.remove(&token) { + self.total_bytes = self.total_bytes.saturating_sub(entry.size_bytes); + } + } + self.order.retain(|token| self.entries.contains_key(token)); + removed + } +} + +#[derive(Debug)] +struct StoredEntry { + owner_id: String, + value: StoredValue, + size_bytes: u64, + size_chars: Option, + expires_at: Instant, +} + +#[derive(Debug)] +enum StoredValue { + Text(String), + Binary(Vec), +} + +impl StoredValue { + fn size_bytes(&self) -> u64 { + match self { + Self::Text(value) => value.len() as u64, + Self::Binary(value) => value.len() as u64, + } + } + + fn size_chars(&self) -> Option { + match self { + Self::Text(value) => Some(value.chars().count() as u64), + Self::Binary(_) => None, + } + } + + fn preview(&self, max_bytes: usize) -> PreviewParts { + match self { + Self::Text(value) => { + let mut end = value.len().min(max_bytes); + while end > 0 && !value.is_char_boundary(end) { + end -= 1; + } + let preview = &value[..end]; + PreviewParts { + value: preview.to_owned(), + value_type: LargeValueType::Text, + size_bytes: value.len() as u64, + size_chars: Some(value.chars().count() as u64), + loaded_bytes: end as u64, + loaded_chars: Some(preview.chars().count() as u64), + truncated: end < value.len(), + encoding: LargeValueEncoding::Utf8, + } + } + Self::Binary(value) => { + let included = value.len().min(max_bytes); + PreviewParts { + value: BASE64_STANDARD.encode(&value[..included]), + value_type: LargeValueType::Binary, + size_bytes: value.len() as u64, + size_chars: None, + loaded_bytes: included as u64, + loaded_chars: None, + truncated: included < value.len(), + encoding: LargeValueEncoding::Base64, + } + } + } + } + + fn chunk( + &self, + offset: u64, + limit: u32, + size_bytes: u64, + size_chars: Option, + ) -> Result { + match self { + Self::Text(value) => text_chunk(value, offset, limit, size_bytes, size_chars), + Self::Binary(value) => binary_chunk(value, offset, limit, size_bytes), + } + } + + fn encoded_chunk( + &self, + offset: u64, + limit: u32, + size_bytes: u64, + size_chars: Option, + ) -> Result { + match self { + Self::Text(value) => byte_chunk( + value.as_bytes(), + offset, + limit, + size_bytes, + size_chars, + LargeValueType::Text, + "text/plain", + ), + Self::Binary(value) => binary_chunk(value, offset, limit, size_bytes), + } + } +} + +struct PreviewParts { + value: String, + value_type: LargeValueType, + size_bytes: u64, + size_chars: Option, + loaded_bytes: u64, + loaded_chars: Option, + truncated: bool, + encoding: LargeValueEncoding, +} + +impl PreviewParts { + fn into_public(self, token: Option) -> LargeValuePreview { + LargeValuePreview { + value: self.value, + large_value: self.truncated, + large_value_id: token, + value_type: self.value_type, + size_bytes: self.size_bytes, + size_chars: self.size_chars, + loaded_bytes: self.loaded_bytes, + loaded_chars: self.loaded_chars, + truncated: self.truncated, + encoding: self.encoding, + } + } +} + +fn text_chunk( + value: &str, + offset: u64, + limit: u32, + size_bytes: u64, + size_chars: Option, +) -> Result { + let length = size_chars.unwrap_or_else(|| value.chars().count() as u64); + if offset > length { + return Err(LargeValueError::InvalidRange { offset, length }); + } + + let offset_index = + usize::try_from(offset).map_err(|_| LargeValueError::InvalidRange { offset, length })?; + let start = if offset == length { + value.len() + } else { + value + .char_indices() + .nth(offset_index) + .map(|(index, _)| index) + .ok_or(LargeValueError::InvalidRange { offset, length })? + }; + + let tail = &value[start..]; + let mut included = 0_u64; + let mut end = tail.len(); + for (index, _) in tail.char_indices() { + if included == u64::from(limit) { + end = index; + break; + } + included += 1; + } + let next_offset = offset + included; + Ok(LargeValueChunk { + value: tail[..end].to_owned(), + offset, + next_offset, + eof: next_offset == length, + size_bytes, + size_chars: Some(length), + encoding: LargeValueEncoding::Utf8, + content_type: "text/plain".to_owned(), + display_mode: LargeValueType::Text, + }) +} + +fn binary_chunk( + value: &[u8], + offset: u64, + limit: u32, + size_bytes: u64, +) -> Result { + byte_chunk( + value, + offset, + limit, + size_bytes, + None, + LargeValueType::Binary, + "application/octet-stream", + ) +} + +fn byte_chunk( + value: &[u8], + offset: u64, + limit: u32, + size_bytes: u64, + size_chars: Option, + display_mode: LargeValueType, + content_type: &str, +) -> Result { + let length = value.len() as u64; + if offset > length { + return Err(LargeValueError::InvalidRange { offset, length }); + } + let start = + usize::try_from(offset).map_err(|_| LargeValueError::InvalidRange { offset, length })?; + let end = start.saturating_add(limit as usize).min(value.len()); + Ok(LargeValueChunk { + value: BASE64_STANDARD.encode(&value[start..end]), + offset, + next_offset: end as u64, + eof: end == value.len(), + size_bytes, + size_chars, + encoding: LargeValueEncoding::Base64, + content_type: content_type.to_owned(), + display_mode, + }) +} + +fn validate_owner(owner_id: &str) -> Result<(), LargeValueError> { + if owner_id.trim().is_empty() { + Err(LargeValueError::InvalidOwner) + } else { + Ok(()) + } +} + +fn parse_token(token: &str) -> Result { + Uuid::parse_str(token).map_err(|_| LargeValueError::InvalidToken) +} diff --git a/crates/chat2db-core/src/lib.rs b/crates/chat2db-core/src/lib.rs index 57a62fe..73c8ec4 100644 --- a/crates/chat2db-core/src/lib.rs +++ b/crates/chat2db-core/src/lib.rs @@ -7,6 +7,7 @@ mod datasource_session; mod driver_pack; mod engine_manager; mod error; +mod large_value; mod native_mysql; mod operation; mod query; @@ -41,7 +42,12 @@ pub use agent::AgentRunSubscription; pub use community::load_fixed_community_classpath; pub use engine_manager::EngineLease; pub use error::{AppError, AppErrorKind}; +pub use large_value::{ + LargeValueChunk, LargeValueEncoding, LargeValueError, LargeValuePreview, LargeValueStoreStats, + LargeValueType, +}; pub use operation::OperationSubscription; +pub use query::{MysqlConsoleCancellation, MysqlConsoleRequest, MysqlConsoleResult}; use engine_manager::{ DEFAULT_ENGINE_IDLE_TIMEOUT, EngineManagerOwner, EngineManagerStatus, EngineProvider, @@ -64,6 +70,7 @@ pub(crate) struct ApplicationInner { engine: EngineProvider, drivers: Vec, managed_driver_ids: Option>, + large_values: large_value::LargeValueStore, agent_runs: AgentRunHub, operations: OperationHub, accepting_work: Mutex, @@ -239,6 +246,7 @@ impl Application { engine, drivers, managed_driver_ids, + large_values: large_value::LargeValueStore::default(), agent_runs: AgentRunHub::new(), operations: OperationHub::new(), accepting_work: Mutex::new(true), @@ -254,6 +262,116 @@ impl Application { self.inner.storage.as_ref() } + /// Creates one unpredictable scope for large values returned by a Console execution. + #[must_use] + pub fn create_large_value_owner(&self) -> String { + large_value::new_owner_id() + } + + /// Retains a potentially large UTF-8 cell and returns its bounded preview. + /// + /// # Errors + /// + /// Returns a closed validation or capacity failure without exposing retained values. + pub fn retain_large_text( + &self, + owner_id: &str, + value: String, + ) -> Result { + self.inner + .large_values + .retain_text(owner_id, value) + .map(|preview| large_value::scope_preview(owner_id, preview)) + .map_err(large_value_app_error) + } + + /// Retains a potentially large binary cell and returns its base64 preview. + /// + /// # Errors + /// + /// Returns a closed validation or capacity failure without exposing retained values. + pub fn retain_large_binary( + &self, + owner_id: &str, + value: Vec, + ) -> Result { + self.inner + .large_values + .retain_binary(owner_id, value) + .map(|preview| large_value::scope_preview(owner_id, preview)) + .map_err(large_value_app_error) + } + + /// Reads one bounded chunk through an opaque owner-scoped token. + /// + /// # Errors + /// + /// Returns a closed invalid, expired, inaccessible, or range error. + pub fn read_large_value_chunk( + &self, + large_value_id: &str, + offset: u64, + limit: u32, + ) -> Result { + let (owner_id, token) = + large_value::scoped_token(large_value_id).map_err(large_value_app_error)?; + self.inner + .large_values + .read_chunk(owner_id, token, offset, limit) + .map_err(large_value_app_error) + } + + /// Reads one base64 chunk whose offsets and limit are measured in raw bytes. + /// + /// # Errors + /// + /// Returns a closed invalid, expired, inaccessible, or range error. + pub fn read_large_value_encoded_chunk( + &self, + large_value_id: &str, + offset: u64, + limit: u32, + ) -> Result { + let (owner_id, token) = + large_value::scoped_token(large_value_id).map_err(large_value_app_error)?; + self.inner + .large_values + .read_encoded_chunk(owner_id, token, offset, limit) + .map_err(large_value_app_error) + } + + /// Removes one retained value addressed by its opaque token. + /// + /// # Errors + /// + /// Returns a closed invalid, expired, or inaccessible token error. + pub fn remove_large_value(&self, large_value_id: &str) -> Result<(), AppError> { + let (owner_id, token) = + large_value::scoped_token(large_value_id).map_err(large_value_app_error)?; + self.inner + .large_values + .remove_token(owner_id, token) + .map_err(large_value_app_error) + } + + /// Removes all retained values associated with one execution owner. + #[must_use] + pub fn remove_large_value_owner(&self, owner_id: &str) -> usize { + self.inner.large_values.remove_owner(owner_id) + } + + /// Releases expired retained values. + #[must_use] + pub fn cleanup_expired_large_values(&self) -> usize { + self.inner.large_values.cleanup_expired() + } + + /// Returns current retained-value usage after expiry cleanup. + #[must_use] + pub fn large_value_stats(&self) -> LargeValueStoreStats { + self.inner.large_values.stats() + } + pub(crate) fn require_storage(&self) -> Result { self.inner.storage.clone().ok_or_else(|| { AppError::unavailable( @@ -658,6 +776,31 @@ impl Application { } } +fn large_value_app_error(error: LargeValueError) -> AppError { + match error { + LargeValueError::NotFound | LargeValueError::Expired | LargeValueError::OwnerMismatch => { + AppError::not_found( + "largeCellValue.tokenExpired", + "The large cell value is no longer available", + ) + } + LargeValueError::CapacityExceeded { .. } => AppError::new( + AppErrorKind::ResourceExhausted, + chat2db_contract::ApiError::new( + "largeCellValue.fullValueUnsupported", + "The large cell value exceeds the retained-value limit", + ), + ), + LargeValueError::InvalidOwner + | LargeValueError::InvalidToken + | LargeValueError::InvalidLimit + | LargeValueError::InvalidRange { .. } => AppError::invalid( + "invalid_large_cell_value_request", + "The large cell value request is invalid", + ), + } +} + impl RuntimeHost { /// Opens production storage and discovers drivers without starting Java. /// diff --git a/crates/chat2db-core/src/native_mysql.rs b/crates/chat2db-core/src/native_mysql.rs index 23de413..8924be5 100644 --- a/crates/chat2db-core/src/native_mysql.rs +++ b/crates/chat2db-core/src/native_mysql.rs @@ -1,3 +1,4 @@ +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; use chat2db_contract::{ ApiError, CommunityDatabase, CommunityDatabaseList, CommunityForeignKey, CommunityForeignKeyList, CommunityFunction, CommunityFunctionList, CommunityFunctionParameter, @@ -6,8 +7,9 @@ use chat2db_contract::{ CommunityProcedureParameterList, CommunitySchemaList, CommunityTable, CommunityTableColumn, CommunityTableColumnList, CommunityTableIndex, CommunityTableIndexColumn, CommunityTableIndexList, CommunityTableList, CommunityTablePreviewAccepted, CommunityTrigger, - CommunityTriggerList, CommunityViewList, DatasourceConnection, QueryLimits, ResultMetadata, - StartCommunityTablePreviewRequest, StartQueryRequest, + CommunityTriggerList, CommunityViewList, DatasourceConnection, JdbcValue, JdbcValueType, + QueryLimits, ResultColumn, ResultMetadata, ResultRow, StartCommunityTablePreviewRequest, + StartQueryRequest, }; use chat2db_engine_protocol::wire; use chat2db_java_bridge::QueryOptions; @@ -18,7 +20,11 @@ use mysql_async::{ prelude::{FromValue, Queryable}, }; use prost::Message; -use std::{future::Future, time::Duration}; +use std::{ + future::Future, + mem::size_of, + time::{Duration, Instant}, +}; use tokio::sync::watch; use url::Url; @@ -26,7 +32,9 @@ use crate::{ AppError, AppErrorKind, Application, datasource_session::{ResolvedDatasourceConnection, resolve_datasource_connection}, operation::CancellationRequest, - query::{PreparedQuery, QueryTaskError, RetainedWriter}, + query::{ + MysqlConsoleRequest, MysqlConsoleResult, PreparedQuery, QueryTaskError, RetainedWriter, + }, }; const MYSQL_SCHEME: &str = "mysql://"; @@ -44,7 +52,11 @@ const MAX_BATCH_BYTES: u32 = wire::JdbcProtocolLimit::MaxBatchBytes as u32; const MAX_COLUMNS: usize = wire::JdbcProtocolLimit::MaxColumns as usize; const MAX_SQL_BYTES: usize = wire::JdbcProtocolLimit::MaxSqlBytes as usize; const MAX_SCALAR_BYTES: usize = wire::JdbcProtocolLimit::MaxScalarBytes as usize; +const MAX_CONSOLE_VALUE_BYTES: usize = 32 * 1024 * 1024; +const MAX_CONSOLE_RESULT_BYTES: u64 = DEFAULT_RESULT_BYTES; +const MAX_CONSOLE_STATEMENTS: usize = 1_000; const MAX_IDENTIFIER_BYTES: usize = 256; +const MAX_CONSOLE_PAGE_SIZE: u32 = 10_000; type TableRow = ( String, String, @@ -936,6 +948,865 @@ pub(crate) async fn start_table_preview( }) } +struct ConsoleStatementExecution { + results: Vec, + failure: Option, +} + +enum ConsoleExecutionError { + Cancelled(Option), + Fatal(AppError), +} + +pub(crate) async fn execute_console( + application: &Application, + request: MysqlConsoleRequest, + mut cancellation: watch::Receiver, +) -> Result, AppError> { + let (page_offset, page_end) = validate_console_request(&request)?; + let mut statements = if request.single { + vec![request.sql.trim().to_owned()] + } else { + split_mysql_script(&request.sql)? + }; + if statements.is_empty() { + return Err(AppError::invalid( + "invalid_mysql_console_request", + "sql must contain at least one MySQL statement", + )); + } + if request.explain { + for statement in &mut statements { + *statement = format!("EXPLAIN {statement}"); + } + } + + let initial_cancellation = { cancellation.borrow().clone() }; + if let CancellationRequest::Requested { reason } = initial_cancellation { + return Err(mysql_console_cancelled(reason)); + } + let resolved = resolve_native_connection(application, &request.datasource_id).await?; + if resolved.connection.read_only { + validate_read_only_console(&statements)?; + } + let options = connection_opts(&resolved.connection)?; + let mut conn = match open_query_connection(options.clone(), &mut cancellation).await { + Ok(conn) => conn, + Err(QueryTaskError::Cancelled(reason)) => return Err(mysql_console_cancelled(reason)), + Err(QueryTaskError::Failed(error)) => return Err(error), + }; + let connection_id = conn.id(); + + if !request.database_name.trim().is_empty() { + let database_name = quote_console_identifier(&request.database_name, "databaseName")?; + if let Err(error) = execute_console_control( + &mut conn, + options.clone(), + connection_id, + &format!("USE {database_name}"), + &mut cancellation, + ) + .await + { + return finish_console_error(conn, options, connection_id, error).await; + } + } + + let mut results = Vec::new(); + let mut retained_result_bytes = 0_u64; + for (index, statement) in statements.into_iter().enumerate() { + let statement_sequence = u32::try_from(index) + .ok() + .and_then(|index| index.checked_add(1)) + .ok_or_else(AppError::internal)?; + let started = Instant::now(); + let execution = execute_console_statement( + &mut conn, + options.clone(), + connection_id, + &statement, + statement_sequence, + page_offset, + page_end, + request.result_set_id, + &mut retained_result_bytes, + &mut cancellation, + ) + .await; + match execution { + Ok(mut execution) => { + results.append(&mut execution.results); + if let Some(error) = execution.failure { + results.push(console_failure_result( + statement_sequence, + statement, + &error, + elapsed_millis(started), + )); + if !request.error_continue { + break; + } + } + } + Err(error) => { + return finish_console_error(conn, options, connection_id, error).await; + } + } + } + + disconnect_connection(conn).await?; + Ok(results) +} + +#[allow(clippy::too_many_arguments, clippy::too_many_lines)] +async fn execute_console_statement( + conn: &mut Conn, + options: Opts, + connection_id: u32, + statement: &str, + statement_sequence: u32, + page_offset: u64, + page_end: u64, + selected_result_set_id: Option, + retained_result_bytes: &mut u64, + cancellation: &mut watch::Receiver, +) -> Result { + let statement_started = Instant::now(); + let query = conn.query_iter(statement); + tokio::pin!(query); + let mut cancellation_open = true; + let mut query_result = loop { + tokio::select! { + biased; + changed = cancellation.changed(), if cancellation_open => { + if changed.is_err() { + cancellation_open = false; + continue; + } + let request = { cancellation.borrow().clone() }; + if let CancellationRequest::Requested { reason } = request { + cancel_console_connection(options, connection_id).await + .map_err(ConsoleExecutionError::Fatal)?; + return Err(ConsoleExecutionError::Cancelled(reason)); + } + } + result = &mut query => { + match result { + Ok(result) => break result, + Err(error) if matches!(&error, MysqlError::Server(_)) => { + return Ok(ConsoleStatementExecution { + results: Vec::new(), + failure: Some(mysql_query_error(error)), + }); + } + Err(error) => { + return Err(ConsoleExecutionError::Fatal(mysql_query_error(error))); + } + } + } + } + }; + + let mut results = Vec::new(); + let mut result_set_id = 0_u32; + while let Some(columns) = query_result.columns() { + let result_started = Instant::now(); + let columns = columns.to_vec(); + let tabular = !columns.is_empty(); + let current_result_set_id = if tabular { + result_set_id = result_set_id + .checked_add(1) + .ok_or_else(|| ConsoleExecutionError::Fatal(AppError::internal()))?; + Some(result_set_id) + } else { + None + }; + let retain = current_result_set_id + .is_none_or(|id| selected_result_set_id.is_none_or(|selected| selected == id)); + let update_count = query_result.affected_rows(); + let info = query_result.info().into_owned(); + let converted_columns = if tabular && retain { + if columns.len() > MAX_COLUMNS { + return Err(ConsoleExecutionError::Fatal(resource_error( + "mysql_result_too_wide", + format!("MySQL returned more than {MAX_COLUMNS} columns"), + ))); + } + columns + .iter() + .enumerate() + .map(|(index, column)| console_column(index, column)) + .collect::, _>>() + .map_err(ConsoleExecutionError::Fatal)? + } else { + Vec::new() + }; + let mut rows = Vec::new(); + let mut row_count = 0_u64; + loop { + let next = tokio::select! { + biased; + changed = cancellation.changed(), if cancellation_open => { + if changed.is_err() { + cancellation_open = false; + continue; + } + let request = { cancellation.borrow().clone() }; + if let CancellationRequest::Requested { reason } = request { + cancel_console_connection(options.clone(), connection_id).await + .map_err(ConsoleExecutionError::Fatal)?; + return Err(ConsoleExecutionError::Cancelled(reason)); + } + continue; + } + row = query_result.next() => row, + }; + let row = match next { + Ok(Some(row)) => row, + Ok(None) => break, + Err(error) if matches!(&error, MysqlError::Server(_)) => { + return Ok(ConsoleStatementExecution { + results, + failure: Some(mysql_query_error(error)), + }); + } + Err(error) => { + return Err(ConsoleExecutionError::Fatal(mysql_query_error(error))); + } + }; + if retain && (page_offset..page_end).contains(&row_count) { + let row = console_row(row, &columns).map_err(ConsoleExecutionError::Fatal)?; + reserve_console_result_bytes(retained_result_bytes, &row) + .map_err(ConsoleExecutionError::Fatal)?; + rows.push(row); + } + row_count = row_count + .checked_add(1) + .ok_or_else(|| ConsoleExecutionError::Fatal(AppError::internal()))?; + } + + if retain { + results.push(MysqlConsoleResult { + statement_sequence, + result_set_id: current_result_set_id, + sql: statement.to_owned(), + success: true, + message: if info.is_empty() { + "Statement executed successfully".to_owned() + } else { + info + }, + update_count: if tabular { 0 } else { update_count }, + columns: converted_columns, + rows, + row_count, + has_more: row_count > page_end, + duration_ms: elapsed_millis(result_started), + error: None, + }); + } + } + + if results.is_empty() && selected_result_set_id.is_none() { + results.push(MysqlConsoleResult { + statement_sequence, + result_set_id: None, + sql: statement.to_owned(), + success: true, + message: "Statement executed successfully".to_owned(), + update_count: query_result.affected_rows(), + columns: Vec::new(), + rows: Vec::new(), + row_count: 0, + has_more: false, + duration_ms: elapsed_millis(statement_started), + error: None, + }); + } + Ok(ConsoleStatementExecution { + results, + failure: None, + }) +} + +async fn execute_console_control( + conn: &mut Conn, + options: Opts, + connection_id: u32, + sql: &str, + cancellation: &mut watch::Receiver, +) -> Result<(), ConsoleExecutionError> { + let query = conn.query_drop(sql); + tokio::pin!(query); + let mut cancellation_open = true; + loop { + tokio::select! { + biased; + changed = cancellation.changed(), if cancellation_open => { + if changed.is_err() { + cancellation_open = false; + continue; + } + let request = { cancellation.borrow().clone() }; + if let CancellationRequest::Requested { reason } = request { + cancel_console_connection(options, connection_id).await + .map_err(ConsoleExecutionError::Fatal)?; + return Err(ConsoleExecutionError::Cancelled(reason)); + } + } + result = &mut query => { + return result + .map_err(mysql_query_error) + .map_err(ConsoleExecutionError::Fatal); + } + } + } +} + +async fn finish_console_error( + conn: Conn, + options: Opts, + connection_id: u32, + error: ConsoleExecutionError, +) -> Result { + drop(conn); + match error { + ConsoleExecutionError::Cancelled(reason) => Err(mysql_console_cancelled(reason)), + ConsoleExecutionError::Fatal(error) => { + terminate_connection_quietly(options, connection_id).await; + Err(error) + } + } +} + +fn console_failure_result( + statement_sequence: u32, + sql: String, + error: &AppError, + duration_ms: u64, +) -> MysqlConsoleResult { + let api_error = error.api_error(); + MysqlConsoleResult { + statement_sequence, + result_set_id: None, + sql, + success: false, + message: api_error.message.clone(), + update_count: 0, + columns: Vec::new(), + rows: Vec::new(), + row_count: 0, + has_more: false, + duration_ms, + error: Some(api_error), + } +} + +fn validate_console_request(request: &MysqlConsoleRequest) -> Result<(u64, u64), AppError> { + if request.datasource_id.trim().is_empty() { + return Err(AppError::invalid( + "invalid_mysql_console_request", + "dataSourceId cannot be empty", + )); + } + if request.sql.trim().is_empty() { + return Err(AppError::invalid( + "invalid_mysql_console_request", + "sql cannot be empty", + )); + } + if request.sql.len() > MAX_SQL_BYTES { + return Err(resource_error( + "mysql_console_script_too_large", + format!("MySQL Console scripts are limited to {MAX_SQL_BYTES} bytes"), + )); + } + if !request.database_name.trim().is_empty() { + quote_console_identifier(&request.database_name, "databaseName")?; + } + if request.page_no == 0 || request.page_size == 0 || request.page_size > MAX_CONSOLE_PAGE_SIZE { + return Err(AppError::invalid( + "invalid_mysql_console_request", + format!( + "pageNo must be positive and pageSize must be between 1 and {MAX_CONSOLE_PAGE_SIZE}" + ), + )); + } + if request.result_set_id == Some(0) { + return Err(AppError::invalid( + "invalid_mysql_console_request", + "resultSetId must be a positive one-based integer", + )); + } + let (page_offset, page_end) = if request.page_size_all { + (0, u64::from(MAX_CONSOLE_PAGE_SIZE)) + } else { + let page_offset = u64::from(request.page_no - 1) * u64::from(request.page_size); + let page_end = page_offset + .checked_add(u64::from(request.page_size)) + .ok_or_else(AppError::internal)?; + (page_offset, page_end) + }; + Ok((page_offset, page_end)) +} + +fn validate_read_only_console(statements: &[String]) -> Result<(), AppError> { + for statement in statements { + let tokens = sql_tokens(statement)?; + let words = tokens + .iter() + .filter_map(|token| match token { + SqlToken::Word(word) => Some(word.as_str()), + SqlToken::Semicolon => None, + }) + .collect::>(); + let allowed = match words.as_slice() { + ["SELECT", ..] => validate_read_sql(statement).is_ok(), + [ + "SHOW" | "DESCRIBE" | "DESC" | "EXPLAIN" | "USE" | "COMMIT" | "ROLLBACK", + .., + ] + | ["START", "TRANSACTION", "READ", "ONLY", ..] => true, + _ => false, + }; + if !allowed { + return Err(AppError::new( + AppErrorKind::Conflict, + ApiError::new( + "datasource_read_only", + "The datasource connection is configured as read-only", + ), + )); + } + } + Ok(()) +} + +fn mysql_console_cancelled(reason: Option) -> AppError { + AppError::new( + AppErrorKind::Conflict, + ApiError::new( + "mysql_console_cancelled", + reason.unwrap_or_else(|| "The MySQL Console execution was cancelled".to_owned()), + ), + ) +} + +fn elapsed_millis(started: Instant) -> u64 { + u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX) +} + +fn console_column(index: usize, column: &Column) -> Result { + let column = mysql_column(index, column)?; + let value_type = match wire::JdbcValueType::try_from(column.value_type) { + Ok(wire::JdbcValueType::Boolean) => JdbcValueType::Boolean, + Ok(wire::JdbcValueType::SignedInteger) => JdbcValueType::SignedInteger, + Ok(wire::JdbcValueType::UnsignedInteger) => JdbcValueType::UnsignedInteger, + Ok(wire::JdbcValueType::Float32) => JdbcValueType::Float32, + Ok(wire::JdbcValueType::Float64) => JdbcValueType::Float64, + Ok(wire::JdbcValueType::Decimal) => JdbcValueType::Decimal, + Ok(wire::JdbcValueType::Text) => JdbcValueType::Text, + Ok(wire::JdbcValueType::Binary) => JdbcValueType::Binary, + Ok(wire::JdbcValueType::Date) => JdbcValueType::Date, + Ok(wire::JdbcValueType::Time) => JdbcValueType::Time, + Ok(wire::JdbcValueType::Timestamp) => JdbcValueType::Timestamp, + Ok(wire::JdbcValueType::TimestampWithTimeZone) => JdbcValueType::TimestampWithTimeZone, + Ok(wire::JdbcValueType::Json) => JdbcValueType::Json, + Ok(wire::JdbcValueType::Uuid) => JdbcValueType::Uuid, + Ok(wire::JdbcValueType::Opaque) => JdbcValueType::Opaque, + Ok(wire::JdbcValueType::Unspecified) | Err(_) => return Err(AppError::internal()), + }; + let nullability = match wire::ColumnNullability::try_from(column.nullability) { + Ok(wire::ColumnNullability::Unknown) => chat2db_contract::ColumnNullability::Unknown, + Ok(wire::ColumnNullability::NoNulls) => chat2db_contract::ColumnNullability::NoNulls, + Ok(wire::ColumnNullability::Nullable) => chat2db_contract::ColumnNullability::Nullable, + Err(_) => return Err(AppError::internal()), + }; + Ok(ResultColumn { + ordinal: column.ordinal, + label: column.label, + name: column.name, + jdbc_type: column.jdbc_type, + jdbc_type_name: column.jdbc_type_name, + value_type, + nullability, + precision: column.precision, + scale: column.scale, + display_size: column.display_size, + signed: column.signed, + catalog_name: column.catalog_name, + schema_name: column.schema_name, + table_name: column.table_name, + }) +} + +fn console_row(row: Row, columns: &[Column]) -> Result { + if row.len() != columns.len() { + return Err(AppError::internal()); + } + Ok(ResultRow { + values: row + .unwrap() + .into_iter() + .zip(columns) + .map(|(value, column)| console_mysql_value(value, column)) + .collect::, _>>()?, + }) +} + +fn reserve_console_result_bytes(total: &mut u64, row: &ResultRow) -> Result<(), AppError> { + let row_bytes = console_row_retained_bytes(row); + let next = total.saturating_add(row_bytes); + if next > MAX_CONSOLE_RESULT_BYTES { + return Err(resource_error( + "mysql_console_result_too_large", + format!( + "MySQL Console results are limited to {MAX_CONSOLE_RESULT_BYTES} retained bytes" + ), + )); + } + *total = next; + Ok(()) +} + +fn console_row_retained_bytes(row: &ResultRow) -> u64 { + let mut bytes = u64::try_from(size_of::()).unwrap_or(u64::MAX); + bytes = bytes.saturating_add( + u64::try_from(row.values.capacity()) + .unwrap_or(u64::MAX) + .saturating_mul(u64::try_from(size_of::()).unwrap_or(u64::MAX)), + ); + for value in &row.values { + let value_bytes = match value { + JdbcValue::Null | JdbcValue::Boolean { .. } => 0, + JdbcValue::SignedInteger { value } + | JdbcValue::UnsignedInteger { value } + | JdbcValue::Float32 { value } + | JdbcValue::Float64 { value } + | JdbcValue::Decimal { value } + | JdbcValue::Text { value } + | JdbcValue::Binary { value } + | JdbcValue::Date { value } + | JdbcValue::Time { value } + | JdbcValue::Timestamp { value } + | JdbcValue::TimestampWithTimeZone { value } + | JdbcValue::Json { value } + | JdbcValue::Uuid { value } => value.capacity(), + JdbcValue::Opaque { + type_name, + display_value, + } => type_name + .capacity() + .saturating_add(display_value.capacity()), + }; + bytes = bytes.saturating_add(u64::try_from(value_bytes).unwrap_or(u64::MAX)); + } + bytes +} + +fn console_mysql_value(value: Value, column: &Column) -> Result { + if matches!(value, Value::NULL) { + return Ok(JdbcValue::Null); + } + match mysql_value_type(column) { + wire::JdbcValueType::Text => Ok(JdbcValue::Text { + value: mysql_text_with_limit(value, MAX_CONSOLE_VALUE_BYTES)?, + }), + wire::JdbcValueType::Binary => Ok(JdbcValue::Binary { + value: BASE64_STANDARD.encode(mysql_binary_with_limit(value, MAX_CONSOLE_VALUE_BYTES)?), + }), + wire::JdbcValueType::Json => Ok(JdbcValue::Json { + value: mysql_text_with_limit(value, MAX_CONSOLE_VALUE_BYTES)?, + }), + _ => console_value(mysql_value(value, column)?), + } +} + +fn console_value(value: wire::JdbcValue) -> Result { + use wire::jdbc_value::Value as WireValue; + + Ok(match value.value.ok_or_else(AppError::internal)? { + WireValue::NullValue(_) => JdbcValue::Null, + WireValue::BooleanValue(value) => JdbcValue::Boolean { value }, + WireValue::SignedIntegerValue(value) => JdbcValue::SignedInteger { + value: value.to_string(), + }, + WireValue::UnsignedIntegerValue(value) => JdbcValue::UnsignedInteger { + value: value.to_string(), + }, + WireValue::Float32Value(value) => JdbcValue::Float32 { + value: console_float32(value), + }, + WireValue::Float64Value(value) => JdbcValue::Float64 { + value: console_float64(value), + }, + WireValue::DecimalValue(value) => JdbcValue::Decimal { value }, + WireValue::TextValue(value) => JdbcValue::Text { value }, + WireValue::BinaryValue(value) => JdbcValue::Binary { + value: BASE64_STANDARD.encode(value), + }, + WireValue::DateValue(value) => JdbcValue::Date { value }, + WireValue::TimeValue(value) => JdbcValue::Time { value }, + WireValue::TimestampValue(value) => JdbcValue::Timestamp { value }, + WireValue::TimestampWithTimeZoneValue(value) => JdbcValue::TimestampWithTimeZone { value }, + WireValue::JsonValue(value) => JdbcValue::Json { value }, + WireValue::UuidValue(value) => JdbcValue::Uuid { value }, + WireValue::OpaqueValue(value) => JdbcValue::Opaque { + type_name: value.type_name, + display_value: value.display_value, + }, + }) +} + +fn console_float32(value: f32) -> String { + if value.is_nan() { + "NaN".to_owned() + } else if value == f32::INFINITY { + "Infinity".to_owned() + } else if value == f32::NEG_INFINITY { + "-Infinity".to_owned() + } else { + value.to_string() + } +} + +fn console_float64(value: f64) -> String { + if value.is_nan() { + "NaN".to_owned() + } else if value == f64::INFINITY { + "Infinity".to_owned() + } else if value == f64::NEG_INFINITY { + "-Infinity".to_owned() + } else { + value.to_string() + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum ScriptState { + Normal, + SingleQuote, + DoubleQuote, + Backtick, + LineComment, + BlockComment, +} + +fn split_mysql_script(script: &str) -> Result, AppError> { + let bytes = script.as_bytes(); + let mut delimiter = b";".to_vec(); + let mut statements = Vec::new(); + let mut state = ScriptState::Normal; + let mut statement_start = 0_usize; + let mut index = 0_usize; + let mut at_line_start = true; + + while index < bytes.len() { + if state == ScriptState::Normal + && at_line_start + && bytes[statement_start..index] + .iter() + .all(u8::is_ascii_whitespace) + && let Some((new_delimiter, next_line)) = delimiter_directive(script, index)? + { + delimiter = new_delimiter.into_bytes(); + statement_start = next_line; + index = next_line; + at_line_start = true; + continue; + } + + match state { + ScriptState::Normal => { + if bytes[index..].starts_with(&delimiter) { + push_mysql_statement(&mut statements, &script[statement_start..index])?; + index += delimiter.len(); + statement_start = index; + at_line_start = false; + continue; + } + match bytes[index] { + b'\'' => state = ScriptState::SingleQuote, + b'"' => state = ScriptState::DoubleQuote, + b'`' => state = ScriptState::Backtick, + b'#' => state = ScriptState::LineComment, + b'-' if is_mysql_dash_comment(bytes, index) => { + state = ScriptState::LineComment; + index += 1; + } + b'/' if bytes.get(index + 1) == Some(&b'*') => { + state = ScriptState::BlockComment; + index += 1; + } + b'\n' => at_line_start = true, + byte if byte.is_ascii_whitespace() && at_line_start => {} + _ => at_line_start = false, + } + } + ScriptState::SingleQuote | ScriptState::DoubleQuote | ScriptState::Backtick => { + let quote = match state { + ScriptState::SingleQuote => b'\'', + ScriptState::DoubleQuote => b'"', + ScriptState::Backtick => b'`', + _ => unreachable!("quoted state is matched above"), + }; + if bytes[index] == b'\\' && state != ScriptState::Backtick { + index = index.saturating_add(1); + } else if bytes[index] == quote { + if bytes.get(index + 1) == Some("e) { + index += 1; + } else { + state = ScriptState::Normal; + } + } + } + ScriptState::LineComment => { + if bytes[index] == b'\n' { + state = ScriptState::Normal; + at_line_start = true; + } + } + ScriptState::BlockComment => { + if bytes[index] == b'\n' { + at_line_start = true; + } else if bytes[index] == b'*' && bytes.get(index + 1) == Some(&b'/') { + state = ScriptState::Normal; + index += 1; + } + } + } + index += 1; + } + + match state { + ScriptState::Normal | ScriptState::LineComment => {} + ScriptState::SingleQuote => { + return Err(invalid_console_script("unterminated string literal")); + } + ScriptState::DoubleQuote => { + return Err(invalid_console_script("unterminated quoted string literal")); + } + ScriptState::Backtick => { + return Err(invalid_console_script("unterminated quoted identifier")); + } + ScriptState::BlockComment => { + return Err(invalid_console_script("unterminated block comment")); + } + } + push_mysql_statement(&mut statements, &script[statement_start..])?; + Ok(statements) +} + +fn delimiter_directive( + script: &str, + line_start: usize, +) -> Result, AppError> { + const KEYWORD: &str = "delimiter"; + + let line_end = script.as_bytes()[line_start..] + .iter() + .position(|byte| *byte == b'\n') + .map_or(script.len(), |offset| line_start + offset); + let line = script[line_start..line_end].trim(); + let Some(prefix) = line.get(..KEYWORD.len()) else { + return Ok(None); + }; + if !prefix.eq_ignore_ascii_case(KEYWORD) + || line + .as_bytes() + .get(KEYWORD.len()) + .is_some_and(|byte| !byte.is_ascii_whitespace()) + { + return Ok(None); + } + let delimiter = line[KEYWORD.len()..].trim(); + if delimiter.is_empty() + || delimiter.len() > 16 + || delimiter.bytes().any(|byte| byte.is_ascii_whitespace()) + { + return Err(invalid_console_script( + "DELIMITER must name one non-whitespace token of at most 16 bytes", + )); + } + let next_line = if line_end < script.len() { + line_end + 1 + } else { + line_end + }; + Ok(Some((delimiter.to_owned(), next_line))) +} + +fn is_mysql_dash_comment(bytes: &[u8], index: usize) -> bool { + bytes.get(index + 1) == Some(&b'-') + && bytes + .get(index + 2) + .is_none_or(|byte| byte.is_ascii_whitespace() || byte.is_ascii_control()) +} + +fn push_mysql_statement(statements: &mut Vec, statement: &str) -> Result<(), AppError> { + let statement = statement.trim(); + if !statement.is_empty() { + if statements.len() >= MAX_CONSOLE_STATEMENTS { + return Err(resource_error( + "mysql_console_too_many_statements", + format!("MySQL Console scripts are limited to {MAX_CONSOLE_STATEMENTS} statements"), + )); + } + statements.push(statement.to_owned()); + } + Ok(()) +} + +fn invalid_console_script(detail: &str) -> AppError { + AppError::invalid( + "invalid_mysql_console_script", + format!("The MySQL Console script contains an {detail}"), + ) +} + +fn quote_console_identifier(value: &str, field: &str) -> Result { + if value.trim().is_empty() || value.len() > MAX_IDENTIFIER_BYTES || value.contains('\0') { + return Err(AppError::invalid( + "invalid_mysql_console_request", + format!("{field} is invalid"), + )); + } + Ok(format!("`{}`", value.replace('`', "``"))) +} + +async fn cancel_console_connection(options: Opts, connection_id: u32) -> Result<(), AppError> { + let mut control = open_connection_with_opts(options).await?; + let query_cancel = + kill_console_target(&mut control, format!("KILL QUERY {connection_id}")).await; + let connection_cancel = + kill_console_target(&mut control, format!("KILL CONNECTION {connection_id}")).await; + disconnect_quietly(control).await; + match (query_cancel, connection_cancel) { + (_, Ok(())) => Ok(()), + (Ok(()), Err(error)) => Err(error), + (Err(query_error), Err(connection_error)) => { + tracing::warn!(error = %query_error, "native MySQL query cancellation failed before connection termination"); + Err(connection_error) + } + } +} + +async fn kill_console_target(conn: &mut Conn, sql: String) -> Result<(), AppError> { + let result = tokio::time::timeout(CONTROL_TIMEOUT, conn.query_drop(sql)) + .await + .map_err(|_| { + AppError::unavailable( + "mysql_termination_timeout", + "The MySQL Console connection could not be terminated in time", + ) + })?; + match result { + Ok(()) => Ok(()), + Err(MysqlError::Server(server)) if server.code == 1094 => Ok(()), + Err(error) => Err(mysql_query_error(error)), + } +} + #[allow(clippy::too_many_lines)] pub(crate) async fn execute_query_task( application: &Application, @@ -1544,8 +2415,12 @@ fn mysql_f64(value: Value) -> Result { } fn mysql_text(value: Value) -> Result { + mysql_text_with_limit(value, MAX_SCALAR_BYTES) +} + +fn mysql_text_with_limit(value: Value, max_bytes: usize) -> Result { match value { - Value::Bytes(value) => mysql_utf8(value), + Value::Bytes(value) => mysql_utf8_with_limit(value, max_bytes), Value::Int(value) => Ok(value.to_string()), Value::UInt(value) => Ok(value.to_string()), Value::Float(value) => Ok(value.to_string()), @@ -1561,21 +2436,29 @@ fn mysql_text(value: Value) -> Result { } fn mysql_binary(value: Value) -> Result, AppError> { + mysql_binary_with_limit(value, MAX_SCALAR_BYTES) +} + +fn mysql_binary_with_limit(value: Value, max_bytes: usize) -> Result, AppError> { match value { - Value::Bytes(value) if value.len() <= MAX_SCALAR_BYTES => Ok(value), + Value::Bytes(value) if value.len() <= max_bytes => Ok(value), Value::Bytes(_) => Err(resource_error( "mysql_scalar_too_large", - format!("A MySQL value exceeds {MAX_SCALAR_BYTES} bytes"), + format!("A MySQL value exceeds {max_bytes} bytes"), )), - other => Ok(mysql_text(other)?.into_bytes()), + other => Ok(mysql_text_with_limit(other, max_bytes)?.into_bytes()), } } fn mysql_utf8(value: Vec) -> Result { - if value.len() > MAX_SCALAR_BYTES { + mysql_utf8_with_limit(value, MAX_SCALAR_BYTES) +} + +fn mysql_utf8_with_limit(value: Vec, max_bytes: usize) -> Result { + if value.len() > max_bytes { return Err(resource_error( "mysql_scalar_too_large", - format!("A MySQL value exceeds {MAX_SCALAR_BYTES} bytes"), + format!("A MySQL value exceeds {max_bytes} bytes"), )); } String::from_utf8(value).map_err(|_| result_decode_error()) @@ -2447,14 +3330,336 @@ fn mysql_query_error(error: MysqlError) -> AppError { #[cfg(test)] mod tests { - use chat2db_contract::{DatasourceConnection, DatasourceConnectionProperty}; + use chat2db_contract::{ + DatasourceConnection, DatasourceConnectionProperty, JdbcValue, ResultRow, + }; + use mysql_async::{Conn, Opts}; + use tokio::sync::watch; use super::{ - community_column, community_foreign_key, community_function_parameter, community_indexes, - community_procedure_parameter, connection_opts, is_mysql_database_type, - is_native_read_candidate, normalize_table_type, qualified_identifier, quote_identifier, - validate_read_sql, + ConsoleExecutionError, ConsoleStatementExecution, MAX_CONSOLE_PAGE_SIZE, + MAX_CONSOLE_RESULT_BYTES, community_column, community_foreign_key, + community_function_parameter, community_indexes, community_procedure_parameter, + connection_opts, execute_console_statement, is_mysql_database_type, + is_native_read_candidate, normalize_table_type, open_connection_with_opts, + qualified_identifier, quote_identifier, reserve_console_result_bytes, split_mysql_script, + validate_console_request, validate_read_only_console, validate_read_sql, }; + use crate::{MysqlConsoleRequest, operation::CancellationRequest}; + + #[test] + fn mysql_console_splitter_respects_literals_identifiers_and_comments() { + let statements = split_mysql_script( + "SELECT ';' AS value; SELECT `odd;name` FROM items; -- keep ; here\nSELECT 3", + ) + .expect("valid script should split"); + + assert_eq!(statements.len(), 3); + assert_eq!(statements[0], "SELECT ';' AS value"); + assert_eq!(statements[1], "SELECT `odd;name` FROM items"); + assert!(statements[2].ends_with("SELECT 3")); + } + + #[test] + fn mysql_console_splitter_supports_delimiter_routine_scripts() { + let statements = split_mysql_script( + "DELIMITER $$\nCREATE PROCEDURE load_items()\nBEGIN\n SELECT ';' AS semi;\n SELECT '$$' AS marker;\nEND$$\nDELIMITER ;\nCALL load_items();", + ) + .expect("routine script should split on client delimiters"); + + assert_eq!(statements.len(), 2); + assert!(statements[0].starts_with("CREATE PROCEDURE load_items()")); + assert!(statements[0].ends_with("END")); + assert_eq!(statements[1], "CALL load_items()"); + } + + #[test] + fn mysql_console_splitter_does_not_treat_arithmetic_dashes_as_comments() { + let statements = split_mysql_script("SELECT 4--2; SELECT 3") + .expect("arithmetic dashes should remain SQL"); + + assert_eq!(statements, ["SELECT 4--2", "SELECT 3"]); + } + + #[test] + fn mysql_console_splitter_rejects_unterminated_lexemes_and_bad_delimiters() { + for script in [ + "SELECT 'unterminated", + "SELECT `unterminated", + "SELECT 1 /* unterminated", + "DELIMITER\nSELECT 1", + "DELIMITER too long delimiter\nSELECT 1", + ] { + let error = split_mysql_script(script).expect_err("script must be rejected"); + assert!( + matches!( + error.api_error().code.as_str(), + "invalid_mysql_console_script" + ), + "unexpected error for {script}: {error}" + ); + } + } + + #[test] + fn mysql_console_splitter_rejects_excessive_statement_counts() { + let script = std::iter::repeat_n("SELECT 1", 1_001) + .collect::>() + .join(";"); + let error = split_mysql_script(&script).expect_err("statement flood must be rejected"); + + assert_eq!(error.api_error().code, "mysql_console_too_many_statements"); + } + + #[tokio::test] + #[allow(clippy::too_many_lines)] + #[ignore = "requires MYSQL_TEST_HOST, MYSQL_TEST_PORT, MYSQL_TEST_USER, and MYSQL_TEST_PASSWORD"] + async fn live_mysql_console_kernel_preserves_session_results_and_cancellation() { + let host = std::env::var("MYSQL_TEST_HOST").expect("MYSQL_TEST_HOST is required"); + let port = std::env::var("MYSQL_TEST_PORT") + .expect("MYSQL_TEST_PORT is required") + .parse::() + .expect("MYSQL_TEST_PORT must be a TCP port"); + let user = std::env::var("MYSQL_TEST_USER").expect("MYSQL_TEST_USER is required"); + let password = + std::env::var("MYSQL_TEST_PASSWORD").expect("MYSQL_TEST_PASSWORD is required"); + let options = connection_opts(&DatasourceConnection { + jdbc_url: format!("mysql://{host}:{port}?sslMode=DISABLED"), + properties: vec![ + DatasourceConnectionProperty { + key: "user".to_owned(), + value: user, + sensitive: false, + }, + DatasourceConnectionProperty { + key: "password".to_owned(), + value: password, + sensitive: true, + }, + ], + read_only: false, + }) + .expect("live MySQL options should build"); + let mut conn = open_connection_with_opts(options.clone()) + .await + .expect("live MySQL should connect"); + let connection_id = conn.id(); + let (_sender, mut cancellation) = watch::channel(CancellationRequest::Waiting); + + live_console_statement( + &mut conn, + &options, + connection_id, + "USE mysql", + 1, + None, + &mut cancellation, + ) + .await; + live_console_statement( + &mut conn, + &options, + connection_id, + "CREATE TEMPORARY TABLE chat2db_console_kernel (id INT PRIMARY KEY)", + 2, + None, + &mut cancellation, + ) + .await; + live_console_statement( + &mut conn, + &options, + connection_id, + "START TRANSACTION", + 3, + None, + &mut cancellation, + ) + .await; + let insert = live_console_statement( + &mut conn, + &options, + connection_id, + "INSERT INTO chat2db_console_kernel VALUES (1)", + 4, + None, + &mut cancellation, + ) + .await; + assert!( + !insert.results.is_empty(), + "insert returned no results: {:?}", + insert.failure + ); + assert_eq!(insert.results[0].update_count, 1); + + let inside_transaction = live_console_statement( + &mut conn, + &options, + connection_id, + "SELECT COUNT(*) AS item_count FROM chat2db_console_kernel", + 5, + None, + &mut cancellation, + ) + .await; + assert_console_integer(&inside_transaction, "1"); + live_console_statement( + &mut conn, + &options, + connection_id, + "ROLLBACK", + 6, + None, + &mut cancellation, + ) + .await; + let after_rollback = live_console_statement( + &mut conn, + &options, + connection_id, + "SELECT COUNT(*) AS item_count FROM chat2db_console_kernel", + 7, + None, + &mut cancellation, + ) + .await; + assert_console_integer(&after_rollback, "0"); + + let selected_result = live_console_statement( + &mut conn, + &options, + connection_id, + "SELECT 11 AS value; SELECT 22 AS value", + 8, + Some(2), + &mut cancellation, + ) + .await; + assert_eq!(selected_result.results.len(), 1); + assert_eq!(selected_result.results[0].result_set_id, Some(2)); + assert_console_integer(&selected_result, "22"); + + let failed = live_console_statement( + &mut conn, + &options, + connection_id, + "SELECT missing_column FROM chat2db_console_kernel", + 9, + None, + &mut cancellation, + ) + .await; + assert_eq!( + failed + .failure + .expect("invalid statement should be represented as a recoverable failure") + .api_error() + .code, + "mysql_query_failed" + ); + let after_failure = live_console_statement( + &mut conn, + &options, + connection_id, + "SELECT 33 AS value", + 10, + None, + &mut cancellation, + ) + .await; + assert_console_integer(&after_failure, "33"); + + let (cancel, mut cancellation) = watch::channel(CancellationRequest::Waiting); + let mut retained_result_bytes = 0; + let cancel_task = tokio::spawn(async move { + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + let _ = cancel.send(CancellationRequest::Requested { + reason: Some("live cancellation".to_owned()), + }); + }); + let cancelled = execute_console_statement( + &mut conn, + options.clone(), + connection_id, + "SELECT SLEEP(30)", + 11, + 0, + 1, + None, + &mut retained_result_bytes, + &mut cancellation, + ) + .await; + assert!(matches!( + cancelled, + Err(ConsoleExecutionError::Cancelled(reason)) + if reason.as_deref() == Some("live cancellation") + )); + cancel_task.await.expect("cancellation task should join"); + drop(conn); + } + + async fn live_console_statement( + conn: &mut Conn, + options: &Opts, + connection_id: u32, + sql: &str, + statement_sequence: u32, + result_set_id: Option, + cancellation: &mut watch::Receiver, + ) -> ConsoleStatementExecution { + let mut retained_result_bytes = 0; + match execute_console_statement( + conn, + options.clone(), + connection_id, + sql, + statement_sequence, + 0, + 100, + result_set_id, + &mut retained_result_bytes, + cancellation, + ) + .await + { + Ok(execution) => execution, + Err(ConsoleExecutionError::Cancelled(_)) => { + panic!("live statement was unexpectedly cancelled") + } + Err(ConsoleExecutionError::Fatal(error)) => { + panic!("live statement failed fatally: {error}") + } + } + } + + #[test] + fn mysql_console_result_budget_is_global_and_closed() { + let row = ResultRow { + values: vec![JdbcValue::Text { + value: "bounded".to_owned(), + }], + }; + let mut retained = 0; + reserve_console_result_bytes(&mut retained, &row).expect("a small Console row must fit"); + assert!(retained > 0); + + retained = MAX_CONSOLE_RESULT_BYTES; + let error = reserve_console_result_bytes(&mut retained, &row) + .expect_err("the global Console result budget must be enforced"); + assert_eq!(error.api_error().code, "mysql_console_result_too_large"); + assert_eq!(retained, MAX_CONSOLE_RESULT_BYTES); + } + + fn assert_console_integer(execution: &ConsoleStatementExecution, expected: &str) { + let value = &execution.results[0].rows[0].values[0]; + match value { + chat2db_contract::JdbcValue::SignedInteger { value } + | chat2db_contract::JdbcValue::UnsignedInteger { value } => assert_eq!(value, expected), + other => panic!("expected an integer Console value, got {other:?}"), + } + } #[test] fn jdbc_url_and_properties_build_native_options_without_exposing_jdbc() { @@ -2567,6 +3772,59 @@ mod tests { } } + #[test] + fn console_read_only_policy_allows_inspection_and_rejects_writes() { + for sql in [ + "SELECT 1", + "SHOW TABLES", + "DESCRIBE items", + "EXPLAIN SELECT * FROM items", + "USE inventory", + "START TRANSACTION READ ONLY", + "COMMIT", + "ROLLBACK", + ] { + validate_read_only_console(&[sql.to_owned()]) + .unwrap_or_else(|error| panic!("{sql} should be read-only: {error}")); + } + for sql in [ + "UPDATE items SET label = 'changed'", + "DELETE FROM items", + "CALL mutating_procedure()", + "SELECT * FROM items FOR UPDATE", + "WITH cte AS (SELECT 1) SELECT * FROM cte", + ] { + let error = validate_read_only_console(&[sql.to_owned()]) + .expect_err("writes and ambiguous statements must fail closed"); + assert_eq!(error.api_error().code, "datasource_read_only"); + } + } + + #[test] + fn console_page_size_all_uses_the_bounded_complete_window() { + let mut request = MysqlConsoleRequest { + datasource_id: "datasource-1".to_owned(), + database_name: "inventory".to_owned(), + sql: "SELECT 1".to_owned(), + page_no: 2, + page_size: 10, + result_set_id: None, + single: false, + page_size_all: false, + explain: false, + error_continue: true, + }; + assert_eq!( + validate_console_request(&request).expect("paged request must validate"), + (10, 20) + ); + request.page_size_all = true; + assert_eq!( + validate_console_request(&request).expect("all-rows request must validate"), + (0, u64::from(MAX_CONSOLE_PAGE_SIZE)) + ); + } + #[test] fn mysql_column_metadata_preserves_community_projection() { let column = community_column( diff --git a/crates/chat2db-core/src/query.rs b/crates/chat2db-core/src/query.rs index 0c3b07f..cf3428d 100644 --- a/crates/chat2db-core/src/query.rs +++ b/crates/chat2db-core/src/query.rs @@ -1,6 +1,8 @@ use std::time::Duration; -use chat2db_contract::{ApiError, QueryAccepted, QueryLimits, StartQueryRequest}; +use chat2db_contract::{ + ApiError, QueryAccepted, QueryLimits, ResultColumn, ResultRow, StartQueryRequest, +}; use chat2db_engine_protocol::wire; use chat2db_java_bridge::{ BridgeError, CancelDisposition as BridgeCancelDisposition, ConnectionProperty, JdbcParameter, @@ -29,6 +31,118 @@ pub(crate) struct PreparedQuery { pub(crate) force_read_only: bool, } +/// One native `MySQL` Console execution request. +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[allow(clippy::struct_excessive_bools)] +#[serde(rename_all = "camelCase")] +pub struct MysqlConsoleRequest { + /// Opaque datasource id resolved by Core. + pub datasource_id: String, + /// Optional `MySQL` database selected on the Console connection. + #[serde(default)] + pub database_name: String, + /// One statement or a semicolon-delimited `MySQL` script. + pub sql: String, + /// One-based result page number. + pub page_no: u32, + /// Number of rows retained for each tabular result set. + pub page_size: u32, + /// Optional one-based tabular result-set id to retain per statement. + #[serde(default)] + pub result_set_id: Option, + /// Whether the submitted SQL must be dispatched as one preserved statement. + #[serde(default)] + pub single: bool, + /// Whether the bounded all-rows window is used instead of the requested page. + #[serde(default)] + pub page_size_all: bool, + /// Whether each parsed statement is executed through `EXPLAIN`. + #[serde(default)] + pub explain: bool, + /// Whether execution proceeds to the next statement after a database error. + #[serde(default = "default_console_error_continue")] + pub error_continue: bool, +} + +/// One statement result emitted by native `MySQL` Console execution. +#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct MysqlConsoleResult { + /// One-based statement position in the submitted script. + pub statement_sequence: u32, + /// One-based tabular result-set position within the statement. + #[serde(skip_serializing_if = "Option::is_none")] + pub result_set_id: Option, + /// The exact statement sent to `MySQL`. + pub sql: String, + /// Whether this individual statement result succeeded. + pub success: bool, + /// Safe database or execution message. + pub message: String, + /// Server-reported affected rows for non-tabular results. + pub update_count: u64, + /// Portable result columns in display order. + pub columns: Vec, + /// The requested page of portable result rows. + pub rows: Vec, + /// Exact rows observed in the complete tabular result set. + pub row_count: u64, + /// Whether rows exist beyond the requested page. + pub has_more: bool, + /// Wall-clock execution and fetch time in milliseconds. + pub duration_ms: u64, + /// Safe statement failure, present only when `success` is false. + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +/// Cloneable cancellation source for one native `MySQL` Console execution. +#[derive(Debug, Clone)] +pub struct MysqlConsoleCancellation { + sender: watch::Sender, +} + +impl MysqlConsoleCancellation { + /// Creates an active cancellation source. + #[must_use] + pub fn new() -> Self { + let (sender, _receiver) = watch::channel(CancellationRequest::Waiting); + Self { sender } + } + + /// Requests cancellation once and preserves the first supplied reason. + #[must_use] + pub fn cancel(&self, reason: Option) -> bool { + self.sender.send_if_modified(|state| { + if *state != CancellationRequest::Waiting { + return false; + } + *state = CancellationRequest::Requested { reason }; + true + }) + } + + /// Reports whether cancellation was already requested. + #[must_use] + pub fn is_cancelled(&self) -> bool { + *self.sender.borrow() != CancellationRequest::Waiting + } + + fn subscribe(&self) -> watch::Receiver { + self.sender.subscribe() + } +} + +impl Default for MysqlConsoleCancellation { + fn default() -> Self { + Self::new() + } +} + +const fn default_console_error_continue() -> bool { + true +} + enum QueryBackend { Java { engine: EngineLease, @@ -67,6 +181,21 @@ pub(crate) struct RetainedWriter { } impl Application { + /// Executes an arbitrary `MySQL` Console script natively on one connection. + /// + /// # Errors + /// + /// Returns validation, datasource, connection, cancellation, or unrecoverable + /// protocol failures. Recoverable `MySQL` statement errors are represented in + /// the returned result list so `error_continue` can be honored. + pub async fn execute_mysql_console( + &self, + request: MysqlConsoleRequest, + cancellation: MysqlConsoleCancellation, + ) -> Result, AppError> { + crate::native_mysql::execute_console(self, request, cancellation.subscribe()).await + } + /// Accepts a query for asynchronous execution and returns its operation id. /// /// # Errors @@ -792,10 +921,40 @@ mod tests { use chat2db_java_bridge::{BridgeError, RemoteEngineError}; use super::{ - AppError, QueryTaskError, is_inactive_credit_grant_error, preserve_primary_outcome, - validate_credit_grant, + AppError, MysqlConsoleCancellation, MysqlConsoleRequest, QueryTaskError, + is_inactive_credit_grant_error, preserve_primary_outcome, validate_credit_grant, }; + #[test] + fn mysql_console_cancellation_preserves_the_first_reason() { + let cancellation = MysqlConsoleCancellation::new(); + let receiver = cancellation.subscribe(); + + assert!(cancellation.cancel(Some("first".to_owned()))); + assert!(!cancellation.cancel(Some("second".to_owned()))); + assert!(cancellation.is_cancelled()); + assert!(matches!( + &*receiver.borrow(), + super::CancellationRequest::Requested { reason } + if reason.as_deref() == Some("first") + )); + } + + #[test] + fn mysql_console_json_defaults_to_continuing_after_statement_errors() { + let request: MysqlConsoleRequest = serde_json::from_value(serde_json::json!({ + "datasourceId": "mysql-1", + "sql": "SELECT 1", + "pageNo": 1, + "pageSize": 200 + })) + .expect("console request should deserialize"); + + assert!(request.error_continue); + assert_eq!(request.result_set_id, None); + assert!(request.database_name.is_empty()); + } + #[test] fn query_progress_requires_the_requested_credit_to_be_accepted() { validate_credit_grant(1).expect("one accepted credit makes progress"); diff --git a/crates/chat2db-core/tests/large_value.rs b/crates/chat2db-core/tests/large_value.rs new file mode 100644 index 0000000..11ffc28 --- /dev/null +++ b/crates/chat2db-core/tests/large_value.rs @@ -0,0 +1,327 @@ +#[allow(dead_code)] +#[path = "../src/large_value.rs"] +mod large_value; + +use std::time::Duration; + +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64_STANDARD}; +use large_value::{ + LargeValueEncoding, LargeValueError, LargeValueStore, LargeValueStoreConfig, LargeValueType, +}; +use uuid::{Uuid, Version}; + +fn config() -> LargeValueStoreConfig { + LargeValueStoreConfig { + preview_bytes: 4, + max_chunk_size: 4, + max_value_bytes: 64, + max_total_bytes: 128, + max_entries: 8, + ttl: Duration::from_secs(600), + } +} + +fn token(preview: &large_value::LargeValuePreview) -> &str { + preview + .large_value_id + .as_deref() + .expect("truncated values must receive a token") +} + +#[test] +fn unicode_and_binary_previews_are_bounded_and_explicitly_encoded() { + let store = LargeValueStore::new(config()); + let text = store + .retain_text("execution-1", "a你🙂z".to_owned()) + .expect("text should retain"); + assert_eq!(text.value, "a你"); + assert_eq!(text.loaded_bytes, 4); + assert_eq!(text.loaded_chars, Some(2)); + assert_eq!(text.size_bytes, 9); + assert_eq!(text.size_chars, Some(4)); + assert_eq!(text.value_type, LargeValueType::Text); + assert_eq!(text.encoding, LargeValueEncoding::Utf8); + assert!(text.large_value); + assert!(text.truncated); + + let bytes = vec![0, 1, 2, 3, 4, 255]; + let binary = store + .retain_binary("execution-1", bytes.clone()) + .expect("binary should retain"); + assert_eq!(binary.value, BASE64_STANDARD.encode(&bytes[..4])); + assert_eq!(binary.loaded_bytes, 4); + assert_eq!(binary.loaded_chars, None); + assert_eq!(binary.size_chars, None); + assert_eq!(binary.value_type, LargeValueType::Binary); + assert_eq!(binary.encoding, LargeValueEncoding::Base64); + + let inline = store + .retain_text("execution-1", "rust".to_owned()) + .expect("small text should remain inline"); + assert!(!inline.large_value); + assert!(!inline.truncated); + assert!(inline.large_value_id.is_none()); +} + +#[test] +fn tokens_are_unique_opaque_uuid_v4_values_and_owner_bound() { + let store = LargeValueStore::new(config()); + let first = store + .retain_text("execution-1", "first value".to_owned()) + .expect("first value should retain"); + let second = store + .retain_text("execution-1", "second value".to_owned()) + .expect("second value should retain"); + assert_ne!(token(&first), token(&second)); + for value in [&first, &second] { + let parsed = Uuid::parse_str(token(value)).expect("token must be a UUID"); + assert_eq!(parsed.get_version(), Some(Version::Random)); + } + + assert_eq!( + store.read_chunk("execution-2", token(&first), 0, 1), + Err(LargeValueError::OwnerMismatch) + ); + assert_eq!( + store.read_chunk("", token(&first), 0, 1), + Err(LargeValueError::InvalidOwner) + ); + assert_eq!( + store.read_chunk("execution-1", "not-a-token", 0, 1), + Err(LargeValueError::InvalidToken) + ); +} + +#[test] +fn zero_ttl_expires_on_first_read_and_releases_capacity() { + let mut limits = config(); + limits.ttl = Duration::ZERO; + limits.max_entries = 1; + limits.max_total_bytes = 16; + let store = LargeValueStore::new(limits); + let preview = store + .retain_text("execution-1", "expired".to_owned()) + .expect("value should receive an immediately expiring token"); + + assert_eq!( + store.read_chunk("execution-1", token(&preview), 0, 1), + Err(LargeValueError::Expired) + ); + assert_eq!(store.stats().entries, 0); + assert_eq!(store.stats().total_bytes, 0); + assert_eq!(store.cleanup_expired(), 0); + + let replacement = store + .retain_text("execution-1", "another".to_owned()) + .expect("expired capacity must be reusable"); + assert!(replacement.large_value_id.is_some()); +} + +#[test] +fn capacity_limits_reject_oversized_values_and_evict_oldest_entries() { + let mut limits = config(); + limits.preview_bytes = 1; + limits.max_value_bytes = 6; + limits.max_total_bytes = 10; + limits.max_entries = 2; + let store = LargeValueStore::new(limits.clone()); + + let too_large = store + .retain_text("execution-1", "1234567".to_owned()) + .expect_err("single value limit must be enforced"); + assert_eq!( + too_large, + LargeValueError::CapacityExceeded { + requested_bytes: 7, + max_value_bytes: 6, + max_total_bytes: 10, + max_entries: 2, + } + ); + + let first = store + .retain_text("execution-1", "12345".to_owned()) + .expect("first value should retain"); + let second = store + .retain_text("execution-1", "abcde".to_owned()) + .expect("second value should retain"); + let third = store + .retain_text("execution-1", "vwxyz".to_owned()) + .expect("third value should evict the oldest"); + assert_eq!(store.stats().entries, 2); + assert_eq!(store.stats().total_bytes, 10); + assert_eq!( + store.read_chunk("execution-1", token(&first), 0, 1), + Err(LargeValueError::NotFound) + ); + assert!( + store + .read_chunk("execution-1", token(&second), 0, 1) + .is_ok() + ); + assert!(store.read_chunk("execution-1", token(&third), 0, 1).is_ok()); + + limits.max_entries = 0; + let disabled = LargeValueStore::new(limits); + assert!(matches!( + disabled.retain_binary("execution-1", vec![1, 2]), + Err(LargeValueError::CapacityExceeded { .. }) + )); +} + +#[test] +fn text_chunks_preserve_unicode_without_overlap_or_gaps() { + let mut limits = config(); + limits.preview_bytes = 1; + limits.max_chunk_size = 2; + let store = LargeValueStore::new(limits); + let source = "A你🙂BC界"; + let preview = store + .retain_text("execution-1", source.to_owned()) + .expect("text should retain"); + + let mut offset = 0; + let mut rebuilt = String::new(); + loop { + let chunk = store + .read_chunk("execution-1", token(&preview), offset, u32::MAX) + .expect("chunk should read"); + assert_eq!(chunk.offset, offset); + assert_eq!(chunk.encoding, LargeValueEncoding::Utf8); + assert_eq!(chunk.content_type, "text/plain"); + assert_eq!(chunk.display_mode, LargeValueType::Text); + rebuilt.push_str(&chunk.value); + assert!(chunk.next_offset > offset || chunk.eof); + offset = chunk.next_offset; + if chunk.eof { + break; + } + } + assert_eq!(rebuilt, source); + assert_eq!(offset, source.chars().count() as u64); +} + +#[test] +fn encoded_text_chunks_use_raw_byte_offsets() { + let mut limits = config(); + limits.preview_bytes = 1; + limits.max_chunk_size = 3; + let store = LargeValueStore::new(limits); + let source = "A你🙂BC界"; + let preview = store + .retain_text("execution-1", source.to_owned()) + .expect("text should retain"); + + let mut offset = 0; + let mut rebuilt = Vec::new(); + loop { + let chunk = store + .read_encoded_chunk("execution-1", token(&preview), offset, u32::MAX) + .expect("encoded chunk should read"); + let decoded = BASE64_STANDARD + .decode(&chunk.value) + .expect("encoded text chunk must be base64"); + assert_eq!(chunk.offset, offset); + assert_eq!(chunk.next_offset, offset + decoded.len() as u64); + assert_eq!(chunk.encoding, LargeValueEncoding::Base64); + assert_eq!(chunk.content_type, "text/plain"); + assert_eq!(chunk.display_mode, LargeValueType::Text); + rebuilt.extend(decoded); + offset = chunk.next_offset; + if chunk.eof { + break; + } + } + assert_eq!(rebuilt, source.as_bytes()); + assert_eq!(offset, source.len() as u64); +} + +#[test] +fn binary_chunks_round_trip_and_range_validation_is_closed() { + let mut limits = config(); + limits.preview_bytes = 1; + limits.max_chunk_size = 3; + let store = LargeValueStore::new(limits); + let source = (0_u8..=12).collect::>(); + let preview = store + .retain_binary("execution-1", source.clone()) + .expect("binary should retain"); + + let mut offset = 0; + let mut rebuilt = Vec::new(); + loop { + let chunk = store + .read_chunk("execution-1", token(&preview), offset, u32::MAX) + .expect("chunk should read"); + let decoded = BASE64_STANDARD + .decode(&chunk.value) + .expect("chunk must be base64"); + assert_eq!(chunk.offset, offset); + assert_eq!(chunk.next_offset, offset + decoded.len() as u64); + assert_eq!(chunk.encoding, LargeValueEncoding::Base64); + assert_eq!(chunk.content_type, "application/octet-stream"); + rebuilt.extend(decoded); + offset = chunk.next_offset; + if chunk.eof { + break; + } + } + assert_eq!(rebuilt, source); + assert_eq!( + store.read_chunk("execution-1", token(&preview), 0, 0), + Err(LargeValueError::InvalidLimit) + ); + assert_eq!( + store.read_chunk("execution-1", token(&preview), 14, 1), + Err(LargeValueError::InvalidRange { + offset: 14, + length: 13, + }) + ); + let eof = store + .read_chunk("execution-1", token(&preview), 13, 1) + .expect("offset at the end should be valid"); + assert!(eof.eof); + assert!(eof.value.is_empty()); +} + +#[test] +fn explicit_token_and_owner_cleanup_leave_no_retained_bytes() { + let mut limits = config(); + limits.preview_bytes = 1; + let store = LargeValueStore::new(limits); + let first = store + .retain_text("execution-1", "first".to_owned()) + .expect("first should retain"); + let second = store + .retain_binary("execution-1", vec![1, 2, 3, 4]) + .expect("second should retain"); + let other = store + .retain_text("execution-2", "other".to_owned()) + .expect("other should retain"); + + assert_eq!( + store.remove_token("execution-2", token(&first)), + Err(LargeValueError::OwnerMismatch) + ); + store + .remove_token("execution-1", token(&first)) + .expect("owner should remove token"); + assert_eq!( + store.read_chunk("execution-1", token(&first), 0, 1), + Err(LargeValueError::NotFound) + ); + assert_eq!(store.remove_owner("execution-1"), 1); + assert_eq!( + store.read_chunk("execution-1", token(&second), 0, 1), + Err(LargeValueError::NotFound) + ); + assert_eq!(store.stats().entries, 1); + assert_eq!(store.remove_owner("execution-2"), 1); + assert_eq!(store.stats().entries, 0); + assert_eq!(store.stats().total_bytes, 0); + assert_eq!( + store.remove_token("execution-2", token(&other)), + Err(LargeValueError::NotFound) + ); +} diff --git a/crates/chat2db-core/tests/native_mysql_console_docker.rs b/crates/chat2db-core/tests/native_mysql_console_docker.rs new file mode 100644 index 0000000..8c5c0b7 --- /dev/null +++ b/crates/chat2db-core/tests/native_mysql_console_docker.rs @@ -0,0 +1,764 @@ +use std::{panic::AssertUnwindSafe, time::Duration}; + +use base64::{Engine as _, engine::general_purpose::STANDARD}; +use chat2db_contract::{ + ComponentState, CreateDatasourceRequest, DatasourceConnection, DatasourceConnectionProperty, + JdbcValue, +}; +use chat2db_core::{ + Application, MysqlConsoleCancellation, MysqlConsoleRequest, MysqlConsoleResult, RuntimeConfig, + RuntimeHost, +}; +use chat2db_java_bridge::{EngineCommand, EngineConfig}; +use futures_util::FutureExt as _; +use mysql_async::{Conn, Opts, OptsBuilder, prelude::Queryable}; +use tempfile::TempDir; +use uuid::Uuid; + +const MYSQL_SLEEP_SQL: &str = "SELECT SLEEP(30)"; +const MYSQL_WAIT_TIMEOUT: Duration = Duration::from_secs(15); + +struct MysqlTestConfig { + host: String, + port: u16, + user: String, + password: String, +} + +impl MysqlTestConfig { + fn from_environment() -> Self { + let host = required_env("MYSQL_TEST_HOST"); + assert!( + !host.trim().is_empty() + && !host.chars().any(char::is_control) + && !host.contains(['/', '?', '#']), + "MYSQL_TEST_HOST is invalid" + ); + let port = required_env("MYSQL_TEST_PORT") + .parse::() + .expect("MYSQL_TEST_PORT must be a TCP port"); + assert_ne!(port, 0, "MYSQL_TEST_PORT cannot be zero"); + let user = required_env("MYSQL_TEST_USER"); + assert!(!user.is_empty(), "MYSQL_TEST_USER cannot be empty"); + Self { + host, + port, + user, + password: required_env("MYSQL_TEST_PASSWORD"), + } + } + + fn native_options(&self) -> Opts { + OptsBuilder::default() + .ip_or_hostname(self.host.clone()) + .tcp_port(self.port) + .user(Some(self.user.clone())) + .pass(Some(self.password.clone())) + .prefer_socket(Some(false)) + .into() + } + + fn connection(&self, database_name: &str) -> DatasourceConnection { + let host = if self.host.contains(':') + && !(self.host.starts_with('[') && self.host.ends_with(']')) + { + format!("[{}]", self.host) + } else { + self.host.clone() + }; + DatasourceConnection { + jdbc_url: format!( + "jdbc:mysql://{host}:{}/{database_name}?useSSL=false&serverTimezone=UTC", + self.port + ), + properties: vec![ + DatasourceConnectionProperty { + key: "user".to_owned(), + value: self.user.clone(), + sensitive: false, + }, + DatasourceConnectionProperty { + key: "password".to_owned(), + value: self.password.clone(), + sensitive: true, + }, + ], + read_only: false, + } + } +} + +#[tokio::test] +#[ignore = "requires MYSQL_TEST_* variables and a real MySQL server"] +async fn native_mysql_console_matches_community_execution_semantics() { + let config = MysqlTestConfig::from_environment(); + let suffix = Uuid::new_v4().simple().to_string(); + let database_name = format!("chat2db_console_it_{suffix}"); + let table_name = format!("console_items_{}", &suffix[..8]); + let procedure_name = format!("console_results_{}", &suffix[..8]); + provision_database(&config, &database_name).await; + + let verification = AssertUnwindSafe(verify_console( + &config, + &database_name, + &table_name, + &procedure_name, + )) + .catch_unwind() + .await; + let cleanup = cleanup_database(&config, &database_name).await; + if let Err(payload) = verification { + if let Err(error) = cleanup { + eprintln!("native MySQL Console cleanup also failed: {error}"); + } + std::panic::resume_unwind(payload); + } + cleanup.expect("native MySQL Console fixture must be removed"); +} + +async fn verify_console( + config: &MysqlTestConfig, + database_name: &str, + table_name: &str, + procedure_name: &str, +) { + let directory = TempDir::new().expect("temporary native MySQL Console runtime"); + let missing_java = directory.path().join("missing-java"); + let runtime = RuntimeConfig::new(EngineConfig::new(EngineCommand::new(missing_java))) + .with_data_dir(directory.path().join("data")) + .with_vault_master_key_base64(STANDARD.encode([0x6e; 32])); + let mut host = RuntimeHost::open(runtime) + .await + .expect("native MySQL Console runtime must open without Java"); + let application = host.application(); + let datasource = application + .create_datasource(CreateDatasourceRequest { + name: "Native MySQL Console Docker".to_owned(), + driver_id: "mysql".to_owned(), + connection: Some(config.connection(database_name)), + }) + .await + .expect("native MySQL datasource must persist without a JDBC driver pack"); + assert_java_dormant(&application); + + verify_ddl_and_dml(&application, &datasource.id, database_name, table_name).await; + create_multi_result_procedure(&application, &datasource.id, database_name, procedure_name) + .await; + verify_multi_result_sets(&application, &datasource.id, database_name, procedure_name).await; + verify_error_continue(&application, &datasource.id, database_name, table_name).await; + verify_transactions(&application, &datasource.id, database_name, table_name).await; + verify_large_value(&application, &datasource.id, database_name, table_name).await; + verify_console_options(&application, &datasource.id, database_name).await; + verify_read_only(config, &application, database_name, table_name).await; + verify_cancellation(config, &application, &datasource.id, database_name).await; + assert_java_dormant(&application); + + host.shutdown() + .await + .expect("native-only runtime must shut down cleanly"); +} + +async fn verify_ddl_and_dml( + application: &Application, + datasource_id: &str, + database_name: &str, + table_name: &str, +) { + let ddl = execute( + application, + request( + datasource_id, + database_name, + format!( + "CREATE TABLE `{table_name}` (\ + `id` BIGINT NOT NULL, `label` VARCHAR(128) NOT NULL, \ + `score` INT NOT NULL, PRIMARY KEY (`id`)\ + ) ENGINE=InnoDB" + ), + false, + ), + ) + .await; + assert_single_success(&ddl, 0); + + let insert = execute( + application, + request( + datasource_id, + database_name, + format!( + "INSERT INTO `{table_name}` (`id`, `label`, `score`) \ + VALUES (1, 'first', 10), (2, 'second', 20)" + ), + false, + ), + ) + .await; + assert_single_success(&insert, 2); + + let update = execute( + application, + request( + datasource_id, + database_name, + format!("UPDATE `{table_name}` SET `score` = 11 WHERE `id` = 1"), + false, + ), + ) + .await; + assert_single_success(&update, 1); + + let delete = execute( + application, + request( + datasource_id, + database_name, + format!("DELETE FROM `{table_name}` WHERE `id` = 2"), + false, + ), + ) + .await; + assert_single_success(&delete, 1); + + let multi = execute( + application, + request( + datasource_id, + database_name, + format!( + "INSERT INTO `{table_name}` VALUES (3, 'third', 30); \ + UPDATE `{table_name}` SET `score` = 31 WHERE `id` = 3; \ + SELECT `id`, `label`, `score` FROM `{table_name}` ORDER BY `id`" + ), + false, + ), + ) + .await; + assert_eq!( + multi + .iter() + .map(|result| result.statement_sequence) + .collect::>(), + vec![1, 2, 3] + ); + assert!(multi.iter().all(|result| result.success)); + let selected = multi.last().expect("multi-statement SELECT result"); + assert_eq!(selected.result_set_id, Some(1)); + assert_eq!(selected.row_count, 2); + assert_eq!(scalar_text(&selected.rows[0].values[1]), "first"); + assert_eq!(scalar_text(&selected.rows[1].values[1]), "third"); + assert_eq!(scalar_text(&selected.rows[1].values[2]), "31"); +} + +async fn verify_large_value( + application: &Application, + datasource_id: &str, + database_name: &str, + table_name: &str, +) { + let alter = execute( + application, + request( + datasource_id, + database_name, + format!("ALTER TABLE `{table_name}` ADD COLUMN `payload` LONGTEXT NULL"), + false, + ), + ) + .await; + assert_single_success(&alter, 0); + + let update = execute( + application, + request( + datasource_id, + database_name, + format!( + "UPDATE `{table_name}` SET `payload` = REPEAT('large-value-', 500000) \ + WHERE `id` = 1" + ), + false, + ), + ) + .await; + assert_single_success(&update, 1); + + let result = execute( + application, + request( + datasource_id, + database_name, + format!("SELECT `payload` FROM `{table_name}` WHERE `id` = 1"), + false, + ), + ) + .await; + let payload = statement_value(result.first().expect("large-value SELECT result")); + assert_eq!(payload.len(), 6_000_000); + assert!(payload.starts_with("large-value-")); + assert!(payload.ends_with("large-value-")); +} + +async fn verify_multi_result_sets( + application: &Application, + datasource_id: &str, + database_name: &str, + procedure_name: &str, +) { + let all_results = execute( + application, + request( + datasource_id, + database_name, + format!("CALL `{procedure_name}`()"), + false, + ), + ) + .await; + let tabular = all_results + .iter() + .filter(|result| result.result_set_id.is_some()) + .collect::>(); + assert_eq!(tabular.len(), 2); + assert_eq!(tabular[0].result_set_id, Some(1)); + assert_eq!(scalar_text(&tabular[0].rows[0].values[0]), "11"); + assert_eq!(tabular[1].result_set_id, Some(2)); + assert_eq!(scalar_text(&tabular[1].rows[0].values[0]), "22"); + + let mut selected_request = request( + datasource_id, + database_name, + format!("CALL `{procedure_name}`()"), + false, + ); + selected_request.result_set_id = Some(2); + let selected = execute(application, selected_request).await; + let selected_tabular = selected + .iter() + .filter(|result| result.result_set_id.is_some()) + .collect::>(); + assert_eq!(selected_tabular.len(), 1); + assert_eq!(selected_tabular[0].result_set_id, Some(2)); + assert_eq!(scalar_text(&selected_tabular[0].rows[0].values[0]), "22"); +} + +async fn verify_error_continue( + application: &Application, + datasource_id: &str, + database_name: &str, + table_name: &str, +) { + let stop = execute( + application, + request( + datasource_id, + database_name, + format!( + "INSERT INTO `{table_name}` VALUES (100, 'stop-first', 100); \ + INSERT INTO `{table_name}` VALUES (100, 'duplicate', 100); \ + INSERT INTO `{table_name}` VALUES (101, 'must-not-run', 101)" + ), + false, + ), + ) + .await; + assert_eq!(stop.len(), 2); + assert!(stop[0].success); + assert!(!stop[1].success); + assert_eq!(stop[0].statement_sequence, 1); + assert_eq!(stop[1].statement_sequence, 2); + assert!(stop[1].error.is_some()); + assert_eq!( + query_count(application, datasource_id, database_name, table_name, 101).await, + "0" + ); + + let proceed = execute( + application, + request( + datasource_id, + database_name, + format!( + "INSERT INTO `{table_name}` VALUES (200, 'continue-first', 200); \ + INSERT INTO `{table_name}` VALUES (200, 'duplicate', 200); \ + INSERT INTO `{table_name}` VALUES (201, 'did-run', 201)" + ), + true, + ), + ) + .await; + assert_eq!(proceed.len(), 3); + assert!(proceed[0].success); + assert!(!proceed[1].success); + assert!(proceed[2].success); + assert_eq!( + proceed + .iter() + .map(|result| result.statement_sequence) + .collect::>(), + vec![1, 2, 3] + ); + assert_eq!( + query_count(application, datasource_id, database_name, table_name, 201).await, + "1" + ); +} + +async fn verify_transactions( + application: &Application, + datasource_id: &str, + database_name: &str, + table_name: &str, +) { + let rollback = execute( + application, + request( + datasource_id, + database_name, + format!( + "BEGIN; INSERT INTO `{table_name}` VALUES (300, 'rollback', 300); \ + ROLLBACK; SELECT COUNT(*) FROM `{table_name}` WHERE `id` = 300" + ), + false, + ), + ) + .await; + assert!(rollback.iter().all(|result| result.success)); + assert_eq!(rollback.last().map(statement_value), Some("0")); + + let commit = execute( + application, + request( + datasource_id, + database_name, + format!( + "BEGIN; INSERT INTO `{table_name}` VALUES (301, 'commit', 301); \ + COMMIT; SELECT COUNT(*) FROM `{table_name}` WHERE `id` = 301" + ), + false, + ), + ) + .await; + assert!(commit.iter().all(|result| result.success)); + assert_eq!(commit.last().map(statement_value), Some("1")); + assert_eq!( + query_count(application, datasource_id, database_name, table_name, 301).await, + "1" + ); +} + +async fn verify_console_options( + application: &Application, + datasource_id: &str, + database_name: &str, +) { + let mut single = request( + datasource_id, + database_name, + "SELECT 7 AS `first_value`; SELECT 8 AS `second_value`".to_owned(), + false, + ); + single.single = true; + let single_results = execute(application, single).await; + let tabular = single_results + .iter() + .filter(|result| result.result_set_id.is_some()) + .collect::>(); + assert_eq!(tabular.len(), 2); + assert!(tabular.iter().all(|result| result.statement_sequence == 1)); + assert_eq!(tabular[0].result_set_id, Some(1)); + assert_eq!(tabular[1].result_set_id, Some(2)); + assert_eq!(statement_value(tabular[0]), "7"); + assert_eq!(statement_value(tabular[1]), "8"); + + let mut explain = request( + datasource_id, + database_name, + "SELECT 1 AS `value`".to_owned(), + false, + ); + explain.explain = true; + let explained = execute(application, explain).await; + assert_eq!(explained.len(), 1); + assert!(explained[0].success); + assert!(explained[0].sql.starts_with("EXPLAIN SELECT 1")); + assert!( + explained[0] + .columns + .iter() + .any(|column| column.label.eq_ignore_ascii_case("select_type")) + ); + + let series_sql = "WITH RECURSIVE `seq` (`n`) AS (\ + SELECT 1 UNION ALL SELECT `n` + 1 FROM `seq` WHERE `n` < 25\ + ) SELECT `n` FROM `seq` ORDER BY `n`"; + let mut paged = request(datasource_id, database_name, series_sql.to_owned(), false); + paged.page_no = 2; + paged.page_size = 5; + let page = execute(application, paged.clone()).await; + assert_eq!(page[0].rows.len(), 5); + assert_eq!(scalar_text(&page[0].rows[0].values[0]), "6"); + assert!(page[0].has_more); + + paged.page_size_all = true; + let all_rows = execute(application, paged).await; + assert_eq!(all_rows[0].rows.len(), 25); + assert_eq!(scalar_text(&all_rows[0].rows[0].values[0]), "1"); + assert_eq!(scalar_text(&all_rows[0].rows[24].values[0]), "25"); + assert!(!all_rows[0].has_more); +} + +async fn verify_read_only( + config: &MysqlTestConfig, + application: &Application, + database_name: &str, + table_name: &str, +) { + let mut connection = config.connection(database_name); + connection.read_only = true; + let datasource = application + .create_datasource(CreateDatasourceRequest { + name: "Native MySQL Read Only Docker".to_owned(), + driver_id: "mysql".to_owned(), + connection: Some(connection), + }) + .await + .expect("read-only native MySQL datasource must persist"); + + let inspected = execute( + application, + request( + &datasource.id, + database_name, + format!("SELECT COUNT(*) FROM `{table_name}`"), + false, + ), + ) + .await; + assert!(inspected[0].success); + + let error = application + .execute_mysql_console( + request( + &datasource.id, + database_name, + format!("INSERT INTO `{table_name}` (`id`, `label`, `score`) VALUES (900, 'blocked', 900)"), + false, + ), + MysqlConsoleCancellation::new(), + ) + .await + .expect_err("read-only datasource must reject writes"); + assert_eq!(error.api_error().code, "datasource_read_only"); + assert_eq!( + query_count(application, &datasource.id, database_name, table_name, 900).await, + "0" + ); +} + +async fn verify_cancellation( + config: &MysqlTestConfig, + application: &Application, + datasource_id: &str, + database_name: &str, +) { + let cancellation = MysqlConsoleCancellation::new(); + let execution_cancellation = cancellation.clone(); + let execution_application = application.clone(); + let execution_request = request( + datasource_id, + database_name, + MYSQL_SLEEP_SQL.to_owned(), + false, + ); + let execution = tokio::spawn(async move { + execution_application + .execute_mysql_console(execution_request, execution_cancellation) + .await + }); + + wait_for_active_sleep(config, database_name).await; + assert!(cancellation.cancel(Some("Docker integration test cancellation".to_owned()))); + let error = tokio::time::timeout(MYSQL_WAIT_TIMEOUT, execution) + .await + .expect("native MySQL Console cancellation must finish before timeout") + .expect("native MySQL Console cancellation task must not panic") + .expect_err("cancelled native MySQL Console execution must fail"); + assert_eq!(error.api_error().code, "mysql_console_cancelled"); +} + +async fn query_count( + application: &Application, + datasource_id: &str, + database_name: &str, + table_name: &str, + id: u64, +) -> String { + let results = execute( + application, + request( + datasource_id, + database_name, + format!("SELECT COUNT(*) FROM `{table_name}` WHERE `id` = {id}"), + false, + ), + ) + .await; + statement_value(results.first().expect("COUNT result")).to_owned() +} + +async fn execute( + application: &Application, + request: MysqlConsoleRequest, +) -> Vec { + application + .execute_mysql_console(request, MysqlConsoleCancellation::new()) + .await + .expect("native MySQL Console request must complete") +} + +fn request( + datasource_id: &str, + database_name: &str, + sql: String, + error_continue: bool, +) -> MysqlConsoleRequest { + MysqlConsoleRequest { + datasource_id: datasource_id.to_owned(), + database_name: database_name.to_owned(), + sql, + page_no: 1, + page_size: 100, + result_set_id: None, + single: false, + page_size_all: false, + explain: false, + error_continue, + } +} + +fn assert_single_success(results: &[MysqlConsoleResult], update_count: u64) { + assert_eq!(results.len(), 1); + assert!(results[0].success); + assert_eq!(results[0].statement_sequence, 1); + assert_eq!(results[0].update_count, update_count); + assert!(results[0].error.is_none()); +} + +fn statement_value(result: &MysqlConsoleResult) -> &str { + let row = result.rows.first().expect("result must contain one row"); + let value = row.values.first().expect("result must contain one column"); + scalar_text(value) +} + +fn scalar_text(value: &JdbcValue) -> &str { + match value { + JdbcValue::SignedInteger { value } + | JdbcValue::UnsignedInteger { value } + | JdbcValue::Float32 { value } + | JdbcValue::Float64 { value } + | JdbcValue::Decimal { value } + | JdbcValue::Text { value } + | JdbcValue::Binary { value } + | JdbcValue::Date { value } + | JdbcValue::Time { value } + | JdbcValue::Timestamp { value } + | JdbcValue::TimestampWithTimeZone { value } + | JdbcValue::Json { value } + | JdbcValue::Uuid { value } => value, + JdbcValue::Opaque { display_value, .. } => display_value, + JdbcValue::Null | JdbcValue::Boolean { .. } => { + panic!("expected a scalar value with a textual representation") + } + } +} + +async fn provision_database(config: &MysqlTestConfig, database_name: &str) { + let mut conn = Conn::new(config.native_options()) + .await + .expect("native MySQL Console fixture connection must open"); + conn.query_drop(format!( + "CREATE DATABASE `{database_name}` CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci" + )) + .await + .expect("native MySQL Console fixture database must create"); + conn.disconnect() + .await + .expect("native MySQL Console fixture connection must close"); +} + +async fn create_multi_result_procedure( + application: &Application, + datasource_id: &str, + database_name: &str, + procedure_name: &str, +) { + let results = execute( + application, + request( + datasource_id, + database_name, + format!( + "DELIMITER $$\nCREATE PROCEDURE `{procedure_name}`()\n\ + BEGIN\nSELECT 11 AS `first_value`; SELECT 22 AS `second_value`;\nEND$$\n\ + DELIMITER ;" + ), + false, + ), + ) + .await; + assert_single_success(&results, 0); +} + +async fn wait_for_active_sleep(config: &MysqlTestConfig, database_name: &str) { + let mut conn = Conn::new(config.native_options()) + .await + .expect("native MySQL process-list probe must connect"); + tokio::time::timeout(MYSQL_WAIT_TIMEOUT, async { + loop { + let active = conn + .exec_first::( + "SELECT COUNT(*) FROM information_schema.PROCESSLIST \ + WHERE DB = ? AND INFO LIKE 'SELECT SLEEP(30)%'", + (database_name,), + ) + .await + .expect("native MySQL process-list probe must succeed") + .unwrap_or_default(); + if active > 0 { + break; + } + tokio::time::sleep(Duration::from_millis(25)).await; + } + }) + .await + .expect("native MySQL sleep query must become active before cancellation"); + conn.disconnect() + .await + .expect("native MySQL process-list probe must disconnect"); +} + +async fn cleanup_database(config: &MysqlTestConfig, database_name: &str) -> Result<(), String> { + let mut conn = Conn::new(config.native_options()) + .await + .map_err(|error| error.to_string())?; + conn.query_drop(format!("DROP DATABASE IF EXISTS `{database_name}`")) + .await + .map_err(|error| error.to_string())?; + conn.disconnect().await.map_err(|error| error.to_string()) +} + +fn assert_java_dormant(application: &Application) { + let engine = application + .health() + .components + .into_iter() + .find(|component| component.id == "database-engine") + .expect("database engine health must be present"); + assert_eq!(engine.state, ComponentState::Ready); + assert_eq!(engine.detail, "Available on demand; Java is not running"); +} + +fn required_env(name: &str) -> String { + std::env::var(name).unwrap_or_else(|_| panic!("{name} must be configured")) +} diff --git a/crates/chat2db-storage/migrations/004_operation_log.sql b/crates/chat2db-storage/migrations/004_operation_log.sql new file mode 100644 index 0000000..a64c400 --- /dev/null +++ b/crates/chat2db-storage/migrations/004_operation_log.sql @@ -0,0 +1,35 @@ +BEGIN IMMEDIATE; + +CREATE TABLE operation_logs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT, + data_source_id TEXT, + data_source_name TEXT, + connectable INTEGER CHECK (connectable IS NULL OR connectable IN (0, 1)), + database_name TEXT, + database_type TEXT, + ddl TEXT NOT NULL, + status TEXT NOT NULL, + operation_rows INTEGER CHECK (operation_rows IS NULL OR operation_rows >= 0), + use_time INTEGER CHECK (use_time IS NULL OR use_time >= 0), + extend_info TEXT, + schema_name TEXT, + organization_id INTEGER, + user_name TEXT, + more INTEGER NOT NULL DEFAULT 0 CHECK (more IN (0, 1)), + operation_type TEXT NOT NULL, + created_at_ms INTEGER NOT NULL, + updated_at_ms INTEGER NOT NULL CHECK (updated_at_ms >= created_at_ms) +) STRICT; + +CREATE INDEX operation_logs_created_idx + ON operation_logs (created_at_ms DESC, id DESC); + +CREATE INDEX operation_logs_scope_idx + ON operation_logs ( + operation_type, data_source_id, database_name, schema_name, + created_at_ms DESC, id DESC + ); + +PRAGMA user_version = 4; +COMMIT; diff --git a/crates/chat2db-storage/src/error.rs b/crates/chat2db-storage/src/error.rs index 7908e23..e7ba7eb 100644 --- a/crates/chat2db-storage/src/error.rs +++ b/crates/chat2db-storage/src/error.rs @@ -68,6 +68,9 @@ pub enum StorageError { /// A saved Console field or page request violates the durable contract. #[error("invalid saved Console: {0}")] InvalidSavedConsole(&'static str), + /// An operation-log field or page request violates the durable contract. + #[error("invalid operation log: {0}")] + InvalidOperationLog(&'static str), /// The requested provider profile does not exist. #[error("provider profile not found: {0}")] ProviderNotFound(String), diff --git a/crates/chat2db-storage/src/lib.rs b/crates/chat2db-storage/src/lib.rs index 039e399..7f5f59f 100644 --- a/crates/chat2db-storage/src/lib.rs +++ b/crates/chat2db-storage/src/lib.rs @@ -3,6 +3,7 @@ mod agent; mod datasource; mod error; +mod operation_log; mod provider; mod result_store; mod saved_console; @@ -36,6 +37,9 @@ pub use datasource::{ CreateDatasource, DatasourceRecord, SecretChange, SecretCleanupReport, UpdateDatasource, }; pub use error::StorageError; +pub use operation_log::{ + CreateOperationLog, OperationLogListQuery, OperationLogPage, OperationLogRecord, +}; pub use provider::{ CreateProviderProfile, ProviderKind, ProviderProfileRecord, UpdateProviderProfile, }; @@ -55,7 +59,7 @@ pub use vault::OsSecretVault; const DATABASE_FILE: &str = "chat2db.sqlite3"; const LOCK_FILE: &str = ".chat2db.lock"; const RESULTS_DIRECTORY: &str = "results"; -const CURRENT_SCHEMA_VERSION: i64 = 3; +const CURRENT_SCHEMA_VERSION: i64 = 4; #[cfg(test)] #[derive(Clone, Copy)] @@ -383,6 +387,13 @@ fn migrate(connection: &Connection) -> Result<(), StorageError> { connection, include_str!("../migrations/003_saved_console.sql"), )?; + version = 3; + } + if version == 3 { + apply_migration( + connection, + include_str!("../migrations/004_operation_log.sql"), + )?; } Ok(()) } @@ -525,7 +536,7 @@ mod tests { let synchronous: i64 = connection .pragma_query_value(None, "synchronous", |row| row.get(0)) .expect("synchronous reads"); - assert_eq!(version, 3); + assert_eq!(version, 4); assert_eq!(foreign_keys, 1); assert_eq!(journal_mode.to_ascii_lowercase(), "wal"); assert_eq!(synchronous, 2); @@ -544,22 +555,22 @@ mod tests { let database = directory.path().join(DATABASE_FILE); Connection::open(&database) .expect("database opens") - .execute_batch("PRAGMA user_version = 4") + .execute_batch("PRAGMA user_version = 5") .expect("test version updates"); let error = Storage::open(directory.path(), vault()).expect_err("newer schema must fail"); assert!(matches!( error, StorageError::UnsupportedSchema { - found: 4, - supported: 3 + found: 5, + supported: 4 } )); let version: i64 = Connection::open(database) .expect("database opens") .pragma_query_value(None, "user_version", |row| row.get(0)) .expect("schema version reads"); - assert_eq!(version, 4); + assert_eq!(version, 5); } #[test] @@ -593,7 +604,7 @@ mod tests { |row| row.get(0), ) .expect("provider table count reads"); - assert_eq!(version, 3); + assert_eq!(version, 4); assert_eq!(provider_table, 1); assert!( storage @@ -637,7 +648,7 @@ mod tests { |row| row.get(0), ) .expect("saved Console table count reads"); - assert_eq!(version, 3); + assert_eq!(version, 4); assert_eq!(saved_console_table, 1); assert!( storage @@ -697,6 +708,103 @@ mod tests { assert_eq!(prior_indexes, 0); } + #[test] + fn version_three_upgrades_atomically_and_preserves_existing_state() { + let directory = TempDir::new().expect("temp dir"); + let database = directory.path().join(DATABASE_FILE); + let connection = Connection::open(&database).expect("database opens"); + connection + .execute_batch(include_str!("../migrations/001_initial.sql")) + .expect("version one schema creates"); + connection + .execute_batch(include_str!("../migrations/002_agent.sql")) + .expect("version two schema creates"); + connection + .execute_batch(include_str!("../migrations/003_saved_console.sql")) + .expect("version three schema creates"); + connection + .execute( + "INSERT INTO saved_consoles ( + id, name, ddl, status, tab_opened, operation_type, + created_at_ms, updated_at_ms + ) VALUES (42, 'Existing', 'SELECT 1', 'DRAFT', 'y', 'console', 1, 1)", + [], + ) + .expect("version three state inserts"); + drop(connection); + + let storage = Storage::open(directory.path(), vault()).expect("version three upgrades"); + let connection = storage.connection().expect("connection opens"); + let version: i64 = connection + .pragma_query_value(None, "user_version", |row| row.get(0)) + .expect("schema version reads"); + let operation_log_table: i64 = connection + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type = 'table' AND name = 'operation_logs'", + [], + |row| row.get(0), + ) + .expect("operation log table count reads"); + assert_eq!(version, 4); + assert_eq!(operation_log_table, 1); + assert!( + storage + .get_saved_console(42) + .expect("saved Console reads") + .is_some() + ); + } + + #[test] + fn failed_version_four_upgrade_rolls_back_operation_log_schema() { + let directory = TempDir::new().expect("temp dir"); + let database = directory.path().join(DATABASE_FILE); + let connection = Connection::open(&database).expect("database opens"); + connection + .execute_batch(include_str!("../migrations/001_initial.sql")) + .expect("version one schema creates"); + connection + .execute_batch(include_str!("../migrations/002_agent.sql")) + .expect("version two schema creates"); + connection + .execute_batch(include_str!("../migrations/003_saved_console.sql")) + .expect("version three schema creates"); + connection + .execute_batch( + "CREATE TABLE migration_sentinel (value INTEGER); + CREATE INDEX operation_logs_scope_idx + ON migration_sentinel (value);", + ) + .expect("conflicting index creates"); + drop(connection); + + Storage::open(directory.path(), vault()).expect_err("version four upgrade must fail"); + let connection = Connection::open(database).expect("database reopens"); + let version: i64 = connection + .pragma_query_value(None, "user_version", |row| row.get(0)) + .expect("schema version reads"); + let operation_log_tables: i64 = connection + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type = 'table' AND name = 'operation_logs'", + [], + |row| row.get(0), + ) + .expect("operation log table count reads"); + let prior_indexes: i64 = connection + .query_row( + "SELECT COUNT(*) FROM sqlite_master + WHERE type = 'index' AND name = 'operation_logs_created_idx'", + [], + |row| row.get(0), + ) + .expect("partial operation log index count reads"); + assert_eq!(version, 3); + assert_eq!(operation_log_tables, 0); + assert_eq!(prior_indexes, 0); + } + #[test] fn failed_version_two_upgrade_rolls_back_every_new_agent_table() { let directory = TempDir::new().expect("temp dir"); diff --git a/crates/chat2db-storage/src/operation_log.rs b/crates/chat2db-storage/src/operation_log.rs new file mode 100644 index 0000000..2395bb4 --- /dev/null +++ b/crates/chat2db-storage/src/operation_log.rs @@ -0,0 +1,773 @@ +use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params}; + +use crate::{Storage, StorageError, now_millis}; + +const MAX_NAME_BYTES: usize = 512; +const MAX_DATASOURCE_ID_BYTES: usize = 512; +const MAX_SCOPE_BYTES: usize = 1_024; +const MAX_DATABASE_TYPE_BYTES: usize = 255; +const MAX_STATE_BYTES: usize = 255; +const MAX_DDL_BYTES: usize = 16 * 1024 * 1024; +const MAX_EXTEND_INFO_BYTES: usize = 16 * 1024 * 1024; +const MAX_SEARCH_KEY_BYTES: usize = 256; +const MAX_PAGE_SIZE: u32 = 1_000; + +const OPERATION_LOG_COLUMNS: &str = "id, name, data_source_id, data_source_name, connectable,\ + database_name, database_type, ddl, status, operation_rows, use_time, extend_info,\ + schema_name, organization_id, user_name, more, operation_type, created_at_ms, updated_at_ms"; + +/// Fields captured for one durable Community SQL execution-history entry. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct CreateOperationLog { + pub name: Option, + /// Opaque Rust datasource id corresponding to Community's `dataSourceId`. + pub data_source_id: Option, + pub data_source_name: Option, + pub connectable: Option, + pub database_name: Option, + /// Community database type code, exposed as the historical `type` field. + pub database_type: Option, + pub ddl: String, + pub status: String, + pub operation_rows: Option, + /// Execution duration in milliseconds, exposed as the historical `useTime` field. + pub use_time: Option, + /// Opaque JSON-like metadata retained without interpretation. + pub extend_info: Option, + pub schema_name: Option, + pub organization_id: Option, + pub user_name: Option, + /// Whether the list representation's SQL text is truncated. + pub more: bool, + /// Community history discriminator, normally `SQL_EXECUTE` or `SQL_AUDIT`. + pub operation_type: String, +} + +/// Community-compatible filters and stable paging controls for execution history. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OperationLogListQuery { + pub data_source_id: Option, + pub database_name: Option, + pub schema_name: Option, + pub operation_type: Option, + pub search_key: Option, + pub page_no: u32, + pub page_size: u32, +} + +impl Default for OperationLogListQuery { + fn default() -> Self { + Self { + data_source_id: None, + database_name: None, + schema_name: None, + operation_type: None, + search_key: None, + page_no: 1, + page_size: 20, + } + } +} + +/// One durable Community SQL execution-history entry. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OperationLogRecord { + pub id: i64, + pub name: Option, + pub data_source_id: Option, + pub data_source_name: Option, + pub connectable: Option, + pub database_name: Option, + pub database_type: Option, + pub ddl: String, + pub status: String, + pub operation_rows: Option, + pub use_time: Option, + pub extend_info: Option, + pub schema_name: Option, + pub organization_id: Option, + pub user_name: Option, + pub more: bool, + pub operation_type: String, + /// Creation time as Unix epoch milliseconds. + pub created_at_ms: i64, + /// Last update time as Unix epoch milliseconds. + pub updated_at_ms: i64, +} + +/// One newest-first stable page of durable execution history. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct OperationLogPage { + pub records: Vec, + pub total: u64, + pub page_no: u32, + pub page_size: u32, +} + +impl Storage { + /// Creates one execution-history entry and returns its durable record. + /// + /// # Errors + /// + /// Returns validation or `SQLite` failures. + pub fn create_operation_log( + &self, + input: CreateOperationLog, + ) -> Result { + validate_create(&input)?; + let CreateOperationLog { + name, + data_source_id, + data_source_name, + connectable, + database_name, + database_type, + ddl, + status, + operation_rows, + use_time, + extend_info, + schema_name, + organization_id, + user_name, + more, + operation_type, + } = input; + let timestamp = now_millis()?; + let mut connection = self.connection()?; + let transaction = connection.transaction_with_behavior(TransactionBehavior::Immediate)?; + transaction.execute( + "INSERT INTO operation_logs ( + name, data_source_id, data_source_name, connectable, database_name, + database_type, ddl, status, operation_rows, use_time, extend_info, + schema_name, organization_id, user_name, more, operation_type, + created_at_ms, updated_at_ms + ) VALUES ( + ?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14, + ?15, ?16, ?17, ?17 + )", + params![ + name, + data_source_id, + data_source_name, + connectable, + database_name, + database_type, + ddl, + status, + operation_rows, + use_time, + extend_info, + schema_name, + organization_id, + user_name, + more, + operation_type, + timestamp, + ], + )?; + let id = transaction.last_insert_rowid(); + let record = load_operation_log(&transaction, id)?.ok_or_else(|| { + StorageError::Integrity("operation log disappeared before create commit".to_owned()) + })?; + transaction.commit()?; + Ok(record) + } + + /// Loads one execution-history entry by its numeric Community id. + /// + /// # Errors + /// + /// Returns `SQLite` or persisted-data validation failures. + pub fn get_operation_log(&self, id: i64) -> Result, StorageError> { + if id <= 0 { + return Ok(None); + } + load_operation_log(&self.connection()?, id) + } + + /// Lists execution history newest-first with exact scope filters and stable paging. + /// + /// `search_key` is a case-insensitive literal substring match over SQL and + /// the user-visible names and state stored with the entry. + /// + /// # Errors + /// + /// Returns validation, `SQLite`, numeric-range, or persisted-data failures. + pub fn list_operation_logs( + &self, + query: &OperationLogListQuery, + ) -> Result { + validate_list_query(query)?; + let offset = u64::from(query.page_no - 1) + .checked_mul(u64::from(query.page_size)) + .ok_or(StorageError::NumericRange("operation log page offset"))?; + let offset = i64::try_from(offset) + .map_err(|_| StorageError::NumericRange("operation log page offset"))?; + let limit = i64::from(query.page_size); + let search_pattern = query + .search_key + .as_deref() + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(like_pattern); + let connection = self.connection()?; + + let total: i64 = connection.query_row( + "SELECT COUNT(*) FROM operation_logs + WHERE (?1 IS NULL OR data_source_id = ?1) + AND (?2 IS NULL OR database_name = ?2) + AND (?3 IS NULL OR schema_name = ?3) + AND (?4 IS NULL OR operation_type = ?4) + AND ( + ?5 IS NULL + OR COALESCE(name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(data_source_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(database_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(schema_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR ddl COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR status COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(database_type, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(user_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + )", + params![ + query.data_source_id.as_deref(), + query.database_name.as_deref(), + query.schema_name.as_deref(), + query.operation_type.as_deref(), + search_pattern.as_deref(), + ], + |row| row.get(0), + )?; + + let sql = format!( + "SELECT {OPERATION_LOG_COLUMNS} FROM operation_logs + WHERE (?1 IS NULL OR data_source_id = ?1) + AND (?2 IS NULL OR database_name = ?2) + AND (?3 IS NULL OR schema_name = ?3) + AND (?4 IS NULL OR operation_type = ?4) + AND ( + ?5 IS NULL + OR COALESCE(name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(data_source_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(database_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(schema_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR ddl COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR status COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(database_type, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + OR COALESCE(user_name, '') COLLATE NOCASE LIKE ?5 ESCAPE '\\' + ) + ORDER BY created_at_ms DESC, id DESC + LIMIT ?6 OFFSET ?7" + ); + let mut statement = connection.prepare(&sql)?; + let rows = statement.query_map( + params![ + query.data_source_id.as_deref(), + query.database_name.as_deref(), + query.schema_name.as_deref(), + query.operation_type.as_deref(), + search_pattern.as_deref(), + limit, + offset, + ], + raw_operation_log, + )?; + let mut records = Vec::new(); + for row in rows { + records.push(decode_operation_log(row?)?); + } + + Ok(OperationLogPage { + records, + total: u64::try_from(total) + .map_err(|_| StorageError::NumericRange("operation log total"))?, + page_no: query.page_no, + page_size: query.page_size, + }) + } +} + +fn validate_create(input: &CreateOperationLog) -> Result<(), StorageError> { + validate_fields( + input.name.as_deref(), + input.data_source_id.as_deref(), + input.data_source_name.as_deref(), + input.database_name.as_deref(), + input.schema_name.as_deref(), + input.database_type.as_deref(), + input.user_name.as_deref(), + &input.ddl, + &input.status, + input.operation_rows, + input.use_time, + input.extend_info.as_deref(), + &input.operation_type, + ) +} + +fn validate_record(record: &OperationLogRecord) -> Result<(), StorageError> { + if record.id <= 0 { + return Err(StorageError::InvalidOperationLog( + "persisted id must be a positive signed 64-bit integer", + )); + } + if record.created_at_ms < 0 || record.updated_at_ms < record.created_at_ms { + return Err(StorageError::InvalidOperationLog( + "persisted timestamps are invalid", + )); + } + validate_fields( + record.name.as_deref(), + record.data_source_id.as_deref(), + record.data_source_name.as_deref(), + record.database_name.as_deref(), + record.schema_name.as_deref(), + record.database_type.as_deref(), + record.user_name.as_deref(), + &record.ddl, + &record.status, + record.operation_rows, + record.use_time, + record.extend_info.as_deref(), + &record.operation_type, + ) +} + +#[allow(clippy::too_many_arguments)] +fn validate_fields( + name: Option<&str>, + data_source_id: Option<&str>, + data_source_name: Option<&str>, + database_name: Option<&str>, + schema_name: Option<&str>, + database_type: Option<&str>, + user_name: Option<&str>, + ddl: &str, + status: &str, + operation_rows: Option, + use_time: Option, + extend_info: Option<&str>, + operation_type: &str, +) -> Result<(), StorageError> { + if name.is_some_and(|value| value.len() > MAX_NAME_BYTES) { + return Err(StorageError::InvalidOperationLog( + "name must be at most 512 UTF-8 bytes", + )); + } + if data_source_id.is_some_and(|value| value.len() > MAX_DATASOURCE_ID_BYTES) { + return Err(StorageError::InvalidOperationLog( + "datasource id must be at most 512 UTF-8 bytes", + )); + } + for value in [data_source_name, database_name, schema_name, user_name] + .into_iter() + .flatten() + { + if value.len() > MAX_SCOPE_BYTES { + return Err(StorageError::InvalidOperationLog( + "datasource, database, schema, and user names must be at most 1024 UTF-8 bytes", + )); + } + } + if database_type.is_some_and(|value| value.len() > MAX_DATABASE_TYPE_BYTES) { + return Err(StorageError::InvalidOperationLog( + "database type must be at most 255 UTF-8 bytes", + )); + } + if ddl.trim().is_empty() || ddl.len() > MAX_DDL_BYTES { + return Err(StorageError::InvalidOperationLog( + "SQL text must be non-empty and at most 16 MiB", + )); + } + if status.is_empty() || status.len() > MAX_STATE_BYTES { + return Err(StorageError::InvalidOperationLog( + "status must be non-empty and at most 255 UTF-8 bytes", + )); + } + if operation_rows.is_some_and(|value| value < 0) { + return Err(StorageError::InvalidOperationLog( + "operation rows must be non-negative", + )); + } + if use_time.is_some_and(|value| value < 0) { + return Err(StorageError::InvalidOperationLog( + "use time must be non-negative", + )); + } + if extend_info.is_some_and(|value| value.len() > MAX_EXTEND_INFO_BYTES) { + return Err(StorageError::InvalidOperationLog( + "extended information must be at most 16 MiB", + )); + } + if operation_type.is_empty() || operation_type.len() > MAX_STATE_BYTES { + return Err(StorageError::InvalidOperationLog( + "operation type must be non-empty and at most 255 UTF-8 bytes", + )); + } + Ok(()) +} + +fn validate_list_query(query: &OperationLogListQuery) -> Result<(), StorageError> { + if query.page_no == 0 { + return Err(StorageError::InvalidOperationLog( + "page number must be greater than zero", + )); + } + if query.page_size == 0 || query.page_size > MAX_PAGE_SIZE { + return Err(StorageError::InvalidOperationLog( + "page size must be between 1 and 1000", + )); + } + if query + .search_key + .as_ref() + .is_some_and(|value| value.len() > MAX_SEARCH_KEY_BYTES) + { + return Err(StorageError::InvalidOperationLog( + "search key must be at most 256 UTF-8 bytes", + )); + } + Ok(()) +} + +fn like_pattern(value: &str) -> String { + let mut pattern = String::with_capacity(value.len() + 2); + pattern.push('%'); + for character in value.chars() { + if matches!(character, '%' | '_' | '\\') { + pattern.push('\\'); + } + pattern.push(character); + } + pattern.push('%'); + pattern +} + +struct RawOperationLog { + id: i64, + name: Option, + data_source_id: Option, + data_source_name: Option, + connectable: Option, + database_name: Option, + database_type: Option, + ddl: String, + status: String, + operation_rows: Option, + use_time: Option, + extend_info: Option, + schema_name: Option, + organization_id: Option, + user_name: Option, + more: bool, + operation_type: String, + created_at_ms: i64, + updated_at_ms: i64, +} + +fn raw_operation_log(row: &rusqlite::Row<'_>) -> rusqlite::Result { + Ok(RawOperationLog { + id: row.get(0)?, + name: row.get(1)?, + data_source_id: row.get(2)?, + data_source_name: row.get(3)?, + connectable: row.get(4)?, + database_name: row.get(5)?, + database_type: row.get(6)?, + ddl: row.get(7)?, + status: row.get(8)?, + operation_rows: row.get(9)?, + use_time: row.get(10)?, + extend_info: row.get(11)?, + schema_name: row.get(12)?, + organization_id: row.get(13)?, + user_name: row.get(14)?, + more: row.get(15)?, + operation_type: row.get(16)?, + created_at_ms: row.get(17)?, + updated_at_ms: row.get(18)?, + }) +} + +fn decode_operation_log(raw: RawOperationLog) -> Result { + let record = OperationLogRecord { + id: raw.id, + name: raw.name, + data_source_id: raw.data_source_id, + data_source_name: raw.data_source_name, + connectable: raw.connectable, + database_name: raw.database_name, + database_type: raw.database_type, + ddl: raw.ddl, + status: raw.status, + operation_rows: raw.operation_rows, + use_time: raw.use_time, + extend_info: raw.extend_info, + schema_name: raw.schema_name, + organization_id: raw.organization_id, + user_name: raw.user_name, + more: raw.more, + operation_type: raw.operation_type, + created_at_ms: raw.created_at_ms, + updated_at_ms: raw.updated_at_ms, + }; + validate_record(&record)?; + Ok(record) +} + +fn load_operation_log( + connection: &Connection, + id: i64, +) -> Result, StorageError> { + let sql = format!("SELECT {OPERATION_LOG_COLUMNS} FROM operation_logs WHERE id = ?1"); + connection + .query_row(&sql, [id], raw_operation_log) + .optional()? + .map(decode_operation_log) + .transpose() +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + + use tempfile::TempDir; + + use super::{CreateOperationLog, OperationLogListQuery}; + use crate::{SecretRef, SecretValue, SecretVault, SecretVaultError, Storage, StorageError}; + + #[derive(Debug)] + struct EmptyVault; + + impl SecretVault for EmptyVault { + fn probe(&self) -> Result<(), SecretVaultError> { + Ok(()) + } + + fn create( + &self, + _reference: &SecretRef, + _value: &SecretValue, + ) -> Result<(), SecretVaultError> { + Ok(()) + } + + fn get(&self, _reference: &SecretRef) -> Result, SecretVaultError> { + Ok(None) + } + + fn delete(&self, _reference: &SecretRef) -> Result<(), SecretVaultError> { + Ok(()) + } + } + + fn open(directory: &TempDir) -> Storage { + Storage::open(directory.path(), Arc::new(EmptyVault)).expect("storage opens") + } + + fn input(name: &str, datasource_id: &str, ddl: &str) -> CreateOperationLog { + CreateOperationLog { + name: Some(name.to_owned()), + data_source_id: Some(datasource_id.to_owned()), + data_source_name: Some("Local MySQL".to_owned()), + connectable: Some(true), + database_name: Some("chat2db".to_owned()), + database_type: Some("MYSQL".to_owned()), + ddl: ddl.to_owned(), + status: "SUCCESS".to_owned(), + operation_rows: Some(1), + use_time: Some(12), + extend_info: Some(r#"{"source":"console","result":null}"#.to_owned()), + schema_name: Some("public".to_owned()), + organization_id: None, + user_name: Some("local-user".to_owned()), + more: false, + operation_type: "SQL_EXECUTE".to_owned(), + } + } + + #[test] + fn operation_log_round_trip_survives_reopen() { + let directory = TempDir::new().expect("temp dir"); + let storage = open(&directory); + let mut create = input("orders", "datasource-1", "SELECT * FROM orders"); + create.connectable = None; + create.operation_rows = None; + create.organization_id = Some(7); + create.more = true; + let created = storage + .create_operation_log(create) + .expect("operation log creates"); + assert!(created.id > 0); + assert_eq!(created.connectable, None); + assert_eq!(created.operation_rows, None); + assert_eq!(created.organization_id, Some(7)); + assert!(created.more); + assert_eq!(created.created_at_ms, created.updated_at_ms); + assert_eq!( + storage + .get_operation_log(created.id) + .expect("operation log reads"), + Some(created.clone()) + ); + + drop(storage); + let reopened = open(&directory); + assert_eq!( + reopened + .get_operation_log(created.id) + .expect("reopened operation log reads"), + Some(created) + ); + assert_eq!( + reopened + .get_operation_log(0) + .expect("invalid identity is absent"), + None + ); + } + + #[test] + fn list_filters_searches_and_pages_in_stable_newest_first_order() { + let directory = TempDir::new().expect("temp dir"); + let storage = open(&directory); + let alpha = storage + .create_operation_log(input("Alpha", "datasource-a", "SELECT alpha")) + .expect("alpha creates"); + + let mut beta_input = input("Beta", "datasource-a", "SELECT beta"); + beta_input.database_name = Some("analytics".to_owned()); + beta_input.schema_name = Some("reporting".to_owned()); + beta_input.user_name = Some("Alice".to_owned()); + let beta = storage + .create_operation_log(beta_input) + .expect("beta creates"); + + let mut gamma_input = input("Gamma", "datasource-b", "DELETE FROM audit_events"); + gamma_input.operation_type = "SQL_AUDIT".to_owned(); + let gamma = storage + .create_operation_log(gamma_input) + .expect("gamma creates"); + + let delta = storage + .create_operation_log(input("literal %_ marker", "datasource-a", "SELECT delta")) + .expect("delta creates"); + + let first_page = storage + .list_operation_logs(&OperationLogListQuery { + page_size: 2, + ..OperationLogListQuery::default() + }) + .expect("first page lists"); + assert_eq!(first_page.total, 4); + assert_eq!( + first_page + .records + .iter() + .map(|record| record.id) + .collect::>(), + vec![delta.id, gamma.id] + ); + + let second_page = storage + .list_operation_logs(&OperationLogListQuery { + page_no: 2, + page_size: 2, + ..OperationLogListQuery::default() + }) + .expect("second page lists"); + assert_eq!( + second_page + .records + .iter() + .map(|record| record.id) + .collect::>(), + vec![beta.id, alpha.id] + ); + + let filtered = storage + .list_operation_logs(&OperationLogListQuery { + data_source_id: Some("datasource-a".to_owned()), + database_name: Some("analytics".to_owned()), + schema_name: Some("reporting".to_owned()), + operation_type: Some("SQL_EXECUTE".to_owned()), + search_key: Some("ALICE".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("combined filters list"); + assert_eq!(filtered.total, 1); + assert_eq!(filtered.records[0].id, beta.id); + + let by_datasource = storage + .list_operation_logs(&OperationLogListQuery { + data_source_id: Some("datasource-b".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("datasource filters"); + assert_eq!(by_datasource.records, vec![gamma.clone()]); + + let by_database = storage + .list_operation_logs(&OperationLogListQuery { + database_name: Some("analytics".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("database filters"); + assert_eq!(by_database.records, vec![beta.clone()]); + + let by_schema = storage + .list_operation_logs(&OperationLogListQuery { + schema_name: Some("reporting".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("schema filters"); + assert_eq!(by_schema.records, vec![beta]); + + let audit = storage + .list_operation_logs(&OperationLogListQuery { + operation_type: Some("SQL_AUDIT".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("operation type filters"); + assert_eq!(audit.records, vec![gamma]); + + let escaped_search = storage + .list_operation_logs(&OperationLogListQuery { + search_key: Some("%_".to_owned()), + ..OperationLogListQuery::default() + }) + .expect("wildcards search literally"); + assert_eq!(escaped_search.records, vec![delta]); + } + + #[test] + fn invalid_operation_logs_and_pages_fail_closed() { + let directory = TempDir::new().expect("temp dir"); + let storage = open(&directory); + let mut invalid = input("invalid", "datasource-1", "SELECT 1"); + invalid.operation_rows = Some(-1); + assert!(matches!( + storage + .create_operation_log(invalid) + .expect_err("negative operation row count must fail"), + StorageError::InvalidOperationLog(_) + )); + + let mut invalid = input("invalid", "datasource-1", "SELECT 1"); + invalid.use_time = Some(-1); + assert!(matches!( + storage + .create_operation_log(invalid) + .expect_err("negative duration must fail"), + StorageError::InvalidOperationLog(_) + )); + + let invalid_page = storage + .list_operation_logs(&OperationLogListQuery { + page_no: 0, + ..OperationLogListQuery::default() + }) + .expect_err("zero page must fail"); + assert!(matches!(invalid_page, StorageError::InvalidOperationLog(_))); + } +} diff --git a/crates/chat2db-storage/tests/public_api.rs b/crates/chat2db-storage/tests/public_api.rs index c40799b..980989c 100644 --- a/crates/chat2db-storage/tests/public_api.rs +++ b/crates/chat2db-storage/tests/public_api.rs @@ -1,9 +1,9 @@ use chat2db_storage::{ AgentCompaction, AgentMessageRole, AgentRunStatus, AgentRunUpdate, AppendAgentMessage, - CompactAgentRun, CreateAgentSession, CreateProviderProfile, CreateSavedConsole, - MAX_AGENT_MESSAGE_BYTES, MAX_RESULT_PAGE_BYTES, MAX_RESULT_PAGE_ROWS, MIN_RESULT_PAGE_BYTES, - PageRequest, ProviderKind, PurgeReport, SavedConsoleListQuery, ToolPermissionDecision, - UpdateAgentSession, UpdateSavedConsole, + CompactAgentRun, CreateAgentSession, CreateOperationLog, CreateProviderProfile, + CreateSavedConsole, MAX_AGENT_MESSAGE_BYTES, MAX_RESULT_PAGE_BYTES, MAX_RESULT_PAGE_ROWS, + MIN_RESULT_PAGE_BYTES, OperationLogListQuery, PageRequest, ProviderKind, PurgeReport, + SavedConsoleListQuery, ToolPermissionDecision, UpdateAgentSession, UpdateSavedConsole, }; #[test] @@ -53,6 +53,39 @@ fn saved_console_contracts_are_nameable_outside_the_crate() { assert_eq!(update.ddl.as_deref(), Some("SELECT 1")); } +#[test] +fn operation_log_contracts_are_nameable_outside_the_crate() { + let create = CreateOperationLog { + name: None, + data_source_id: Some("opaque-datasource-id".to_owned()), + data_source_name: Some("Local MySQL".to_owned()), + connectable: Some(true), + database_name: Some("chat2db".to_owned()), + database_type: Some("MYSQL".to_owned()), + ddl: "SELECT 1".to_owned(), + status: "SUCCESS".to_owned(), + operation_rows: None, + use_time: Some(4), + extend_info: Some(r#"{"source":"console"}"#.to_owned()), + schema_name: None, + organization_id: None, + user_name: None, + more: false, + operation_type: "SQL_EXECUTE".to_owned(), + }; + let query = OperationLogListQuery { + data_source_id: Some("opaque-datasource-id".to_owned()), + operation_type: Some("SQL_EXECUTE".to_owned()), + page_size: 100, + ..OperationLogListQuery::default() + }; + + assert_eq!(create.connectable, Some(true)); + assert_eq!(create.use_time, Some(4)); + assert_eq!(query.page_no, 1); + assert_eq!(query.page_size, 100); +} + #[test] fn provider_and_agent_contracts_are_nameable_outside_the_crate() { let provider = CreateProviderProfile { diff --git a/docs/architecture.md b/docs/architecture.md index 9649bc7..1eb0398 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,8 +16,8 @@ without embedding H2 in the compatibility-engine JAR. The Web and Tauri hosts open the production vault, SQLite storage, and verified driver catalog before exposing a shared `Application`; they do not start Java -during host bootstrap. Native MySQL connection, database/schema/table, preview, -and supported Console SELECT operations do not acquire a Java lease. The Core +during host bootstrap. Native MySQL connection, object metadata, preview, and +Console reads/writes/scripts do not acquire a Java lease. The Core `EngineManager` starts one Java generation on the first JDBC-only database, parser, formatter, completion, builder, or advanced metadata request. It shares that single-flight startup across concurrent callers and issues generation-scoped @@ -45,8 +45,9 @@ workbench. The build exports the unmodified Community frontend tree pinned by contract through Axum; desktop maps the existing `window.javaQuery` contract through one Tauri `legacy_request` command. Both paths call the same Rust legacy dispatcher. The implemented product slice covers native MySQL connection -testing, datasource CRUD/tree, database/schema/table discovery, read-only table -preview, and typed Console SELECT. Signing, distribution, the remaining dialect +testing, datasource CRUD/tree, relational object metadata, read-only table +preview, and bounded unparameterized Console reads, writes, scripts, +transactions, cancellation, history, and large values. Signing, distribution, the remaining dialect estate, broader historical API coverage, and packaging remain target components. CLI and MCP attach to a running host rather than composing a second product runtime. @@ -71,8 +72,8 @@ after preserving the disabled-engine error contract. | Durable state | Rust | SQLite, retained-result files, and a mandatory injected secret-vault contract | | AI agent | Rust | Provider adapters, tool loop, limits, compaction, and cancellation | | MCP and CLI | Rust | Adapters around the same product services and policy | -| Native MySQL product slice | Rust / `mysql_async` | Connection, first-stage read-only object metadata and Community-compatible legacy routes/envelopes, nullable defaults, preview, supported SELECT, typed streaming, limits, cancellation, and retained results | -| Compatibility databases and advanced MySQL operations | Java 17 | Existing SPI/plugins, JDBC, SQL builders, parsing, formatting, completion, writes, transactions, and non-MySQL metadata | +| Native MySQL product slice | Rust / `mysql_async` | Connection, object metadata, Community-compatible routes/envelopes, preview, unparameterized Console reads/writes/scripts, transactions, paging, limits, cancellation, large values, and durable history | +| Compatibility databases and remaining MySQL operations | Java 17 | Existing SPI/plugins, JDBC bind parameters, SQL builders, parsing, formatting, completion, unmapped MySQL features, and non-MySQL metadata | | SQL parsing, formatting, and completion | Java 17 | Existing Java ANTLR grammars, parser behavior, formatter behavior, and completion | | Rust-to-Java IPC | Shared Protobuf contract | Length-prefixed frames over private stdin/stdout | @@ -89,7 +90,7 @@ React in system WebView React in browser <- owner-only local attachment <- rmcp stdio server <- MCP client -> SQLite and result store -> AI agent runtime - -> native MySQL connection / metadata / SELECT + -> native MySQL connection / metadata / Console -> Java process supervisor -> Protobuf stdin/stdout -> Java database compatibility engine @@ -128,23 +129,27 @@ cross-language acceptance gates pass. ## Database boundary Java/JDBC remains the compatibility implementation for other databases and for -unmigrated advanced MySQL operations. The first closed native route uses -upstream `mysql_async 0.37.0` for MySQL connection testing, -database/schema/table discovery, preview, and supported Console SELECT. Core -selects this backend before requesting an `EngineLease`; unrecognized drivers -cannot enter it. Parameterized, CTE-first, locking, server-file, and -multi-statement SELECT is rejected rather than silently starting Java. +unmigrated MySQL operations. The native route uses upstream +`mysql_async 0.37.0` for MySQL connection testing, object metadata, preview, +and unparameterized Console execution. Core selects this backend before +requesting an `EngineLease`; unrecognized drivers cannot enter it. Console +bind parameters remain unsupported rather than silently starting Java. The native MySQL baseline implements: - JDBC-URL and connection-property translation into `mysql_async::Opts`, with explicit Rustls policy, TCP preference, and a 15-second connect deadline; -- database/schema/table metadata and safely quoted bounded table preview; -- one read-only prepared SELECT with ordered typed columns and values emitted - through the existing retained-result wire contract; -- row, result-byte, batch, column, SQL, identifier, and scalar limits; and -- active-query cancellation and truncation through a second bounded connection - issuing `KILL CONNECTION`, followed by deterministic result cleanup. +- native database/schema/table/column/index/key/view/routine/trigger metadata + and safely quoted bounded table preview; +- semicolon and `DELIMITER` script splitting, preserved-single dispatch, + multiple result sets, DDL/DML, explicit transactions, error continuation, + `EXPLAIN`, normal/all-row paging, and datasource read-only enforcement; +- typed values up to 32 MiB, a shared 64 MiB retained-result budget, 64 KiB + previews, owner-scoped expiring tokens, bounded chunks, and desktop downloads; +- active-query cancellation through a second bounded connection issuing + `KILL CONNECTION`, followed by deterministic cleanup; and +- SQLite-backed saved Consoles and per-statement execution history exposed by + the original Web and desktop contracts. The JDBC baseline implements: diff --git a/docs/mysql-community-parity.md b/docs/mysql-community-parity.md index 6181227..fa77811 100644 --- a/docs/mysql-community-parity.md +++ b/docs/mysql-community-parity.md @@ -3,15 +3,22 @@ ## Status - Community baseline: `OtterMind/Chat2DB` `main@3cb8af54cad5bd5caa20bb25f10d9b0e4f01931c`. -- Rust baseline: `OtterMind/Chat2DB-Rust` `main@352838b7d20fad568fd68f5d825e65c56104bd29`. +- Rust baseline: `OtterMind/Chat2DB-Rust` `main@e9789cded5d227ba1f48690264046610eedb0807`. - Product target: the original Community React frontend running against the Rust Web or Tauri host. - Runtime-tested now: datasource CRUD/test; database, schema, table, column, index, foreign-key, primary-key, view, function, procedure, trigger, and routine-parameter metadata; all first-stage historical metadata routes; - bounded table preview; saved Console CRUD; and one unparameterized read-only MySQL - `SELECT` with paging and cancellation. Native metadata keeps Java dormant. - Paged table name/comment search, complete-list filtering, page-size validation, - HTTP binding-error envelopes, and nullable column defaults match the locked + bounded table preview; saved Console CRUD; and native unparameterized MySQL + Console execution. Console coverage includes DDL/DML, semicolon scripts, + `DELIMITER` procedure scripts, explicit transactions, error-continue policy, + preserved-single dispatch, `EXPLAIN`, bounded all-row paging, datasource + read-only enforcement, multiple result sets, exact affected-row counts, + cancellation, a shared 64 MiB retained-result budget, durable history with + cancelled-state projection, and bounded large-cell retrieval/download. Base64 + and hex large-text chunks use UTF-8 byte offsets as the original frontend + expects. Native metadata and Console execution keep Java dormant. Paged table + name/comment search, complete-list filtering, page-size validation, HTTP + binding-error envelopes, and nullable column defaults match the locked Community baseline. - Complete parity: not implemented. @@ -48,10 +55,10 @@ route reaches it and a real MySQL product test covers the behavior. | Table DDL | `/api/rdb/ddl/*`, `/api/rdb/table/modify/sql`, `/delete`, `/truncate`, `/copy`, create/update examples, DDL export | Partial builder infrastructure only | Match create/alter/drop/truncate/copy preview and execution, including columns, indexes, keys, charset, collation, comments, and MySQL types. | | Views | `/api/rdb/view/list`, `/column_list`, `/detail`, `/query`, `/view_meta`, `/modify/sql`, `/delete`, `/drop` | Native list, column list, and `SHOW CREATE VIEW` detail are mapped to the legacy routes | Add data query, metadata, preview, create/alter, and drop flows. | | Functions, procedures, and triggers | `/api/rdb/{function,procedure,trigger}/{list,detail}`, `/api/rdb/routine/{preview_invocation,preview_migration,execute_migration}` | Native list/detail and routine-parameter projections are implemented; every original list/detail route is mapped | Add invocation and migration preview/execution flows. | -| Console SELECT | `/api/rdb/dml/execute`, desktop `sql-execute`/`sql-cancel` | One unparameterized SELECT, paging, limits, cancellation | Add parameters, CTEs, all MySQL read statements, multiple result sets, warnings, affected rows, and Community result shapes. | -| Console scripts and writes | `/api/rdb/dml/execute`, `/execute_ddl` | Not implemented natively | Match Community statement splitting, script execution policy, DDL/DML, transaction settlement, cancellation, and per-statement results. | -| Large cell values | `/api/rdb/cell/value`, `/download`, `/download_path` | Not implemented | Preserve bounded previews and explicit full-value download without loading unbounded cells into the WebView. | -| Saved consoles and SQL history | `/api/operation/saved/*`, `/api/operation/log/{create,list}` and detail | Saved Console CRUD implemented; execution history absent | Persist and filter Community-compatible history and keep restart-safe Console state. | +| Console SELECT | `/api/rdb/dml/execute`, desktop `sql-execute`/`sql-cancel` | Native unparameterized MySQL reads, CTEs, normal/all-row paging, preserved-single dispatch, `EXPLAIN`, limits, multiple result sets, affected-row counts, datasource read-only enforcement, and cancellation implemented | Add JDBC-style bind parameters and close remaining warning/error/result-shape differences. | +| Console scripts and writes | `/api/rdb/dml/execute`, `/execute_ddl` | Native unparameterized DDL/DML, semicolon and `DELIMITER` scripts, explicit transactions, error-continue policy, cancellation, and per-statement results implemented | Add bind parameters and complete exact Community conformance for unsupported edge-case scripts. | +| Large cell values | `/api/rdb/cell/value`, `/download`, `/download_path` | Bounded UTF-8/Base64 previews, owner-scoped expiring tokens, byte-oriented Base64/hex chunk reads, character-oriented text reads, and full-value downloads implemented | Add long-running export/task integration and close remaining content-type/display-mode differences. | +| Saved consoles and SQL history | `/api/operation/saved/*`, `/api/operation/log/{create,list}` and detail | Restart-safe saved Console CRUD plus durable history create/list/detail, filtering, paging, per-statement recording, and cancelled-state projection implemented | Complete remaining Community audit/history fields and non-Console producers. | | SQL parser, formatter, validation, completion | `/api/sql/format`, `/valid_select`, `/api/sql_parser/get_keywords`, `/context/{parser,quick_parser,tip,hover}` | Modern parser/validation/formatter/completion contracts implemented through Java; legacy routes absent | Map every original endpoint to the fixed Community implementation with matching UTF-16 offsets and envelopes. | | Import, export, and tasks | `/api/import/{sql_file,other_file}`, `/api/export/{sql_file,other_file}`, `/api/task/*`, `/api/rdb/dml/export`, table class generation | Not implemented | Add bounded streaming import/export, progress, stop, download, cleanup, and failure recovery. | | Account administration | `/api/rdb/account/{capability,list,grants,preview,execute}` | Not implemented | Match MySQL users, hosts, authentication, privileges, role/grant previews, execution, and current Community escaping rules. | @@ -64,8 +71,10 @@ route reaches it and a real MySQL product test covers the behavior. 1. Complete: native read-only object metadata and every matching original metadata route, with Axum/dispatch contracts and a real MySQL 8.4 product vertical that proves Java remains dormant. -2. Full Console statement execution, multi-result handling, writes, transactions, - history, cancellation, and large-cell retrieval. +2. Implemented slice: native unparameterized Console statement execution, + multi-result handling, writes, transactions, history, cancellation, and + large-cell retrieval. Bind parameters and remaining exact Community edge-case + conformance are still required for complete parity. 3. Table data editing plus database/schema/table/view DDL preview and execution. 4. Import/export/tasks, datasource lifecycle/SSH/import, account administration, routines, structure comparison, pins, and ER metadata. @@ -86,6 +95,10 @@ the complete repository verification gate, and all GitHub Actions jobs. - `third_party/chat2db-community/chat2db-community-server/chat2db-community-web/src/main/java/ai/chat2db/community/web/api/controller/` - `third_party/chat2db-community/chat2db-community-server/chat2db-community-plugins/chat2db-community-mysql/src/main/java/ai/chat2db/plugin/mysql/` - `crates/chat2db-core/src/native_mysql.rs` +- `crates/chat2db-core/src/large_value.rs` +- `crates/chat2db-core/tests/native_mysql_console_docker.rs` +- `crates/chat2db-storage/src/operation_log.rs` +- `crates/chat2db-storage/migrations/004_operation_log.sql` - `crates/chat2db-core/tests/native_mysql_product.rs` - `crates/chat2db-core/src/community.rs` - `apps/chat2db-web/src/legacy.rs` diff --git a/docs/stages.md b/docs/stages.md index 4477eae..5990428 100644 --- a/docs/stages.md +++ b/docs/stages.md @@ -12,7 +12,7 @@ in runtime health until then. | 4 | Complete | Product and result storage foundation | SQLite migration/integrity gates, mandatory vault boundary, revisioned datasource records, durable result frames, bounded paging/quota, expiry, writer cleanup and recovery tests | | 5 | Complete | Product transports | Generated OpenAPI/TypeScript contract, Axum JSON/SSE, Tauri 2 commands/channels, shared SQL workbench, product H2 tests | | 6 | Complete | Agent, MCP, and CLI | Direct providers, durable bounded tool loop, SQL tools/permissions, compaction, Web/Tauri run transports, owner-only local attachment, read-query CLI, and bounded `rmcp` stdio tools | -| 7 | In progress | Chat2DB compatibility estate | 7A managed JDBC packs through 7M bounded table preview are implemented; native `mysql_async` now owns the MySQL browser and supported SELECT path while Java starts on demand for unmigrated compatibility work; the current product retains the original Community layout with historical HTTP/Tauri compatibility for MySQL browsing, saved Consoles, and bounded SELECT execution | +| 7 | In progress | Chat2DB compatibility estate | 7A managed JDBC packs through 7M bounded table preview are implemented; native `mysql_async` now owns the MySQL browser and unparameterized Console data plane while Java starts on demand for unmigrated compatibility work; the current product retains the original Community layout with historical HTTP/Tauri compatibility for MySQL browsing, saved Consoles, scripts, writes, transactions, history, cancellation, and large values | | 8 | Planned | Packaging and release | License authorization, NOTICE/SBOM, jlink runtime, Tauri installers, signed product/engine/driver manifests, atomic update and rollback, size measurement | Stage 3 completion means the versioned Rust-Java bridge can load an external @@ -314,24 +314,28 @@ strict Clippy, all 509 Rust tests, 49 frontend tests, the Tauri bridge contract, and the Community production build passed. Browser click-through was not performed because the browser runtime exposed no browser instance. -This slice is query-only. Arbitrary DDL, DML, and multi-statement Console -execution are not implemented and must not be inferred from the historical -endpoint name. +The native Console follow-up extends this route to DDL/DML, semicolon and +`DELIMITER` scripts, multiple result sets, explicit transactions, +error-continue policy, `single`, `EXPLAIN`, normal/all-row paging, datasource +read-only enforcement, cancellation, durable per-statement history, and bounded +large-cell retrieval/download. Web returns the historical synchronous shape; +desktop emits `rows` exactly once and uses `updateCount` for non-tabular +results. One 64 MiB budget covers every retained row in the complete script. The native MySQL follow-up pins upstream `mysql_async 0.37.0` with Rustls and routes MySQL connection testing; database/schema/table/column/index/key/view/ -function/procedure/trigger metadata; table preview; and supported Console SELECT +function/procedure/trigger metadata; table preview; and Console execution before Java lease acquisition. Original Community HTTP and desktop-dispatch routes expose the read-only metadata projections, including top-level DDL list totals and `SHOW CREATE` details. The adapter also preserves Community's paged table name/comment search, ignored filtering on complete-list endpoints, metadata page-size validation, HTTP 200 error envelopes, and the distinction between a null and empty-string column default. -SELECT executes in a MySQL read-only transaction -and emits the existing typed retained-result wire messages directly from Core. -Parameters, CTE-first SELECT, locking/server-file variants, and multi-statements -fail before Java startup. Row and byte truncation plus cancellation terminate the -active MySQL connection through a separate bounded control connection. The +The original asynchronous SELECT path still emits typed retained-result wire +messages from a read-only transaction. The native Console path owns broader +unparameterized statements and scripts on one session. Bind parameters remain +unsupported. Cancellation terminates the active MySQL connection through a +separate bounded control connection. The explicit `native-mysql-integration` target and MySQL CI job use a deliberately missing Java executable and verify connection, first-stage object metadata, two-row preview, typed three-row Console output, one-row truncation, active @@ -345,7 +349,7 @@ compatibility operations and other database types remain on the lazy Java/JDBC path. Stage 7 remains incomplete. Complete MySQL type conformance, native bind -parameters and CTE-first SELECT, script execution, data import/export, +parameters, remaining exact Community Console edge cases, data import/export, non-relational behavior, remaining builder operations and plugin inventory, driver distribution, and per-dialect conformance are not implemented.