diff --git a/obp-api/src/main/scala/code/api/util/ApiTag.scala b/obp-api/src/main/scala/code/api/util/ApiTag.scala index 6ef32178ff..56a60ae257 100644 --- a/obp-api/src/main/scala/code/api/util/ApiTag.scala +++ b/obp-api/src/main/scala/code/api/util/ApiTag.scala @@ -104,6 +104,7 @@ object ApiTag { val apiTagCache = ResourceDocTag("Cache") val apiTagLogCache = ResourceDocTag("Log-Cache") val apiTagTrading = ResourceDocTag("Trading") + val apiTagTrade = ResourceDocTag("Trade") val apiTagMarket = ResourceDocTag("Market") val apiTagApiCollection = ResourceDocTag("Api-Collection") diff --git a/obp-api/src/main/scala/code/api/util/Glossary.scala b/obp-api/src/main/scala/code/api/util/Glossary.scala index ca4d0ab5b1..a12c87b6f6 100644 --- a/obp-api/src/main/scala/code/api/util/Glossary.scala +++ b/obp-api/src/main/scala/code/api/util/Glossary.scala @@ -5316,6 +5316,138 @@ object Glossary extends MdcLoggable { | """) + glossaryItems += GlossaryItem( + title = "OBP-MCP", + description = + s""" + |# OBP-MCP + | + |**OBP-MCP** is a [Model Context Protocol](https://modelcontextprotocol.io) server for the Open Bank Project API. It lets AI assistants (Claude, Opey, IDE agents, custom LLM tooling) discover and call OBP-API endpoints as MCP *tools*, without hard-coding any knowledge of the 600+ endpoints. + | + |Repository: [github.com/OpenBankProject/OBP-MCP](https://github.com/OpenBankProject/OBP-MCP) + | + |## What it does + | + |OBP-MCP is a thin protocol bridge. AI clients speak **MCP** to it; it speaks **HTTPS / REST** to OBP-API on their behalf, attaching the user's OAuth token or Consent-JWT. + | + |``` + |┌──────────────────┐ MCP ┌────────────────────────┐ HTTPS ┌──────────────┐ + |│ AI client │ ───────▶ │ OBP-MCP │ ─────────▶ │ OBP-API │ + |│ (Claude, Opey, │ ◀─────── │ (FastMCP server) │ ◀───────── │ │ + |│ IDE agent) │ tools │ │ JSON │ │ + |└──────────────────┘ └────────────────────────┘ └──────────────┘ + |``` + | + |## Three-step discovery + call (no RAG, no vector DB) + | + |OBP-MCP avoids embedding the 4 MB OpenAPI spec into the LLM's context. Instead it exposes three tools that work together: + | + |1. **`list_endpoints_by_tag(tags)`** — returns lightweight summaries (~50–100 tokens each) from a local `endpoint_index.json`. Lets the LLM narrow down to a handful of candidate endpoints by tag (e.g. `Account`, `Transaction-Request`, `Consent`). + |2. **`get_endpoint_schema(endpoint_id)`** — lazy-loads the full OpenAPI schema for one endpoint from a local `endpoint_schemas.json`. + |3. **`call_obp_api(endpoint_id, path_params, query_params, body, headers)`** — actually executes the HTTP request against the live OBP-API. + | + |Two further tools cover the glossary itself: **`list_glossary_terms(search_query)`** and **`get_glossary_term(term_id)`**, backed by a local `glossary_index.json` of 800+ banking terms. + | + |## Three kinds of traffic + | + |It is important to understand that OBP-MCP is **not** a documentation lookup tool — it makes real, authenticated business calls: + | + |- **Documentation / discovery** — `list_endpoints_by_tag`, `get_endpoint_schema`, glossary tools. Served from local JSON, no network. + |- **Business calls** — `call_obp_api` proxies whatever the endpoint declares: `GET /banks/{BANK_ID}/accounts`, `POST .../transaction-requests/SEPA`, `PUT /accounts/{ACC}/label`, `DELETE /my/consents/{CONSENT_ID}`, etc. Real money / data moves. + |- **Index refresh** — at startup and on a timer, OBP-MCP re-fetches OBP's resource-docs and swagger to rebuild the local indexes, so discovery stays fast and offline. + | + |## Authentication and authorization + | + |OBP-MCP supports several modes via the `AUTH_PROVIDER` environment variable for client-to-MCP auth: + | + || Mode | Use case | Notes | + ||----------------|---------------------------------------|----------------------------------------------------| + || `bearer-only` | Internal agents (e.g. Opey) | JWT validation only, multi-issuer | + || `obp-oidc` | External MCP clients | Full OAuth 2.1 + Dynamic Client Registration | + || `keycloak` | External MCP clients | OAuth 2.1 + minimal DCR proxy workaround | + || `none` | Development / testing | No auth required | + | + |For onward calls to OBP-API, `OBP_AUTHORIZATION_VIA` selects: + | + |- **`oauth`** — pulls the access token from the MCP request context and sends `Authorization: Bearer ...`. + |- **`consent`** — if the endpoint declares any required roles and no `Consent-JWT` is supplied, the tool returns a `consent_required` payload listing the required roles and bank scope, so the client can elicit user approval and come back with a `Consent-JWT` header. Public / no-role endpoints skip this and call straight through. + |- **`none`** — calls OBP unauthenticated (only useful for genuinely public endpoints). + | + |This means the consent flow is enforced at the MCP layer, not just at OBP-API: an agent cannot accidentally call a privileged endpoint without explicit user consent. + | + |## Why it matters + | + |OBP-MCP is the canonical way to make Open Bank Project endpoints **agent-callable**. Instead of teaching every LLM about every endpoint up front, the LLM is given five generic tools and lets the indexes and schemas guide it to the right call at runtime. The same server can serve internal agents (Opey) and external clients (Claude Desktop, IDE plugins, third-party agents) by switching auth providers. + | + |See also: [Opey](/glossary#Opey), [Consent](/glossary#Consent), [Authentication: OAuth 2.0](/glossary#Authentication:-OAuth-2.0). + | +""") + + + glossaryItems += GlossaryItem( + title = "Opey", + description = + s""" + |# Opey + | + |**Opey** (current generation: **Opey II**) is the Open Bank Project's agentic AI assistant — a chatbot that lets users explore and operate the OBP API in natural language. It is built on [LangGraph](https://www.langchain.com/langgraph), is provider-agnostic across LLMs (Anthropic, OpenAI, Ollama), and is the chat backend used by **OBP-Portal**. + | + |Repository: [github.com/OpenBankProject/OBP-Opey-II](https://github.com/OpenBankProject/OBP-Opey-II) + | + |## Opey is an agent. OBP-MCP is its tool surface. + | + |Since [OBP-MCP](/glossary#OBP-MCP) was introduced, Opey has been refactored from a self-contained chatbot (with its own endpoint search, glossary search, and OBP HTTP client baked in) into a focused **agent** that *consumes* OBP-MCP as its primary tool source. + | + |Opey's `mcp_servers.json` typically points at a running OBP-MCP instance: + | + |```json + |{ + | "servers": [ + | { + | "name": "obp", + | "url": "http://0.0.0.0:9100/mcp", + | "transport": "http", + | "requires_auth": true + | } + | ] + |} + |``` + | + |The Opey README puts it bluntly: *"As a minimum, Opey should be connected to OBP-MCP, or it won't know anything about the Open Bank Project except for what you put in the system prompt."* + | + |## What OBP-MCP took over + | + |Subsystems that used to live in Opey are now generic MCP tools any client can use: + | + || Old Opey responsibility | Now in OBP-MCP | + ||--------------------------------------------------------------------------------------|-------------------------------------------------------------| + || Endpoint Retrieval RAG pipeline (vector store of swagger, query reformulation, etc.) | `list_endpoints_by_tag` + `get_endpoint_schema` | + || Glossary Retrieval RAG pipeline | `list_glossary_terms` + `get_glossary_term` | + || `OBPClient` (aiohttp + OAuth + consent JWT) — the actual HTTP layer to OBP-API | `call_obp_api` (`oauth` / `consent` / `none` modes) | + || "Which endpoint should I call?" logic baked into the agent | Externalised — any MCP client can now discover and call | + | + |## What Opey still uniquely does + | + |OBP-MCP is stateless and has no model — it cannot reason, plan, or hold a conversation. Everything below is what makes Opey *Opey*: + | + |- **The LLM loop itself.** Opey runs the actual reasoning via a LangGraph state machine (`START → Opey Agent → Tools → Sanitize → Opey → Summarize → END`), with **task follow-through**: when a tool call fails (e.g. missing entitlement), Opey reuses tools to self-correct instead of bouncing the problem back to the user. + |- **Human-in-the-loop approval — richer than MCP's `consent_required`.** A `ToolRegistry` classifies operations as **SAFE / MODERATE / DANGEROUS / CRITICAL**. An `ApprovalManager` persists "approve once / session / user / workspace" decisions with TTLs. The human-review node only interrupts when truly needed. OBP-MCP just *says* consent is required; Opey decides **how** to ask, **whether** to ask again, and **remembers** the answer. + |- **Conversation state.** SQLite-backed LangGraph checkpoints (`checkpoints.db`), token counting, automatic summarisation when approaching the model context limit, and graceful degradation in long sessions. + |- **The streaming chat service.** FastAPI endpoints (`POST /invoke`, `POST /stream` SSE, `POST /submit_approval`, `GET /user/consent`, `GET /status`) — this is what OBP-Portal's chat UI actually talks to. Streaming events are produced by dedicated processors (token, tool, human-review, metadata, end). + |- **Session, auth, usage.** OBP user session management, consent-JWT parsing for user identification, rate limiting, usage tracking, and an admin-client singleton for system-level operations. + |- **Domain-tuned system prompt.** Behavioural guidelines such as *Tool-First / Knowledge-Second*, *No Hallucination*, *Proactive Verification*, and *Transparent Errors*. Configurable via `OPEY_SYSTEM_PROMPT`. + |- **Model abstraction.** Provider-agnostic via `MODEL_PROVIDER` / `MODEL_NAME` — swap Claude for GPT or a local Ollama model without touching the graph. New models are registered in `MODEL_CONFIGS` (`src/agent/utils/model_factory.py`). + |- **Evaluation framework.** Parameter-sweep experiments over batch size, k-value, retry thresholds; CSV export of precision / recall / latency P50–P99; combined scoring (e.g. 70% recall + 30% speed) to find sweet spots. Something a tool surface like MCP has no concept of. + | + |## One-line summary + | + |**OBP-MCP is the *tool surface* over OBP-API. Opey II is the *agent* that drives it.** Before OBP-MCP, Opey had to be both. Now OBP-MCP provides discovery and authenticated calls as a generic, multi-client surface (Claude Desktop, IDE plugins, third-party agents can all use it), and Opey II becomes a thinner, more focused orchestrator: planning, approvals, conversation state, streaming, and the chat UX that OBP-Portal embeds. + | + |See also: [OBP-MCP](/glossary#OBP-MCP), [Consent](/glossary#Consent), [Authentication: OAuth 2.0](/glossary#Authentication:-OAuth-2.0). + | +""") + + /////////////////////////////////////////////////////////////////// // NOTE! Some glossary items are generated in ExampleValue.scala ////////////////////////////////////////////////////////////////// diff --git a/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala b/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala index f115f233db..892df63784 100644 --- a/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala +++ b/obp-api/src/main/scala/code/api/util/http4s/AppsPage.scala @@ -19,8 +19,8 @@ object AppsPage { // For each app-key, which probe endpoints it exposes. Rendered as JSON fields // (e.g. "status_url") and HTML links (e.g. [status]) in `probeEndpoints` order. private val appProbes: Map[String, Set[String]] = Map( - "public_obp_portal_url" -> Set("status"), - "public_obp_api_manager_url" -> Set("status"), + "public_obp_portal_url" -> Set("status", "health"), + "public_obp_api_manager_url" -> Set("status", "health"), "public_obp_oidc_url" -> Set("status"), "public_obp_api_url" -> Set("status", "health"), "public_obp_opey_url" -> Set("status", "health"), diff --git a/obp-api/src/main/scala/code/api/util/http4s/IdempotencyMiddleware.scala b/obp-api/src/main/scala/code/api/util/http4s/IdempotencyMiddleware.scala new file mode 100644 index 0000000000..cc9a09c462 --- /dev/null +++ b/obp-api/src/main/scala/code/api/util/http4s/IdempotencyMiddleware.scala @@ -0,0 +1,319 @@ +package code.api.util.http4s + +import cats.data.{Kleisli, OptionT} +import cats.effect._ +import code.api.cache.Redis +import code.util.Helper.MdcLoggable +import net.liftweb.json.{compactRender, parse} +import net.liftweb.json.JsonAST.{JField, JInt, JObject, JString} +import net.liftweb.json.JsonDSL._ +import org.http4s._ +import org.http4s.headers.`Content-Type` +import org.typelevel.ci.CIString +import redis.clients.jedis.Jedis + +import java.security.MessageDigest +import java.util.Base64 + +/** + * Idempotency middleware for http4s mutating requests. + * + * Clients opt in by sending an `Idempotency-Key: ` header on POST, PUT, + * PATCH or DELETE. GET, HEAD and OPTIONS are unaffected. + * + * Behaviour: + * - First request with a given (consumer, key): runs normally; the response + * status, content-type, body and request-body hash are cached for 24h. + * - Replay with same key + same body hash: cached response is returned with + * header `Idempotency-Replay: true` (Stripe convention). + * - Replay with same key + different body hash: 409 Conflict. + * - Concurrent replay while the original is still in flight: 409 Conflict. + * - 5xx responses are NOT cached; clients can retry. + * + * Scope: the key is namespaced by SHA-256 of the consumer id, or — when + * unauthenticated — the Authorization header. This prevents key reuse across + * consumers. + * + * Validation: 8..255 printable-ASCII characters. Anything else → 400. + * + * Storage: Redis via the existing JedisPool. Two keys per request: + * - idem:lock:: → "1" (60s TTL, set with NX) + * - idem:resp:: → JSON envelope (24h TTL) + * + * Resilience: any Redis error is logged and the request is allowed to proceed + * unchanged — the middleware never blocks traffic on cache outages. + * + * The middleware should be installed INSIDE ResourceDocMiddleware so the + * CallContext (and therefore the consumer id) is populated before the scope + * key is computed. + */ +object IdempotencyMiddleware extends MdcLoggable { + + type HttpF[A] = OptionT[IO, A] + + private val IdempotencyKeyHeader = CIString("Idempotency-Key") + private val IdempotencyReplayHeader = CIString("Idempotency-Replay") + private val AuthorizationHeader = CIString("Authorization") + + private val MutatingMethods: Set[Method] = + Set(Method.POST, Method.PUT, Method.PATCH, Method.DELETE) + + private val LockTtlSeconds: Int = 60 + private val ResponseTtlSeconds: Int = 24 * 60 * 60 + + private val MinKeyLength = 8 + private val MaxKeyLength = 255 + + private val LockKeyPrefix = "idem:lock:" + private val ResponseKeyPrefix = "idem:resp:" + + /** + * Wrap routes so that mutating requests carrying an Idempotency-Key header + * are deduplicated. + */ + def apply(routes: HttpRoutes[IO]): HttpRoutes[IO] = + Kleisli[HttpF, Request[IO], Response[IO]] { req => + val keyOpt = req.headers.get(IdempotencyKeyHeader).map(_.head.value) + + if (!MutatingMethods.contains(req.method) || keyOpt.isEmpty) { + routes.run(req) + } else { + val key = keyOpt.get + if (!isValidKey(key)) { + OptionT.liftF(invalidKeyResponse(key)) + } else { + val scope = scopeFor(req) + val bodyHash = sha256Hex(bodyFromCallContextOrEmpty(req)) + val responseKey = ResponseKeyPrefix + scope + ":" + key + val lockKey = LockKeyPrefix + scope + ":" + key + + OptionT.liftF(handle(req, routes, responseKey, lockKey, bodyHash)) + } + } + } + + private def handle( + req: Request[IO], + routes: HttpRoutes[IO], + responseKey: String, + lockKey: String, + requestBodyHash: String + ): IO[Response[IO]] = { + IO.blocking(readResponseKey(responseKey)).attempt.flatMap { + case Right(Some(envelope)) => + if (envelope.requestBodyHash == requestBodyHash) { + IO.pure(rebuildResponse(envelope, replay = true)) + } else { + conflictResponse( + "Idempotency-Key replayed with a different request body. " + + "Use a fresh key for a different request." + ) + } + + case Right(None) => + IO.blocking(tryAcquireLock(lockKey)).attempt.flatMap { + case Right(true) => + runAndCache(req, routes, responseKey, lockKey, requestBodyHash) + case Right(false) => + conflictResponse( + "Idempotent operation already in flight for this Idempotency-Key." + ) + case Left(t) => + logger.warn(s"Idempotency lock unavailable (Redis): ${t.getMessage}") + runRoutes(req, routes) + } + + case Left(t) => + logger.warn(s"Idempotency cache unavailable (Redis): ${t.getMessage}") + runRoutes(req, routes) + } + } + + private def runAndCache( + req: Request[IO], + routes: HttpRoutes[IO], + responseKey: String, + lockKey: String, + requestBodyHash: String + ): IO[Response[IO]] = { + runRoutes(req, routes).flatMap { resp => + // Drain body so we can both cache and re-emit it. + resp.body.compile.toVector.flatMap { vec => + val bodyBytes = vec.toArray + val rebuilt = resp.withBodyStream(fs2.Stream.emits(bodyBytes).covary[IO]) + + val storeOrReleaseLock: IO[Unit] = + if (resp.status.code >= 500) { + // Don't cache transient failures; release the lock so client can retry. + IO.blocking(deleteKey(lockKey)).attempt.map(_ => ()) + } else { + val envelope = Envelope( + status = resp.status.code, + contentType = resp.headers.get(CIString("Content-Type")).map(_.head.value), + bodyB64 = Base64.getEncoder.encodeToString(bodyBytes), + requestBodyHash = requestBodyHash + ) + IO.blocking { + writeResponseKey(responseKey, envelope) + deleteKey(lockKey) + }.attempt.map { e => + e.left.foreach(t => + logger.warn(s"Failed to cache idempotent response: ${t.getMessage}") + ) + () + } + } + storeOrReleaseLock.as(rebuilt) + } + } + } + + private def runRoutes(req: Request[IO], routes: HttpRoutes[IO]): IO[Response[IO]] = + routes.run(req).getOrElseF(IO.pure(Response[IO](Status.NotFound))) + + // ── Validation ───────────────────────────────────────────────────────── + + private def isValidKey(key: String): Boolean = + key.length >= MinKeyLength && + key.length <= MaxKeyLength && + key.forall(c => c >= 0x21 && c <= 0x7E) + + // ── Scope ────────────────────────────────────────────────────────────── + + private def scopeFor(req: Request[IO]): String = { + val ccOpt = req.attributes.lookup(Http4sRequestAttributes.callContextKey) + val raw = ccOpt + .flatMap(_.consumer.map(_.consumerId.get).toOption) + .filter(_.nonEmpty) + .orElse(req.headers.get(AuthorizationHeader).map(_.head.value)) + .getOrElse("anonymous") + sha256Hex(raw).take(16) + } + + // ── Body hash ────────────────────────────────────────────────────────── + + private def bodyFromCallContextOrEmpty(req: Request[IO]): String = + req.attributes + .lookup(Http4sRequestAttributes.callContextKey) + .flatMap(_.httpBody) + .getOrElse("") + + private def sha256Hex(s: String): String = { + val md = MessageDigest.getInstance("SHA-256") + val bytes = md.digest(s.getBytes("UTF-8")) + bytes.map(b => f"$b%02x").mkString + } + + // ── Responses ────────────────────────────────────────────────────────── + + private def invalidKeyResponse(key: String): IO[Response[IO]] = { + val body = compactRender( + ("code" -> 400) ~ + ("message" -> + s"Invalid Idempotency-Key header: must be ${MinKeyLength}..${MaxKeyLength} printable ASCII characters.") + ) + IO.pure( + Response[IO](Status.BadRequest) + .withEntity(body) + .withContentType(`Content-Type`(MediaType.application.json)) + ) + } + + private def conflictResponse(message: String): IO[Response[IO]] = { + val body = compactRender(("code" -> 409) ~ ("message" -> message)) + IO.pure( + Response[IO](Status.Conflict) + .withEntity(body) + .withContentType(`Content-Type`(MediaType.application.json)) + ) + } + + private def rebuildResponse(env: Envelope, replay: Boolean): Response[IO] = { + val bytes = Base64.getDecoder.decode(env.bodyB64) + val status = Status.fromInt(env.status).getOrElse(Status.Ok) + val base = Response[IO](status).withBodyStream(fs2.Stream.emits(bytes).covary[IO]) + val withCt = env.contentType + .flatMap(v => `Content-Type`.parse(v).toOption) + .fold(base)(ct => base.withContentType(ct)) + if (replay) withCt.putHeaders(Header.Raw(IdempotencyReplayHeader, "true")) + else withCt + } + + // ── Storage envelope ─────────────────────────────────────────────────── + + private final case class Envelope( + status: Int, + contentType: Option[String], + bodyB64: String, + requestBodyHash: String + ) + + private def envelopeToJson(env: Envelope): String = { + val obj = + ("status" -> env.status) ~ + ("content_type" -> env.contentType) ~ + ("body_b64" -> env.bodyB64) ~ + ("request_body_hash" -> env.requestBodyHash) + compactRender(obj) + } + + private def envelopeFromJson(s: String): Option[Envelope] = + try { + parse(s) match { + case JObject(fields) => + val map = fields.collect { case JField(k, v) => k -> v }.toMap + for { + status <- map.get("status").collect { case JInt(i) => i.toInt } + body <- map.get("body_b64").collect { case JString(v) => v } + hash <- map.get("request_body_hash").collect { case JString(v) => v } + } yield Envelope( + status = status, + contentType = map.get("content_type").collect { case JString(v) => v }, + bodyB64 = body, + requestBodyHash = hash + ) + case _ => None + } + } catch { + case t: Throwable => + logger.warn(s"Failed to parse idempotency envelope: ${t.getMessage}") + None + } + + // ── Redis primitives (sync; called from IO.blocking) ─────────────────── + + private def withJedis[A](f: Jedis => A): A = { + val jedis = Redis.jedisPool.getResource + try f(jedis) + finally jedis.close() + } + + private def readResponseKey(key: String): Option[Envelope] = + withJedis { j => + Option(j.get(key)).flatMap(envelopeFromJson) + } + + private def writeResponseKey(key: String, env: Envelope): Unit = + withJedis { j => + j.setex(key, ResponseTtlSeconds, envelopeToJson(env)) + () + } + + /** + * SETNX + EXPIRE. The brief window between the two has the lock without a + * TTL, but the key value is only ever a sentinel and the next overwrite (or + * crash recovery via the 60s TTL once expire lands) resolves it. + */ + private def tryAcquireLock(key: String): Boolean = + withJedis { j => + val acquired = j.setnx(key, "1") == 1L + if (acquired) j.expire(key, LockTtlSeconds) + acquired + } + + private def deleteKey(key: String): Unit = + withJedis { j => + j.del(key) + () + } +} diff --git a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala index 8d448ef6f4..2aeddfbfeb 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/Http4s700.scala @@ -11,7 +11,7 @@ import code.api.util.{APIUtil, ApiRole, ApiVersionUtils, CallContext, CustomJson import code.api.util.ApiRole.{canCreateEntitlementAtAnyBank, canCreateEntitlementAtOneBank, canDeleteEntitlementAtAnyBank, canGetAnyUser, canGetCacheConfig, canGetCacheInfo, canGetCacheNamespaces, canGetCardsForBank, canGetConnectorHealth, canGetCustomersAtOneBank, canGetDatabasePoolInfo, canGetMigrations} import code.api.util.ApiTag._ import code.api.util.ErrorMessages._ -import code.api.util.http4s.{ErrorResponseConverter, Http4sRequestAttributes, RequestScopeConnection, ResourceDocMiddleware} +import code.api.util.http4s.{ErrorResponseConverter, Http4sRequestAttributes, IdempotencyMiddleware, RequestScopeConnection, ResourceDocMiddleware} import code.api.util.http4s.Http4sRequestAttributes.{EndpointHelpers, RequestOps} import code.api.util.newstyle.ViewNewStyle import code.api.v1_3_0.JSONFactory1_3_0 @@ -609,6 +609,44 @@ object Http4s700 { http4sPartialFunction = Some(getConnectors) ) + // Route: GET /obp/v7.0.0/api/error-messages + val getErrorMessages: HttpRoutes[IO] = HttpRoutes.of[IO] { + case req @ GET -> `prefixPath` / "api" / "error-messages" => + EndpointHelpers.executeAndRespond(req) { _ => + Future.successful(ListResult("error_messages", JSONFactory700.errorMessagesCatalog)) + } + } + + resourceDocs += ResourceDoc( + null, + implementedInApiVersion, + nameOf(getErrorMessages), + "GET", + "/api/error-messages", + "Get Error Messages", + """Returns the catalog of OBP error codes and messages defined in this API instance. + | + |Each entry has the OBP error code (e.g. `OBP-00001`), the internal name of the + |constant, and the human-readable message text. + | + |The catalog is derived by reflecting over `ErrorMessages` at first access and + |cached for the lifetime of the server. + | + |No Authentication is Required.""".stripMargin, + EmptyBody, + ListResult( + "error_messages", + List(JSONFactory700.ErrorMessageEntryJsonV700( + code = "OBP-00001", + name = "HostnameNotSpecified", + message = "Hostname not specified. Could not get hostname from Props." + )) + ), + List(UnknownError), + apiTagDocumentation :: apiTagApi :: Nil, + http4sPartialFunction = Some(getErrorMessages) + ) + // Route: GET /obp/v7.0.0/providers val getProviders: HttpRoutes[IO] = HttpRoutes.of[IO] { case req @ GET -> `prefixPath` / "providers" => @@ -952,7 +990,7 @@ object Http4s700 { updated_at = "2026-04-15T10:30:00Z" ), List(InvalidJsonFormat, InvalidOfferType, InvalidTradingAmount, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagTrading :: Nil, + apiTagTrading :: apiTagTrade :: Nil, http4sPartialFunction = Some(createTradingOffer) ) @@ -1005,7 +1043,7 @@ object Http4s700 { updated_at = "2026-04-15T10:30:00Z" ), List(OfferNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagTrading :: Nil, + apiTagTrading :: apiTagTrade :: Nil, http4sPartialFunction = Some(getTradingOffer) ) @@ -1082,7 +1120,7 @@ object Http4s700 { ) ), List($AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagTrading :: Nil, + apiTagTrading :: apiTagTrade :: Nil, http4sPartialFunction = Some(getTradingOffers) ) @@ -1117,7 +1155,7 @@ object Http4s700 { status = "cancelled" ), List(OfferNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagTrading :: Nil, + apiTagTrading :: apiTagTrade :: Nil, http4sPartialFunction = Some(cancelTradingOffer) ) @@ -1204,7 +1242,7 @@ object Http4s700 { updated_at = "2026-04-16T00:30:00Z" ), List(InvalidJsonFormat, InvalidOrderSide, InvalidTradingAmount, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(createMarketOrder) ) @@ -1253,7 +1291,7 @@ object Http4s700 { updated_at = "2026-04-16T00:30:00Z" ), List(OrderNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(getMarketOrder) ) @@ -1304,7 +1342,7 @@ object Http4s700 { updated_at = "2026-04-16T00:35:00Z" ), List(OrderNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(cancelMarketOrder) ) @@ -1375,7 +1413,7 @@ object Http4s700 { created_at = "2026-04-16T00:40:00Z" ), List(InvalidJsonFormat, InvalidMatchParameters, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(createMarketMatch) ) @@ -1423,7 +1461,7 @@ object Http4s700 { created_at = "2026-04-16T00:40:00Z" ), List(TradeNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(getMarketTrade) ) @@ -1474,7 +1512,7 @@ object Http4s700 { completed_at = None ), List(InvalidJsonFormat, SettlementFailed, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(requestSettlement) ) @@ -1552,7 +1590,7 @@ object Http4s700 { // created_at = "2026-04-16T00:50:00Z" // ), // List(InvalidJsonFormat, InvalidTradingAmount, InvalidMatchParameters, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), -// apiTagMarket :: Nil, +// apiTagTrading :: apiTagMarket :: Nil, // http4sPartialFunction = Some(notifyDeposit) // ) @@ -1622,7 +1660,7 @@ object Http4s700 { created_at = "2026-04-16T00:55:00Z" ), List(InvalidJsonFormat, InvalidTradingAmount, WithdrawalFailed, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), - apiTagMarket :: Nil, + apiTagTrading :: apiTagMarket :: Nil, http4sPartialFunction = Some(requestWithdrawal) ) @@ -1701,7 +1739,7 @@ object Http4s700 { // updated_at = "2026-04-17T10:00:00Z" // ), // List(InvalidJsonFormat, InvalidTradingAmount, CreatePaymentAuthError, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), -// apiTagMarket :: Nil, +// apiTagTrading :: apiTagMarket :: Nil, // http4sPartialFunction = Some(createPaymentAuth) // ) // @@ -1758,7 +1796,7 @@ object Http4s700 { // updated_at = "2026-04-17T10:05:00Z" // ), // List(PaymentAuthNotFound, InvalidPaymentAuthState, PaymentAuthAlreadyCaptured, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), -// apiTagMarket :: Nil, +// apiTagTrading :: apiTagMarket :: Nil, // http4sPartialFunction = Some(capturePaymentAuth) // ) // @@ -1815,7 +1853,7 @@ object Http4s700 { // updated_at = "2026-04-17T10:10:00Z" // ), // List(PaymentAuthNotFound, InvalidPaymentAuthState, PaymentAuthAlreadyReleased, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), -// apiTagMarket :: Nil, +// apiTagTrading :: apiTagMarket :: Nil, // http4sPartialFunction = Some(releasePaymentAuth) // ) // @@ -1869,7 +1907,7 @@ object Http4s700 { // updated_at = "2026-04-17T10:00:00Z" // ), // List(PaymentAuthNotFound, $AuthenticatedUserIsRequired, $BankNotFound, $BankAccountNotFound, UnknownError), -// apiTagMarket :: Nil, +// apiTagTrading :: apiTagMarket :: Nil, // http4sPartialFunction = Some(getPaymentAuth) // ) @@ -2161,9 +2199,12 @@ object Http4s700 { } } - // Routes wrapped with ResourceDocMiddleware for automatic validation + // Routes wrapped with ResourceDocMiddleware for automatic validation. + // IdempotencyMiddleware is nested inside so that auth/CallContext is populated + // before the idempotency scope key is computed; on a cache hit the inner + // routes (and any DB transaction) are skipped. val allRoutesWithMiddleware: HttpRoutes[IO] = - ResourceDocMiddleware.apply(resourceDocs)(allRoutes) + ResourceDocMiddleware.apply(resourceDocs)(IdempotencyMiddleware(allRoutes)) } // Routes with ResourceDocMiddleware - provides automatic validation based on ResourceDoc metadata diff --git a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala index 852e1f7b5f..008658affc 100644 --- a/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala +++ b/obp-api/src/main/scala/code/api/v7_0_0/JSONFactory7.0.0.scala @@ -2,6 +2,7 @@ package code.api.v7_0_0 import code.api.Constant import code.api.util.APIUtil +import code.api.util.ErrorMessages import code.api.util.ErrorMessages.MandatoryPropertyIsNotSet import code.api.v4_0_0.{EnergySource400, HostedAt400, HostedBy400} import code.util.Helper.MdcLoggable @@ -9,6 +10,31 @@ import com.openbankproject.commons.util.ApiVersion object JSONFactory700 extends MdcLoggable with code.api.util.CustomJsonFormats { + case class ErrorMessageEntryJsonV700(code: String, name: String, message: String) + + // Cached for server lifetime: ErrorMessages is a static catalog of `val X = "OBP-NNNNN: ..."` + // strings, so reflecting over it once at first access is sufficient. Filters: + // - only String-typed fields (skips synthetic lazy-val bitmaps and helper defs) + // - only values starting with "OBP-" (skips helper strings that don't carry a code) + lazy val errorMessagesCatalog: List[ErrorMessageEntryJsonV700] = { + ErrorMessages.getClass.getDeclaredFields.toList + .filter(f => f.getType == classOf[String]) + .flatMap { f => + f.setAccessible(true) + Option(f.get(ErrorMessages)).collect { case s: String => s } + .filter(_.startsWith("OBP-")) + .map { msg => + val colonIdx = msg.indexOf(':') + val (code, text) = + if (colonIdx > 0) (msg.substring(0, colonIdx), msg.substring(colonIdx + 1).trim) + else ("", msg) + ErrorMessageEntryJsonV700(code = code, name = f.getName, message = text) + } + } + .sortBy(e => (e.code, e.name)) + } + + case class APIInfoJsonV700( version: String, version_status: String, diff --git a/security_check.sh b/security_check.sh new file mode 100755 index 0000000000..5af5867d9c --- /dev/null +++ b/security_check.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# security_check.sh — lightweight static scan for high-signal security issues. +# +# Usage: +# ./security_check.sh # scan src + config, exit 1 on any HIGH finding +# ./security_check.sh --all # include tests +# ./security_check.sh --strict # exit 1 on any finding (HIGH or MEDIUM) +# +# Scope: grep-based heuristics. False positives are expected — triage findings, +# don't treat a clean run as proof of no vulnerabilities. Pair with real tools +# (Semgrep, OWASP Dependency-Check, gitleaks) for depth. +# +# Requires GNU grep (or compatible) with -P (PCRE) support — standard on Linux. + +set -u + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +cd "$ROOT" + +INCLUDE_TESTS=0 +STRICT=0 +for arg in "$@"; do + case "$arg" in + --all) INCLUDE_TESTS=1 ;; + --strict) STRICT=1 ;; + -h|--help) sed -n '2,12p' "$0"; exit 0 ;; + *) echo "unknown flag: $arg" >&2; exit 2 ;; + esac +done + +LOG_FILE="${ROOT}/security_check.log" +# Tee stdout+stderr to the log file. Process substitution keeps terminal output live. +exec > >(tee "$LOG_FILE") 2>&1 + +printf 'security_check.sh run at %s (args: %s)\n' "$(date -Iseconds)" "${*:-}" + +SCAN_PATHS=(obp-api/src/main obp-commons/src/main obp-http4s-runner/src/main) +if [[ $INCLUDE_TESTS -eq 1 ]]; then + SCAN_PATHS+=(obp-api/src/test obp-commons/src/test) +fi +CONFIG_PATHS=(obp-api/src/main/resources) + +GREP_EXCLUDES=( + --exclude-dir=target + --exclude-dir=.git + --exclude-dir=node_modules + --exclude='*.min.js' +) + +# Verify grep -P works. ugrep, GNU grep (built with PCRE), and modern BSD grep all support it. +if ! echo x | grep -P 'x' >/dev/null 2>&1; then + echo "error: grep -P (PCRE) not supported by $(grep --version | head -1)" >&2 + echo "install GNU grep, ripgrep, or ugrep." >&2 + exit 2 +fi + +HIGH=0 +MEDIUM=0 + +# scan <regex> [-- <path>...] +# default paths are SCAN_PATHS. To override: pass `--` then the paths. +scan() { + local sev="$1" title="$2" pattern="$3"; shift 3 + local paths=("${SCAN_PATHS[@]}") + if [[ "${1:-}" == "--" ]]; then + shift + paths=("$@") + fi + + local matches + matches=$(grep -rnHP "${GREP_EXCLUDES[@]}" -e "$pattern" "${paths[@]}" 2>/dev/null || true) + [[ -z "$matches" ]] && return 0 + + local count; count=$(printf '%s\n' "$matches" | wc -l | tr -d ' ') + printf '\n[%s] %s — %s match(es)\n' "$sev" "$title" "$count" + printf '%s\n' "$matches" | sed 's/^/ /' + if [[ "$sev" == "HIGH" ]]; then HIGH=$((HIGH+count)); else MEDIUM=$((MEDIUM+count)); fi +} + +echo "security_check.sh — scanning ${SCAN_PATHS[*]}" + +# --- Secrets & credentials ---------------------------------------------- +# "password" literals are excluded by design — treated as a red herring in this codebase. + +scan HIGH "Hardcoded secret/token literal" \ + '(?i)(api[_-]?key|secret|auth[_-]?token|access[_-]?token|bearer)\s*[:=]\s*"[^"$\{<]{8,}"' + +scan HIGH "AWS-style access key" \ + 'AKIA[0-9A-Z]{16}' + +# Excludes the string literal "-----BEGIN PRIVATE KEY-----" — we want embedded PEM, not code that references the header. +scan HIGH "Private key block" \ + '(?<!")-----BEGIN (RSA |EC |DSA |OPENSSH |PGP )?PRIVATE KEY-----(?!")' + +scan MEDIUM "Props file with literal secret/api key" \ + '(?i)^[^#]*(secret|api[_-]?key)\s*=\s*\S+' \ + -- "${CONFIG_PATHS[@]}" + +# --- Weak crypto --------------------------------------------------------- +scan HIGH "Weak hash (MD5/SHA1) used for security" \ + '(?i)MessageDigest\.getInstance\(\s*"(MD5|SHA-?1)"' + +scan HIGH "Weak cipher (DES / RC4 / ECB)" \ + '(?i)Cipher\.getInstance\(\s*"(DES|RC4|[^"]*?/ECB/)' + +scan MEDIUM "java.util.Random used (not SecureRandom)" \ + '\bnew\s+java\.util\.Random\b|\bnew\s+Random\s*\(' + +# --- TLS weakening ------------------------------------------------------- +scan HIGH "Trust-all / disabled cert validation" \ + 'X509TrustManager|TrustAllCerts|setHostnameVerifier\s*\(\s*\(.*\)\s*->\s*true|NoopHostnameVerifier' + +# Excludes common false-positive hosts in fixtures and XML namespace URIs. +scan MEDIUM "HTTP (non-TLS) URL in source" \ + '"http://(?!localhost|127\.0\.0\.1|0\.0\.0\.0|example\.com|www\.example\.com|www\.w3\.org|schemas\.xmlsoap|xmlns)[^"]+"' + +# --- Injection risks ----------------------------------------------------- +scan HIGH "SQL built with string concat" \ + '(?i)"(SELECT|INSERT|UPDATE|DELETE|DROP)\b[^"]*"\s*\+\s*\w' + +scan MEDIUM "Runtime.exec / ProcessBuilder with variable" \ + 'Runtime\.getRuntime\(\)\.exec\s*\(\s*\w|new\s+ProcessBuilder\s*\(\s*\w' + +# --- Deserialization / reflection hazards -------------------------------- +scan MEDIUM "Java native deserialization" \ + 'new\s+ObjectInputStream\s*\(|\breadObject\s*\(' + +# --- Summary ------------------------------------------------------------- +printf '\n---\nSummary: %d HIGH, %d MEDIUM\n' "$HIGH" "$MEDIUM" +printf 'Log written to: %s\n' "$LOG_FILE" + +if [[ $HIGH -gt 0 ]]; then + exit 1 +fi +if [[ $STRICT -eq 1 && $MEDIUM -gt 0 ]]; then + exit 1 +fi +exit 0