Skip to content

EmbeddingSimilarity class-level embedding cache ignores model and prefix — cross-model cache poisoning #211

Description

EmbeddingSimilarity._CACHE is a class attribute keyed only by the normalized input text (py/autoevals/string.py):

_CACHE = {}          # class-level, shared across all instances
...
def _embed(self, value):
    value = normalize_value(value, maybe_object=False)
    with self._CACHE_LOCK:
        if value in self._CACHE:
            return self._CACHE[value]
    result = run_cached_request(..., input=f"{self.prefix}{value}", **self.extra_args)

Two consequences:

  1. Cross-model poisoning: two scorers configured with different embedding models share cache entries — the second scorer silently reuses vectors produced by the first scorer's model, producing wrong similarity scores with no error.
  2. Prefix collisions: the request embeds f"{self.prefix}{value}" but the cache key is the un-prefixed value, so instances with different prefix values collide the same way.

Suggested fix: key the cache on (model, prefix, value) (or the full request), e.g. (self.extra_args.get("model"), self.prefix, value).

Found while writing the regression tests for #166 (PR #209) — the tests had to use unique input strings to avoid the embeddings API call being skipped entirely.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions