Avoiding duplicates?

[coldino]

Love the simplicity of this project in use, but I've come across an issue - how to avoid inserting duplicates?

It appears I would need to bypass the client and check each entry with the underlying connection before db.add. This just feels out of place with the simplicity of the rest of the interface, particularly as it would then need knowledge of the underlying schema and be brittle to future changes.

My particular usecase utilises text to store long hashes, so it would be ideal to treat them as unique - and I assume the same would be true for the plain text uses in the examples too.

[atasoglu]

Hey @coldino, sorry for the late reply on this one! Really appreciate the kind words and the feedback. You're right, having to go around the client and deal with the underlying schema just to avoid duplicates doesn't fit with the simplicity of the rest of the interface.

I've added native support for this with two small API additions. The changes are on the unique-text branch if you want to try them out.

1. unique_text flag on create_table

db.create_table(dim=384, unique_text=True)

This creates a UNIQUE index on the text column.

2. on_conflict parameter on add

Three strategies to choose from:

• "error" (default): current behavior, raises on duplicate
• "ignore": silently skips duplicates
• "replace": upserts, meaning it updates the metadata and embedding of the existing record while keeping its original rowid

For your long-hash use case it would look something like:

db.create_table(dim=384, unique_text=True)

# skip already-seen hashes
db.add(texts=hashes, embeddings=vectors, on_conflict="ignore")

# or, if you want to update the embedding when the hash already exists
db.add(texts=hashes, embeddings=vectors, on_conflict="replace")

Everything stays in sync under the hood. The existing triggers handle the _vec table automatically for all three modes, so no manual cleanup needed.

Both parameters are fully optional and default to the previous behavior, so nothing breaks for existing users.

Let me know if this covers your use case or if you had something different in mind!