Filtering, not benchmarks, decides your vector database
- RAG
- On-Premise
Vector database comparisons all run along the same axis: queries per second, latency, index type.
On our deployment none of those numbers decided anything. Three other things did: how filtering works, where the data is allowed to sit, and what an update costs.
A corporate query is never only semantic
Nobody asks a knowledge base "what is the backup policy". They ask "what is the backup policy for this customer's production environment".
So every query has a semantic half and a hard-constraint half. The second is not up for negotiation: another customer's document cannot be part of a correct answer.
This is exactly where systems part ways. Is the filter applied during the search or after it.
Post-filtering is simple and brittle. Find the nearest k documents, throw out the ones that fail the constraint. Narrow constraint, and you can end up with nothing - the system reports no results while a perfectly good document sits in the index.
Filtering during the search is the correct behaviour and the harder one to build: the index has to traverse with the filter in hand.
You will not see that difference in a benchmark table, because benchmarks measure unfiltered queries.
A small test on our own data
To decide, we set up something small with real documents, real constraints and real questions. What we looked at was not queries per second.
| Question | Why it matters |
|---|---|
| Does a narrow filter still return anything | A system that empties as constraints tighten is useless |
| How much latency does filtering add | Unfiltered speed is misleading |
| What happens when a document changes | Full reindex or in-place update |
| Does deletion actually delete | Mandatory for retention policies |
The last row mattered more than we expected. When a document has to go, its vector has to genuinely go with it, and that does not work identically everywhere.
Where it runs beats how fast it runs
In an environment where data cannot leave the building, hosted services drop out before the comparison starts. That narrows the field a lot.
Among self-hosted options operational load takes over: is this another service to run, how does it get backed up, does it fit the monitoring you already have.
There is an option that tends to get skipped here. At moderate data volumes, running vector search on the PostgreSQL you already operate can come out cheaper than standing up a separate service. The constraints already live on the SQL side, filtering happens during the search naturally, backup and authorisation come from the setup you already maintain.
Not true at every scale, obviously. But the cost of operating a separate vector service appears in no comparison table.
The update profile is the quiet cost
The subject comparisons almost never touch is how often the data changes.
With a static document set you build the index once and you are done. A corporate knowledge base is not static: procedures get revised, systems appear, old ones get retired.
The question is how much work a single changed document causes. Only that document's chunks, or something broader.
Changing the embedding model means a full reindex in every case. That makes the embedding model a more permanent decision than the database - and one worth more care.
Latency numbers played almost no part in our choice. What decided it was whether filtering happens during the search, where the data can sit, and what updates cost.
The most useful thing to do before choosing is to build something small with your own constraints. An unfiltered benchmark measures a question you will never ask.