This is key. Sure you do not need a specialized db to iterate over <10M vectors at a time, as the results show this is the order at which the latency (for this implementation) starts getting too big for a real time system: O(100ms). You can get clever with sharding and such to scale out - calculating cosine similarity and finding the top K is classic mapreduce - but that's just spreading the compute over more computers i.e. for >100M 256 dimension vector like the author chose, you'd be at several seconds of CPU time for each similarity calculation.
I don't think the vector database companies are targeting, at least with the expectation of making money now (maybe later if some grow a lot), people with that few embeddings. They are targeting use cases with WAY more vectors - think about how many vectors chatgpt must be generating for however many conversations it's producing just today, and if the technology continues to improve and use cases grow, how many other business may be generating even more embeddings. In the same way as it doesn't make sense to use Spark on a 10000 cell csv, or to buy a Tank to haul chips out of a grocer store, it doesn't make sense to use a dedicated database under a certain scale.
> Additionally, you get to save yourself the complication of standing up a vector database and waiting the ~100 seconds to index those million embeddings.
Isn't that a one time cost with the subsequent upserts amortized? Or they could even be done outside the hotpath in batches: if the db is just computing similarities to stored vectors, you could add a ton of new vectors to a new db instance/replica and then swap them.
I'm sorry if I'm being overly critical, but to me this seems like exactly what happens any time someone releases a product or technology that improves performance for really big workloads (so, almost every big data system or task-specialized db), and everybody says the technology is stupid because they don't need it at their startup with 1% of the data the technology begins to be useful for. Sure, but others have 100x your data and would benefit from it.
Let me just point out one example for why 10M vectors is actually quite small for some use cases: English wikipedia has 6M articles and it would take much more than one 256 dim vector to accurately encode some of those pages. Even with a single such vector per article you're at 0.3 cpu-s for each nearest-K calculation on an arbitrary page. And that's one admittedly large portion of an amdmittedly large website, but still a fraction of the datasize you'd get from crawling the web, looking at discord/fb/instagram/snapchat messages, reddit comments, etc.
Hi, author here. I totally agree with you that, for large scale, you're going to need a vector database. My hope is more to help people avoid scenarios like the one in this comment: https://news.ycombinator.com/item?id=35552303 Tangentially, I really like the approach that haystack has taken, where they allow you to slot in whichever document store you want, and that document store can scale from in-memory, to sqlite, to postgres, to pinecone https://docs.haystack.deepset.ai/docs/document_store
In terms of the one-time cost of indexing, you're totally right! Although, one thing to call out is that you will have to re-index every time you change your embedding model, such as for fine-tuning. I don't have a good handle on how prevalent this is, though.
I fully agree with you - but also with the article.
If I were to index my personal knowledge base, where I have on the order of 1000 documents, I'd need at most 100k embeddings (generously assuming 100 per document, but that's an overestimate).
I'm literally talking to a company at the moment who's interested in a chatbot to talk to their internal knowledge base. They have... 100 documents, around 40 pages long. Again, this will easily fit within 1 million embeddings.
And using vector databases comes with tradeoffs: you will get worse recall (aka you will sometimes miss the most relevant document).
This is key. Sure you do not need a specialized db to iterate over <10M vectors at a time, as the results show this is the order at which the latency (for this implementation) starts getting too big for a real time system: O(100ms). You can get clever with sharding and such to scale out - calculating cosine similarity and finding the top K is classic mapreduce - but that's just spreading the compute over more computers i.e. for >100M 256 dimension vector like the author chose, you'd be at several seconds of CPU time for each similarity calculation.
I don't think the vector database companies are targeting, at least with the expectation of making money now (maybe later if some grow a lot), people with that few embeddings. They are targeting use cases with WAY more vectors - think about how many vectors chatgpt must be generating for however many conversations it's producing just today, and if the technology continues to improve and use cases grow, how many other business may be generating even more embeddings. In the same way as it doesn't make sense to use Spark on a 10000 cell csv, or to buy a Tank to haul chips out of a grocer store, it doesn't make sense to use a dedicated database under a certain scale.
> Additionally, you get to save yourself the complication of standing up a vector database and waiting the ~100 seconds to index those million embeddings.
Isn't that a one time cost with the subsequent upserts amortized? Or they could even be done outside the hotpath in batches: if the db is just computing similarities to stored vectors, you could add a ton of new vectors to a new db instance/replica and then swap them.
I'm sorry if I'm being overly critical, but to me this seems like exactly what happens any time someone releases a product or technology that improves performance for really big workloads (so, almost every big data system or task-specialized db), and everybody says the technology is stupid because they don't need it at their startup with 1% of the data the technology begins to be useful for. Sure, but others have 100x your data and would benefit from it.
Let me just point out one example for why 10M vectors is actually quite small for some use cases: English wikipedia has 6M articles and it would take much more than one 256 dim vector to accurately encode some of those pages. Even with a single such vector per article you're at 0.3 cpu-s for each nearest-K calculation on an arbitrary page. And that's one admittedly large portion of an amdmittedly large website, but still a fraction of the datasize you'd get from crawling the web, looking at discord/fb/instagram/snapchat messages, reddit comments, etc.