🆕 The new OpenAI embedding models are cheaper and better – what’s not to love? Well, let’s dig a bit deeper. 🤔

🔍 What exactly are embedding models? For each paragraph in your documents, these models create a numerical representation of its meaning. We then use them for efficient searching within documents or for grouping similar documents together (a process known as clustering).

🌟 The new models excel at capturing the essence of text:

  • The smaller model is not only more cost-effective but also superior in understanding meaning compared to its predecessor.
  • There’s also a new, larger model that is even better at capturing meaning.

🧠 So, using the new models seems like a no-brainer, right?

⚠️ Here is the catch: when switching to a different model, you must recreate the embeddings for all the paragraphs of all your documents. This might be low-cost, but

  • it can be time-consuming if you’re dealing with millions of documents and
  • your application will behave differently so you need to test. 📚

👉 This also points to a more long term challenge of evolving our AI systems. What happens when OpenAI stops supporting the embedding you have chosen? Which begs the questions: wouldn’t it be better to take control of the embeddings by running our own model? Unfortunately it is not clear that there is an open source of the same quality as OpenAI. yet…

The End is Never Near