🆕 The new OpenAI embedding models are cheaper and better – what’s not to love? Well, let’s dig a bit deeper. 🤔
🔍 What exactly are embedding models? For each paragraph in your documents, these models create a numerical representation of its meaning. We then use them for efficient searching within documents or for grouping similar documents together (a process known as clustering).
🌟 The new models excel at capturing the essence of text:
- The smaller model is not only more cost-effective but also superior in understanding meaning compared to its predecessor.
- There’s also a new, larger model that is even better at capturing meaning.
🧠 So, using the new models seems like a no-brainer, right?
⚠️ Here is the catch: when switching to a different model, you must recreate the embeddings for all the paragraphs of all your documents. This might be low-cost, but
- it can be time-consuming if you’re dealing with millions of documents and
- your application will behave differently so you need to test. 📚
👉 This also points to a more long term challenge of evolving our AI systems. What happens when OpenAI stops supporting the embedding you have chosen? Which begs the questions: wouldn’t it be better to take control of the embeddings by running our own model? Unfortunately it is not clear that there is an open source of the same quality as OpenAI. yet…
