In this tutorial, we will first convert our documents into vectors using Google’s “Universal Search Encoder (USE)” model and then index them (+ store on disk) using Elasticsearch.
Then during retrieval, we will use the same USE model to convert the query into a vector representation (of length 512) and use Elasticsearch to get most similar vectors.
Elasticsearch is going to match the given query vector to all vectors stored in its database which is not efficient and scalable. So we will experiment with three approximate nearest neighbor techniques to make the retrieval process scalable.
We will use MovieLens-25m dataset for this tutorial.
As medium still doesn’t support Jupyter notebooks in its native form, so we started using fastpages to share notebook based tutorials on GitHub hosted pages.
The complete tutorial (+code) can be found here.