How LSH Random Projection works in search (+Python)

Locality sensitive hashing (LSH) is a widely popular technique used in approximate similarity search. The solution to efficient similarity search is a profitable one - it is at the core of several billion (and even trillion) dollar companies. The problem with similarity search is scale. Many companies deal with millions-to-billions of data points every single day. Given a billion data points, is it feasible to compare all of them with every search? Further, many companies are not performing single searches - Google deals with more than 3.8 million searches every minute. Billions of data points combined with high-frequency searches are problematic - and we haven't considered the dimensionality nor the similarity function itself. Clearly, an exhaustive search across all data points is unrealistic for larger datasets. The solution to searching impossibly huge datasets? Approximate search. Rather than exhaustively comparing every pair, we approximate - restricting the search scope only to high probability matches. 🌲 Pinecone article: https://www.pinecone.io/learn/locality-sensitive-hashing-random-projection/ Download Sift1M: https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf IndexLSH for Fast Similarity Search in Faiss: https://youtu.be/ZLfdQq_u7Eo 🤖 70% Discount on the NLP With Transformers in Python course: https://bit.ly/3DFvvY5 🎉 Sign-up For New Articles Every Week on Medium! https://medium.com/@jamescalam/membership 👾 Discord: https://discord.gg/c5QtDB9RAP 🕹️ Free AI-Powered Code Refactoring with Sourcery: https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff