Information Retrieval

Resources

• Introduction to Information Retrieval. Selected chapters:
  • Language models for information retrieval
  • Text classification and Naive Bayes
  • Vector space classification
  • Matrix decompositions and latent semantic indexing
  • Evaluation in information retrieval
• What is information retrieval IBM
• Doing RAG Vector search is not enough - use BM25 for keywords and hard lookups, combine with vector search (semantics) and re-rank (vector search will always find results; re-ranking allows you to impose a cut-off based on a relevance score which allows you to avoid e.g. too long context or lost in the middle effects)
• Resources on Okapi BM25:
  • Understanding Okapi BM25 A Guide to Modern Information Retrieval - Association of Data Scientists
  • What Is BM25 (Best Match 25) Full Breakdown - Luigi’s Box
  • Relevance in keyword search (BM25 scoring)