🚀 Optimizing a RAG pipeline with a semantic reranker

[cobalt axle]

When building an AI agent powered by Retrieval-Augmented Generation (RAG), you quickly face a tough trade-off:
👉 Retrieve too few documents → you miss the key info.
👉 Retrieve too many → your LLM drowns in irrelevant context.
The solution? A semantic reranker that reorders results and keeps only the passages that truly matter.

In my latest article, I share:
🔹 The “recall vs context window” problem
🔹 Why combining vector search with BM25 boosts accuracy
🔹 The details of my open-source C# implementation
🔹 A concrete NLP pipeline with tokenization, lemmatization, and stop-words
🔹 The improvements in relevance and token cost

👉 Open-source project on GitHub: https://lnkd.in/dJjp59zk
👉 Full article here: https://lnkd.in/db7TDXh6

[elder kindle]

I am a Masters Data Science graduate from NJIT with a Thesis in Drug Discovery using zero shot and few shot prompting with LLMs.

I’ve been actively looking for ML Engineering opportunities and wanted to know if anyone came across any or if there’s a more efficient way to job hunt.

Please connect with me if you are interested and I’ll share my resume with you.