Learning-to-Rank News Recommender
Hybrid retrieval + LambdaMART ranking on Microsoft MIND with position-bias correction
An end-to-end news recommendation system built on the Microsoft MIND dataset — 50K users and real impression logs — covering the full industrial ranking stack from candidate generation to diversity-aware re-ranking.
The retrieval stage combines BM25 lexical search with FAISS approximate nearest-neighbour search for hybrid candidate generation. Candidates are then scored by a LightGBM LambdaMART learning-to-rank model, and the final slate is re-ranked with MMR (Maximal Marginal Relevance) to balance relevance against diversity.
A core focus of the project is position bias: clicks in impression logs are biased toward higher-ranked items, so training naively on clicks teaches the model the old ranker’s mistakes. I corrected for this with inverse-propensity weighting (IPW), recovering ~6% NDCG@10 (0.745 → 0.802) over naive click training in a controlled synthetic experiment.
Stack: LightGBM (LambdaMART) · FAISS · BM25 · Python · ranking metrics (NDCG, MRR)
Source code: github.com/smasoudrezvani/mind-ltr