Two-stage recommendation engine

ML prototype

A two-stage recommendation prototype combining semantic candidate retrieval with learning-to-rank over MovieLens data.

Problem

This prototype separates finding plausible movies from deciding how to order them, using MovieLens data as the setting for that distinction.

Candidate retrieval narrows the available choices, while the ranking stage scores the retrieved movies using additional user and movie signals.

Implemented approach

The embedding script combines movie titles and genres into text, encodes that text with a sentence-embedding model and normalizes the resulting vectors.

At serving time, a user's rating history supplies a weighted combination of movie vectors for the candidate query.

OpenSearch retrieves candidates, then the API builds feature vectors and obtains scores from an XGBoost learning-to-rank model.

The serving code can blend the ranking score with the retrieval score before sorting the result list.

Unknown users receive a popularity-based fallback rather than a personalized ranking inferred from history that isn't available.

Implementation overview
  1. Rating history
  2. User representation
  3. Candidate retrieval
  4. Feature-based ranking
  5. Recommendations

Trade-offs

Retrieval recall and ranking

Separating retrieval from ranking limits the second stage to the returned candidates, so the ranker can't recover a useful movie that retrieval missed.

This makes candidate selection a separate quality question from how well the ranker orders the movies it receives.

Evaluation leakage and score scales

A result would need to be read alongside the data split and candidate-selection procedure before it could support a broader quality claim.

Evaluation leakage and the comparability of retrieval and ranking score scales remain cautions, not verified resolutions.

Evaluation and limits

The ranker is configured for a pairwise learning objective and an NDCG evaluation metric, keeping the training objective distinct from the measure used to assess ordering.

Its training interface groups examples by user and accepts a separate validation dataset, so validation can monitor rankings within those groups.

That describes an evaluation mechanism, not a reproduced benchmark: no measured score or real-time performance result is claimed here.

This remains an ML prototype, with no claim of production deployment or observed user impact.