Methodology

Introduction

Every week, several posts go up on Reddit, X, and other platforms sharing Fantasy Premier League (FPL) predictions from different sources. However, there's not much information about their historical accuracy. The goal of this benchmark is to fairly evaluate the many prediction sources available to the FPL community.

The Leaderboard

Before each gameweek, each source's predictions are used to select a "Free Hit"-style squad using the following rules:

• Each squad consists of 15 players: 2 goalkeepers, 5 defenders, 5 midfielders, and 3 forwards.
• The total value of each squad cannot exceed £100 million.
• At most 3 players can be selected from a single Premier League team.
• Eleven players are selected for the starting lineup. This starting 11 must include exactly 1 goalkeeper, at least 3 defenders, and at least 1 forward.
• A captain and a vice-captain are selected from the starting 11. The captain's points will be doubled. If the selected captain plays no minutes in the gameweek, the vice-captain's points will be doubled instead.
• The non-starting players form an ordered bench. Players in the starting 11 who don't play in the gameweek will be automatically substituted with the next legal bench player.

To select the captain and vice-captain, and to determine the bench order, each squad role is assigned a points multiplier:

• Captain: ×2
• Vice-captain: ×1.0322
• Other starting players: ×1
• Bench goalkeeper: ×0.0194
• First outfield bench player: ×0.2603
• Second outfield bench player: ×0.0751
• Third outfield bench player: ×0.0165

These multipliers estimate the expected contribution of each role based on typical player appearance rates. For example, the first outfield bench slot is more likely to be activated than the third one, and so has a higher multiplier. For each source, the squad that maximizes the predicted total points, including multipliers, is selected. After each gameweek, each squad is scored using official FPL points and scoring rules.

This system intentionally omits several elements of FPL, most notably transfers and chips. Including these would require subjective assumptions about how the game should be played, e.g., when to use each chip, when to save free transfers, and how to plan for future gameweeks. The leaderboard is intended to provide a clean assessment of each source's predictions, independent of any particular transfer/chip strategy.

Metrics

In addition to the leaderboard, several metrics are calculated for each source's predictions in each gameweek:

• Mean absolute error: This measures the average difference between predicted and actual points.
• Root mean squared error: This is similar to mean absolute error but penalizes larger errors more heavily.
• NDCG@15: This measures how well a source ranks the top 15 point scorers.
• Top-15 accuracy: This measures the overlap between the predicted and actual top 15 point scorers.

Each week's NDCG@15 and Top-15 accuracy are calculated across all Premier League players who have at least one fixture in that week. Mean absolute error and root mean squared error are calculated on the subset of those players who are owned by at least 1% of FPL managers. This focuses the evaluation on the players most relevant to FPL managers.

Collection of Predictions

Where possible, predictions will be collected within 24 hours before each gameweek's deadline and at approximately the same time for all sources. Although care is taken to record predictions accurately, collection errors may occur. If a source does not provide a prediction for an eligible player, or if a prediction cannot be collected, the player's predicted points are set to 0 for both metric calculations and leaderboard squad selection.

The collected predictions will not be republished on this website. Please visit each source directly to view its predictions. However, the selected squad (starting lineup, captain, vice-captain, and bench order) for each source will be published before each gameweek's deadline.

Conflict of Interest

'Dep La' is a prediction model that I (the maker of this website) developed. In fact, I made this website to compare Dep La with other FPL prediction models. However, my model receives no special treatment from this benchmark and is evaluated under the same rules as every other source.

Conclusion

None of this changes the fact that football (and, by extension, FPL) involves a lot of luck. Sensible predictions may turn out to be wrong and vice versa. The results reported here may not be statistically significant, especially over a small number of gameweeks. But I hope this gives some idea of which prediction sources are most useful. If you have any questions, thoughts, or feedback on this benchmark, please reach out to me.