A ranked shortlist is a shortlist ordered by score against a defined bar, with the evidence behind each score attached, so a team knows who to look at first, how far apart the options really are, and why each one sits where it does.
What ranking adds to a shortlist
A shortlist says who made the cut. A ranked shortlist adds three things:
- Order. Who to call first when time is short, a competing offer lands, or a round is closing.
- Distance. Whether number 1 is far ahead of number 2 or a coin flip away.
- Reasons. The criteria and evidence behind each position, so anyone can challenge it.
Without the reasons, a ranking is just an opinion with numbers. With them, it is a working document the team can argue over productively. That is why a ranked shortlist depends on evidence-based scoring underneath.
How to rank a shortlist
Ranking several options on several criteria is the problem studied in multiple-criteria decision analysis (see the review by Cinelli and colleagues, 2020). Two practical methods cover most teams:
Weighted score
Give each option a fit score from an evaluation rubric with agreed weighting, then sort from highest to lowest. Fast, consistent, and easy to explain. The weakness is false precision: an 81 and an 80 look ordered, but the gap is smaller than the scoring noise.
Pairwise comparison
Compare options two at a time ("would we rather have A or B?") and count wins. Useful for small lists of senior candidates where criteria are hard to separate. It takes longer and is harder to repeat consistently.
A sensible hybrid: rank by weighted score, then use pairwise discussion only for options whose scores sit within a few points of each other.
Teams with enough past outcomes can sort by a predictive score instead, which orders options by how likely each is to turn out well.
Worked example: ranking six founders
Bluebird Seed, a fictional fund, has six founders on this week's shortlist. Scores are out of 100 on the fund's rubric. Past calibration sessions showed two partners scoring the same founder usually land within about 4 points of each other, so the fund treats gaps under 4 as ties.
| Rank | Company | Score | Strongest criterion | Weakest criterion |
|---|---|---|---|---|
| 1 | Quillmark | 84 | Customer proof: 3 paid pilots | Team: no technical cofounder yet |
| 2 (tied) | Ferrous Labs | 78 | Founder depth in the domain | Market size unclear |
| 2 (tied) | Openshelf | 76 | Speed: shipped 4 releases in 6 weeks | Pricing untested |
| 4 | Kettle & Co | 69 | Distribution partner signed | Retention data thin |
| 5 (tied) | Marlin Health | 63 | Regulatory path mapped | Long sales cycle |
| 5 (tied) | Tessel | 61 | Strong waitlist | No revenue |
The partners meet Quillmark first. For Ferrous Labs and Openshelf, the ranking says "meet both," and the weakest-criterion column tells them what to probe in each meeting. The ranking guides the week. The partners still make every call.
Ranked vs unranked shortlist
| Ranked shortlist | Unranked shortlist | |
|---|---|---|
| Tells you | Who is in, in what order, and by how much | Who is in |
| Best when | Time is limited and options must be sequenced | Everyone on the list will get equal attention anyway |
| Risk | Anchoring: reviewers go easy on number 1 and hard on number 6 | Order defaults to arrival time or the loudest advocate |
| Needs | Consistent scoring and visible evidence | A clear bar for inclusion |
If interviewers see the rank before they meet someone, they may grade toward it. Some teams show rank to the person scheduling meetings but not to the interviewers.
Common ranking mistakes
- Reading small gaps as real. Judgments vary more than people expect. In a McKinsey interview, Olivier Sibony describes analysts at an investment firm whose valuations of the same company differed by 44 percent on average, far more than their leaders expected. Measure your own noise, which is what inter-rater reliability tracks, and treat gaps inside it as ties.
- Hiding the parts. A rank without per-criterion scores cannot be discussed.
- Never re-ranking. New evidence from a first meeting or a reference call should move positions.
- Ranking a list that was never complete. The order is only as good as the longlist it came from.
How ScoringFactory produces ranked shortlists
ScoringFactory ranks founders, companies, and candidates against the bar a team has shown in its own yes and no decisions, and attaches the record behind each score. In a fund, that is how deal flow scoring turns into a weekly list of who to meet next. The team reads the reasons and decides. See the use cases.
Frequently asked questions
What is a ranked shortlist?
A ranked shortlist is a shortlist sorted by score against a defined bar, with the evidence for each score attached. It tells a team who made the cut, the order to review them in, and how large the gaps between options are. The attached reasons let reviewers challenge any position instead of accepting the order on trust.
How do you rank candidates or deals objectively?
Score every option on the same weighted criteria, written down before anyone is reviewed, with evidence for each rating. Have more than one reviewer score independently, measure how much they usually disagree, and treat gaps smaller than that as ties. Use discussion only to separate options that are close.
Should a shortlist be ranked or unranked?
Rank it when time is limited and options must be sequenced, such as a competitive round or a role with several finalists. Keep it unranked when everyone will get equal attention, or when interviewers might anchor on the rank. Many teams rank for scheduling and hide the rank from interviewers.
How often should a ranked shortlist change?
Whenever new evidence arrives. A first meeting, a reference call, or new metrics should update the relevant scores and can move positions. A ranking that never changes after it is first produced is usually being ignored. Keep the earlier versions so the team can see what moved and why.