Weighting

How weighting works

In a scoring model, each criterion gets a score and a weight. The overall score is the weighted sum, the additive model used throughout multiple-criteria decision analysis (Cinelli and colleagues, 2020):

total = w1 x s1 + w2 x s2 + ... + wn x sn, with the weights adding up to 1 (or 100 percent).

Two conditions have to hold for this to mean anything. First, the criterion scores must be on the same scale, or a criterion measured in millions of dollars will swamp one measured 1 to 5. That is what normalization is for. Second, each score must mean the same thing across reviewers, which is the job of a rubric.

Five ways to choose weights

1. Equal weights. Every criterion counts the same. Simple, hard to argue with, and better than it sounds (see below).
2. Point allocation. Give the team 100 points to spread across criteria, then average. Fast, but reflects stated beliefs, which may not match how the team actually decides. In RevOps scoring, where marketing, sales and customer success must all sign off on the weights, it is a quick way to start the conversation.
3. Rank order. Rank the criteria, then convert ranks to weights. The rank-order centroid method gives four criteria weights of about 0.52, 0.27, 0.15 and 0.06.
4. Pairwise comparison. Compare criteria two at a time ("is team more important than market, and by how much?") and derive weights from the answers. This is the core of the analytic hierarchy process (AHP).
5. Fitted to outcomes. Use past decisions and results to estimate which criteria actually predicted success. This is the basis of predictive scoring, and it needs enough history to be reliable.

Worked example: how weights change a ranking

Harbor Analytics, a fictional company, is hiring a data analyst. Two finalists are scored 1 to 5 on three criteria.

Nothing about either candidate changed. The weights decided the order. That is why weights should be set before anyone sees the candidates, and why it is worth running this kind of check: if a small change in weights flips the top of the list, the team has a real choice to make about what the role needs, and should make it on purpose.

Equal weights vs fitted weights

Robyn Dawes found in 1979 that linear models with equal or even arbitrary positive weights often predicted about as well as models with carefully estimated weights, and better than the experts they were compared against. A 2020 hiring study by Yu and Kuncel repeated the test on three samples of managers: a model with consistent but random weights reliably beat the experts' own predictions of job performance. Choosing the right criteria and scoring them consistently mattered more than fine-tuning the weights.

Common weighting mistakes

• Too many small weights. A criterion worth 3 percent cannot change a decision. Cut it or merge it.
• Weighting unnormalized inputs. Raw revenue and a 1 to 5 rating cannot share a sum.
• Adjusting weights after seeing results. Changing weights until a favourite candidate wins is a gut call with arithmetic attached.
• Weights that hide a dealbreaker. A hard requirement belongs in a pass or fail check, not in a 10 percent weight. Vendor scoring handles this with knockouts that run before the weighted sum.
• Ignoring what the weights produce. A fit score is only as good as the weights behind it; compare high and low scorers against outcomes.
• Setting weights once and forgetting them. Markets and roles change. Recheck weights against new outcomes at least yearly, and watch for score drift when the pool itself shifts, for example when a fund moves from seed to Series A.

How ScoringFactory approaches weighting

Teams rarely know their own weights. Ask a partner what matters and they say the founder; look at their last fifty decisions and the picture is often different. ScoringFactory learns what a team actually rewards from its past yes and no decisions, shows each score with the evidence behind it, and leaves the decision to the team. See the flow.