Duplicate detection for feature requests: calibrating an embedding threshold against human merges

FeatureOS runs its own public feedback board. People post feature requests, other people vote on them, and a fair share of those posts ask for something already on the board under a different title. Our support team finds those by hand and merges them. Swathy Ravi did that merging by hand for years.

I wanted to know whether a cheap embedding model, running on a Mac, could do that job. The nice part about asking this on our own board is that the answer key already exists. Every merge the team made over the years is a human saying "these two posts are the same request." We had 101 of them sitting in the database, labelled for free, and I had never scored a model against them.

What a merge actually is

In FeatureOS, a merge folds one post (the child) into another (the parent). The child's votes move over to the parent, and the child stops showing up as its own request. One request, one vote count, one place to post the update when it ships.

That makes the merge log a decent ground truth, with one large bias. Humans only merged what they noticed. If two posts asked for the same thing and nobody spotted it, they sit on the board as two separate posts, and in this experiment they count as "not a duplicate." Keep that in mind, because it comes back to bite the numbers later, and in a direction that flatters the humans.

The pull came to 735 posts: 634 parents or standalone posts, and 101 merged children that hang off 73 parents.

How we tested it

We kept the setup small enough to run on one Mac.

• Model: nomic-embed-text, served locally through Ollama. No task prefix, no fine-tuning.
• Text: titles only. I wanted descriptions too, but our own API rate-limited the body pull on the very first call, and I didn't want to pad the gaps with summaries we wrote ourselves. 50 of the 101 children had no description at all, so titles were the only text we had for every post.
• Similarity: cosine similarity between the two title vectors, cos(a, b) = (a · b) / (‖a‖ ‖b‖). A score of 1 means the vectors point the same way, 0 means they have nothing in common.
• Baseline: TF-IDF over one- and two-word phrases, scored with the same cosine. If an embedding model can't beat word overlap, it doesn't deserve a feature.

The negatives matter more than the positives. For each of the 101 children we picked 20 posts that were not its parent or a sibling: the 10 posts the embedding itself thought were most similar, plus 10 at random. That gave 2,020 negatives against 101 positives, a 20:1 ratio. Both methods scored the same pairs.

Those 10 "hard" negatives came from the embedding. We asked the model for its best wrong answers and then graded it on them. The deck is stacked against the embedding, and I left it that way on purpose. Random negatives alone would make any model look great, since most feedback posts have nothing to do with each other.

The threshold problem

Auto-merge needs a cutoff. Pick a threshold t, merge every pair with cosine at or above it, and leave the rest alone. Then count the damage:

• Precision: of the pairs you merged, how many the humans also merged. P = TP / (TP + FP)
• Recall: of the human merges, how many you caught. R = TP / (TP + FN)
• F1: one number that punishes you for neglecting either. F1 = 2PR / (P + R)

In merge terms, a false positive is a false merge and a false negative is a missed duplicate. Drag the slider below and watch them trade places.

The F1-optimal threshold for the embedding sits at 0.79. There it gets precision 0.648 and recall 0.347: it catches 35 of the 101 human merges, makes 19 false merges, and misses 66. TF-IDF does worse at its own best point (t = 0.21), with precision 0.306, recall 0.376, and 86 false merges.

The 0.79 cutoff is also fairly stable. Its 95% confidence interval runs from 0.73 to 0.81. TF-IDF's optimum wanders anywhere from 0.14 to 0.40, which tells you word overlap has no natural "these are the same" line.

What a wrong merge costs

F1 treats a false merge and a missed duplicate as equally bad. They aren't. A missed duplicate means two posts with split votes, and the support team can still catch it next week. A false merge hides a customer's request inside someone else's, moves their vote to a thing they never asked for, and someone has to notice, apologise and unpick it.

So the threshold should minimise cost, with the two error types priced differently:

t* = argmin_t  C_fm · FM(t) + C_md · MD(t)

Here FM(t) and MD(t) are the false merges and missed duplicates at threshold t, and only the ratio C_fm / C_md matters. We swept it, and each cell reads t (FM, MD):

Once a false merge costs three times a miss, the best embedding threshold jumps to 0.88 and merges 15 of 101 duplicates. At 10x it merges 8. The optimiser, given honest prices, tells you to almost never auto-merge. I found that more useful than any single accuracy number in this post.

How well does it rank?

A threshold answers "is this pair a duplicate?" Ranking asks an easier question: "given this pair and a duplicate pair, which one scores higher?" Two curves capture that.

The ROC curve plots the true positive rate against the false positive rate as the threshold moves. The precision-recall curve plots precision against recall instead. With 20 negatives for every positive, ROC can look healthy while precision is poor, because a small false positive rate still means a lot of false merges in absolute terms. PR shows that pain directly, so it is the curve to trust here.

• ROC AUC: 0.853 [0.810, 0.892] for the embedding, 0.778 [0.724, 0.832] for TF-IDF. An AUC of 0.853 means that if you pull one human merge and one non-merge at random, the embedding scores the merge higher 85% of the time.
• Average precision (the area under the PR curve): 0.438 [0.350, 0.539] against 0.298 [0.220, 0.394].

The brackets are 95% bootstrap confidence intervals. We resampled the positives and negatives with replacement 1,000 times, recomputed each metric every time, and took the 2.5th and 97.5th percentiles. With only 101 positives, a handful of lucky or unlucky pairs can swing a point estimate a long way. The embedding wins, and the AUC intervals barely touch, so the gap is real but not huge.

Cosine is not a probability

It's tempting to read a cosine of 0.80 as "80% sure these are duplicates." Binning the scores shows how wrong that is. Below 0.80, the share of pairs that were real merges stays under 0.2. In the 0.80 to 0.85 bin (24 pairs) it jumps to 0.67. The relationship has a cliff in it, and the cliff isn't where the raw number suggests.

A one-variable logistic regression turns cosine into an estimated probability. At the F1-optimal t of 0.79 it says P(duplicate) = 0.28. Your "best" threshold is a point where the model is wrong more often than right about any single pair it merges at the margin.

The Brier score (mean squared error of the probability) is 0.0347 for the calibrated embedding and 0.0393 for TF-IDF. Always guessing the base rate scores 0.0454. So calibration helps, but modestly, and those fits were scored on the same data they were fitted to, which flatters them.

The feature hiding in the data

Threshold metrics ask the model to make a decision. Recall@k asks something gentler: when a child post is submitted, is its real parent somewhere in the model's top k suggestions out of all 634 candidates?

• Top 1: 48.5%
• Top 3: 70.3%
• Top 5: 76.2% (TF-IDF: 56.4%)
• Top 10: 84.2%

The median rank of the true parent is 2 out of 634. That's a usable product. When someone types a new request, show them five posts that look like it and let them vote on one instead of filing another. The human stays in the loop, a wrong suggestion costs a glance, and three times out of four the right answer is on screen.

Pick a post in the explorer below and look at its neighbours. The red rows are where the model and the humans disagree.

Where the model and the humans disagree

The high-scoring "negatives" are the most fun part of this dataset, because several of them look like duplicates nobody merged. A few of them:

• 0.876: "Ability to update multiple Slack Channels with updates from FeatureOS" and "Slack Integration - Update different slack channels with different updates"
• 0.866: "See merged posts in comment section" and "Comment/Note for merged post"
• 0.856: "Custom fields for posts" and "Custom fields based filters for posts"

I'd merge the first one today. The other two are close calls a human would want to read before touching, which is the point.

The reverse list is stranger. These are real human merges where the titles share almost nothing:

• 0.335: "Customer Engagement Module" merged into "CSAT & NPS survey"
• 0.411: "Vote for new features" merged into "CSAT & NPS survey"

No title embedding is going to put those two together. Someone on the team read the request, understood what the customer wanted, and decided it belonged to the survey work. That's product judgment. Text similarity sees shared words, and the team was looking at the roadmap.

Caveats

• 101 positives. Read the intervals, not the point estimates.
• Titles only. Descriptions might help a lot. We didn't get to test them.
• Partial board. The main board was capped at the first 500 posts. Posts we didn't pull couldn't be parents or negatives.
• Hard negatives came from the embedding. That biases the comparison against the embedding, so the gap over TF-IDF is if anything understated.
• In-sample calibration. The logistic fit and Brier scores use the data they were fitted on.
• Precision is a lower bound. Some "negatives" are unmerged duplicates, like the Slack channels pair, and the model gets marked wrong for finding them.

Do this on your own data

If you run a feedback board, a ticket queue or a CRM with a merge button, you already have this ground truth. The core of the eval is a few lines of numpy:

E /= np.linalg.norm(E, axis=1, keepdims=True)
S = E @ E.T                                         # cosine for every pair
sp = np.array([S[a, b] for a, b in pos])            # human merges
sn = np.array([S[a, b] for a, b in neg])            # sampled non-merges
ts = np.round(np.arange(101) * 0.01, 2)
fm = (sn[None] >= ts[:, None]).sum(1)               # false merges at each t
md = len(sp) - (sp[None] >= ts[:, None]).sum(1)     # missed duplicates at each t
t_best = ts[np.argmin(3 * fm + md)]                 # a false merge costs 3x a miss

The full harness, with the bootstrap, calibration bins and recall@k, is about a hundred lines of Python on top of scikit-learn. If you want the method behind our broader model testing, I wrote it up in how we test AI models on your work.

What ships next

What I want FeatureOS to get next is a possible-duplicates list at submit time, ranked by embedding similarity, with the cutoff set from whatever cost ratio our support team picks rather than whatever maximises F1. Merging stays a human button.

If you have a pile of human decisions sitting in a database and want to know what a model would make of them, start a chat with us on the site, or book a call through the Applied Labs page.