Convergence
in vector space

About me

Pamela Fox

Principal Cloud Advocate at Microsoft

Formerly: UC Berkeley, Coursera,
Khan Academy, Google

Twitter	🔗 @pamelafox
LinkedIn	🔗 linkedin.com/in/pamela-s-fox
GitHub	🔗 github.com/pamelafox
Website	🔗 pamelafox.org

Let's play Convergence

1. Two players face each other.
2. Everyone counts down, "3..2..1"
3. They both yell a word at the same time,
   like "fire" and "whale".
4. Everyone thinks:
   "What single word is between those two words?"
5. Two new players count down and yell their words,
   like "blubber" and "engine".
6. Same word? Everyone wins! 🎉
   Different words? Keep going.

Can computers play it?

tinyurl.com/play-convergence

Vector embeddings

Vector embeddings are lists of numbers that represent items in a high-dimensional space.

Each number is a dimension.

An embedding model turns text into a vector:

"fire" → embedding model →
[-0.030, 0.089, 0.008, 0.048, 0.026, …]

Open-weight embedding models

🔗 MTEB leaderboard

Generate embeddings with Python

Download model weights from Hugging Face
and generate embeddings with SentenceTransformer:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "sentence-transformers/all-MiniLM-L6-v2")

vectors = model.encode(["fire", "whale"],
                       normalize_embeddings=True)
print(vectors.shape)  # (2, 384)
                

🔗 Full code: embed.py

Measuring vector similarity

The most popular method is cosine similarity.

fire, whale = model.encode(
    ["fire", "whale"],
    normalize_embeddings=True)
                

fire[-0.030, 0.089, …]

whale[-0.026, 0.061, …]

# Cosine similarity
np.dot(fire, whale) / (
    np.linalg.norm(fire) *
    np.linalg.norm(whale))  # 0.30
                

\[\frac{\color{#c2410c}{\mathbf{x}} \cdot \color{#3776ab}{\mathbf{y}}}{\lVert\color{#c2410c}{\mathbf{x}}\rVert \, \lVert\color{#3776ab}{\mathbf{y}}\rVert}\]

# Dot product for unit vectors
np.dot(fire, whale)  # 0.30
                

🔗 Full code: cosine.py

Most similar words

Rank every vocabulary word by similarity to "fire":

Word	Similarity
fire	1.00
flame	0.80
firework	0.78
wildfire	0.71
fireball	0.70

🔗 Full code: similarity.py

Operator 1: Centroid

Find the word closest to the average vector.

query = normalize(vec_a + vec_b)
scores = np.dot(vocab.vectors, query)
                

Effectively sums s_a + s_b, so one very high similarity can win.

Operator 2: Product

Find the word with the highest product of similarities.

scores = (np.clip(sim_a, 0, None) *
          np.clip(sim_b, 0, None))
                

The biggest rectangle s_a × s_b wins, so one low similarity hurts.

Operator 3: Maximin

Find the word whose weaker similarity is highest.

scores = np.minimum(sim_a, sim_b)
                

Maximize the minimum similarity.

Operator 4: Textual

Let the model combine them: embed a phrase.

phrase = f"{word_a} and {word_b}"
phrase_vec = model.encode(
    [phrase], normalize_embeddings=True)[0]
scores = np.dot(vocab.vectors, phrase_vec)
                

The wording of the combined phrase changes the winner:

All four operators

The top 5 words for earring + modem, with MiniLM L6:

The game, step by step

1. Start with two words, a and b.
2. Two models each pick their top word with the same operator.
3. Same word? Converged! 🎉
4. Different words? Those two picks become the next pair.
5. Stop if a pair repeats (loop) or after 10 rounds.

The game loop

pair = (word_a, word_b)
for round_number in range(1, max_rounds + 1):
    # Each model picks its top word, excluding the pair
    picks = tuple(top_word(model, operator, pair)
                  for model in (model_a, model_b))
    if picks[0] == picks[1]:
        return "converged", round_number
    if picks in seen:
        return "loop", round_number
    seen.add(picks)
    pair = picks
return "round_limit", max_rounds
                

🔗 Full code: simulate_convergence.py

Let's play again!

• Switch the operator to maximin for hard mode.
• Try random models each round.

tinyurl.com/play-convergence

The battle setup

• 6 models → 15 pairings (+ each model vs. itself)
• Same 100 random starting pairs for every pairing
• Up to 10 rounds per game
• Outcomes: converged, loop, or 10 round limit

uv run model_battle.py --games 100 --operator centroid
                

Runs in ~20 seconds: the vectors are precomputed in SQLite.

Self-play: instant convergence

A model vs. itself converges in round 1, every time.

Same vectors + same math = same word.

Convergence is only interesting when the players see the world differently.

Which models play well with others?

Convergence rates with the centroid operator:

Similar small models think alike; Qwen3 is the odd one out.

Best benchmark ≠ best teammate

The highest MTEB scores converge the least.
Benchmarks measure their tasks, not yours!

Does the operator matter?

The longest roads to convergence

Vectors have a day job

Vector embeddings are the technology powering:

But embedding models differ

Each model captures one way of looking at the world:

Evaluate models for your scenario before you pick one.

One app = one world view

Most apps ship with a single embedding model.
Every user, query, and document goes through it.

One community = many world views

Each of us brings our own way of looking at the world.

Keep playing with humans!

Thank you

Questions?

Twitter	🔗 @pamelafox
LinkedIn	🔗 linkedin.com/in/pamela-s-fox
GitHub	🔗 github.com/pamelafox
Website	🔗 pamelafox.org

PLAY

🔗 tinyurl.com/play-convergence

SLIDES

🔗 pamelafox.github.io/convergence-with-vectors