More Dimensions, Same Blind Spot: What Embeddings Miss in 25 Years of Queens of the Stone Age

TL;DR, ordered by how much I trust each finding

• Solid: every embedding I tested (6 representations, 8 OpenAI dimension variants) detects that later lyrics stop returning to earlier stanzas. That is mostly the refrain disappearing.
• Solid, and negative: none of them detects the direction a song takes, or tells apart the songs where readers see an argument. 256 dimensions see what 3,072 see.
• Strong lead: the band’s topics barely move in 25 years; how it handles them does. Songs with an arc rise from 2 of 34 to 11 of 29.
• Exploratory: half of those arcs are open: offers and promises the lyric never resolves.

Embeddings are good at telling you what a text is about. They are also good at noticing when a text repeats itself. What they lose is the relation between two passages: who does what to whom, in what order, and on what condition. Giving the vector more dimensions does not bring that relation back.

I did not set out to show that. I set out to answer a music question. Listening to Queens of the Stone Age straight through, I had the impression that their songs move from circling to advancing. “Feel Good Hit of the Summer” (2000) is a list of substances that loops like a riff, with no before or after. In “Fortress” (2017), an image changes owners: first the fortress is someone else’s defense, then the voice offers its own in case that one falls. The song stops describing a state and starts proposing a deal.

Testing that impression on 88 lyrics from eight studio albums turned into a test of the tools. The part about the tools is the one that matters if you build retrieval systems or agent memory.

The instruments

I used three, each with its own blind spot.

Two automated readers. gpt-5.4-mini and claude-haiku-4-5 (Mini and Haiku) read each lyric without title, artist or year. They assigned broad topics from a rubric fixed in advance, then six finer subtopics defined after exploring the corpus, each with a mechanism and a development: sustained, attempted, changed, or revalued.

The readers often disagree: on whether a song “has an arc,” κ = 0.34. (κ measures agreement beyond chance: 0 is chance, 1 is perfect.) I show them separately and accept a change only when both move the same way.

Six vector representations of every stanza: OpenAI’s two most recent embedding models, text-embedding-3-small and -large; three open models run locally (E5 large, E5 small, MiniLM); and a purely lexical baseline, TF–IDF. If a pattern shows up in all six, it does not depend on one model. If it also shows up in TF–IDF, it is probably about words rather than meaning.

Close reading of a few songs. Everything is exploratory: I chose the periods (1998–2002, 2005–2007, 2013–2023) after seeing the series. The limits are in a table near the end.

What the vectors see: the loop disappears

On the first three albums, more than a third of the sung lines repeat another line. That is the desert-rock legacy: the voice is one more rhythmic instrument, and the refrain is a mantra. From Lullabies to Paralyze onward, repetition drops and verses carry exposition. Villains, made with producer Mark Ronson and built on groove, brings repetition back while its lyrics are the longest: the music recovers the loop while the words keep arguing.

Line counts are crude, so I asked the vectors. A stanza’s novelty is its distance to the most similar earlier stanza in the same song; low novelty means it returns to something already sung. Novelty falls as any song goes on. On the 1998–2002 albums it falls much further in the last quarter: the ending goes back to what was said, and the loop closes. On the later albums, the final stanzas keep bringing something new.

All six representations agree, song by song (correlations of 0.77 to 0.97), TF–IDF included. This is the study’s most solid result, and a modest one: what the vectors measure is, above all, the abandonment of the refrain.

What the vectors miss: direction

If later songs stop returning, do they go somewhere? I traced each lyric’s path through its stanzas and measured its drift: whether stanzas get farther apart the more separated they are in the song. I compared each song with reshuffled versions of itself. A song with a direction should move away from its start more than its shuffled copies.

The p-values come from permutation tests: shuffle the period labels thousands of times and count how often chance alone produces a gap as large as the real one.

The table surprised me. Drift does not rise over the years because it was always there. In every era, between two-thirds and nine in ten lyrics move away from their start more than chance would. What disappears is the return.

The PCA plane makes this visible. “Feel Good Hit of the Summer” ends almost where it began: the loop in its purest form. But “Regular John,” which opens the first album, travels nearly as far as “Fortress” or “I Sat by the Ocean.” They all zigzag. For a vector, moving away is not the same as making an argument.

More dimensions, same blind spot

One suspicion remained: maybe 1,024 numbers are too few to keep the relation between passages. So I sent the same 513 stanzas to OpenAI’s two most recent embedding models at full dimension (1,536 for small, 3,072 for large) and then cut the vectors down to 512 and 256. It cost a third of a cent.

The answer is clear. In all eight variants, novelty rises, recurrence falls, and drift does not change. No variant separates the songs where the readers see an arc (p between 0.38 and 0.91). At 256 dimensions, the large model ranks songs by novelty almost exactly as it does at 3,072 (ρ = 0.95, a rank correlation).

Dimension is not the bottleneck. What is missing is a representation that records who does what to whom, and in what order.

Across the catalog, albums released further apart sit slightly further apart in the five neural models (ρ between 0.26 and 0.34): not conclusive with eight albums, and invisible to TF–IDF. One detail repeats: In Times New Roman… (2023) is the album closest to Era Vulgaris, …Like Clockwork and Villains in almost every representation.

Two songs up close

The gap is easiest to see in two cases. In six songs I took an early and a late passage. I compared each with two descriptions written in advance: A, the starting position, and B, the transformation.

“Fortress.” The two passages fall on opposite sides of zero in all eighteen dense configurations. It looks like a win. But TF–IDF separates them too, and OpenAI’s models failed the control sentences where only the roles are swapped. The vector notices that the late passage is different. There is no evidence it notices who is protecting whom.

“I Sat by the Ocean.” Two ships pass in silence. The image returns twice near the end, and right after it comes the collision. With one line of context, all three dense models see the shift from distance to collision. With six lines, none does. The wider window also takes in the returns of the distance image, and the vector averages them with the collision. The collision means something because it follows the distance. That order is exactly what averaging destroys. Neither catalog reader records this arc either: their rubric has no slot for an image that changes meaning.

What the readers see: more arcs, mostly open

Where the vectors see no direction, the readers see more arcs on the later albums: both agree on an arc in 2 of 34 lyrics from 1998–2002 and in 11 of 29 from 2013–2023, with Villains at 5 of 9. The rise holds at equal length but lives mostly in the long lyrics. The arc arrives with the form that gives it room, and with 88 songs the two cannot be pulled apart.

One awkward caveat: under the first rubric, fixed before reading, Mini reproduces the rise (p = 0.013) and Haiku does not. I treat the trend as a strong lead, not a fact.

What kind of arc is it? In twelve lyrics both readers also agree on the subtopic, and half are open arcs: something is offered, promised or attempted, and the lyric never says whether it happens. “Villains of Circumstance” promises to come back, and does not. Only four lyrics treat a change as done, and even then partially: in “Emotion Sickness” the fever breaks, but nothing says what broke it.

“Fortress” is missing from the matrix. Haiku sees three open arcs in it (an offer of shelter, a promise, an attempt to hold on); Mini sees the same positions, standing still. The song whose change seems clearest to me is the one that splits the readers. A conditional offer can read as a position or as a movement. That is not a bug; it is how these lyrics build their arcs.

What the band changes, and what it keeps

The topics barely move. Desire and attachment appears in about 60% of the lyrics at both ends of the run; power and control stays above half almost throughout. The subtopics move, and they measure what a lyric does with its topic:

• Offered shelter goes from almost absent (4% on average across readers) to 29%, peaking on Villains.
• Credibility and promise goes from 50% to 79%.
• Traces of a bond go from 24% to 43%.

Inside those subtopics, the mechanisms shift:

• Credibility moves from appearance to trust. Doubt about what is real falls by about half; promises, accusations and withdrawn trust grow. The question becomes whom can you believe.
• Traces get erased rather than kept: the photograph in “Suture Up Your Future,” the perfume in “Emotion Sickness.” Mostly Haiku sees this; Mini detects few traces.
• The failed remedy stays. From “Better Living Through Chemistry” to “Straight Jacket Fitting,” acting on an inner state does not work or costs something. (The count of remedies is unreliable: a line-by-line audit of 2017 and 2023 flips the sign of the difference depending on how borderline cases are counted.)

What this means if you build RAG or agent memory

Each point below rests on a result above. None of the fixes has been tested here.

• Refrains dominate the geometry. Stanza novelty tracks literal repetition (ρ from −0.59 to −0.75). Repeated chunks will crowd retrieval results. Deduplicate repeated sections before indexing, and chunk by section rather than by fixed length.
• Wider windows can erase order. The ships’ collision is visible at one line and gone at six in all three dense models. Store small units with their position and fetch neighbors explicitly.
• Roles do not survive the vector. OpenAI’s models failed both participant-swap controls, and TF–IDF also separated the two “Fortress” passages. If your memory must know who promised what to whom, extract actor, action, addressee and condition as fields before indexing.
• Test retrievers on the passages a claim needs. With four passages per query, catalog-wide search completed 0 of 6 tasks and found 2 of 18 required passages. Restricted to the right songs, it completed 2 of 6, about what random picks give (1.89). Measure coverage of required evidence against a random baseline, and add participant-swap pairs to your eval set.
• Separate “the evidence arrived” from “the claim holds.” One reader, given the right lines from “Feel Good Hit of the Summer,” still accepted successive actions and a change of behavior in what is a list. Valid citations did not stop an unsupported claim.
• More dimensions will not fix it. 256 behaved like 3,072 on every measure here.

Limits, and what would fix them

A cheap validation plan for the next round

1. Human annotation. Two annotators, ideally three, label the 14 lyrics in the matrix plus “Fortress.” They work blind to album, year and model outputs. Each labels subtopic, development and mechanism, and, for each section, actor, action, addressee, condition and outcome.
2. Agreement. Cohen’s κ for two annotators, Krippendorff’s α if there are three, with bootstrap intervals. Report human–human agreement next to human–model agreement. If humans also land near κ = 0.34, the question is ill-posed, not just the models.
3. Preregistration. Before seeing new labels, freeze hypotheses (for example: at least half of agreed arcs are open; later albums have more arcs at matched length), periods, metrics and analysis code, with hashes, as this project already does for its probes.
4. Larger readers. Rerun the subtopic rubric on all 88 lyrics with larger models available to the same accounts, and check whether κ rises and whether they agree better with the human labels. [PENDING: model choice and cost estimate.]
5. Retriever test. Turn the participant-swap controls and the required-passage tasks into a small benchmark for section-level chunking and structured extraction. [PENDING: no result yet.]

[PENDING: estimated annotation hours per lyric.]

What I take away

About the band. The obsessions stay; the terms change. QOTSA sings about desiring and dominating for twenty-five years, but moves from doubting the world to negotiating with someone. The later songs do not tell more; they negotiate more, and their offers stay open. Before the lyrics change what they say, they change how they say it: the loop gives way to the stanza.

About the tools. Three different tests fail in the same place. Drift does not separate the songs with an arc, even at 3,072 dimensions. The vectors do not know who protects whom. The ships’ collision dissolves once the window includes the distance that gives it meaning. A vector summarizes a text; an arc is a relation between two moments of it, with an actor, an addressee and a condition. That convergence is what I would want an engineer to take from a study about a rock band.