My friend Shruti recently went to a speed-dating event with about a hundred people. The organizers seated ten people at each table: five men and five women. Everyone gave a one-minute introduction; then the men moved as a group clockwise to the next table. It does what it promises: every woman eventually meets every man.

But my friend pointed out a small cruelty in the design: each pack of five women travels together. Every woman hears the same four introductions again. And again. And again. The men have the same problem.

An interactive exploration

Here is a scaled-down version with 8 men and 8 women at four tables. Each table seats 2 of each. Compare the original rotation with the improved schedule, then select anyone to trace their evening.

Both schedules cover every possible man-woman pairing. The difference is what happens along the way.

1. In the original rotation, the men remain at fixed tables and each pair of women moves together. Every woman meets all eight men, but hears the same woman’s introduction four times.
2. In perfect mixing, everyone moves. Every person still meets all eight potential dates, but now also meets four different people of their own gender thus avoiding any repeated introduction.

The full mathematical design has a fifth round. It would place the men at two all-male tables and the women at two all-female tables, so the dating version simply drops it. The rest of this post is about the mathematics of designing such a schedule.

Problem Simplification: Drop gender

Let us first make the problem more demanding and, paradoxically, easier to understand. Suppose there are \(N\) people. In every round they split into tables of \(m\) people each. Can we arrange the rounds so that every pair of people shares a table once and only once?

Combinatorialists call this a resolvable Steiner \(2\)-design, or a resolvable block design. In our case, each round partitions the room, and the full evening covers every pair exactly once. Think of this as speed socializing.

The unavoidable constraints

Before searching for a schedule, we can rule out most values of \(N\) and \(m\) with two simple observations.

For tables of four, the possible room sizes begin \(N=4,16,28,40,\ldots\). For instance, twenty people cannot work: one person would need \(19/3\) rounds.

Passing the test does not guarantee a schedule

This is the first delightful trap: the conditions are necessary, but they are not sufficient. For example, \((m,N)=(6,36)\) passes both tests, yet no schedule exists (see for instance [Beth–Jungnickel–Lenz]). Some parts of the existence story are completely understood:

Interestingly, for every fixed \(m\), all sufficiently large \(N\) that pass the divisibility test do work [Ray-Chaudhuri–Wilson]. What remains difficult is the finite territory before “sufficiently large.” There is no known one-line classification for arbitrary \(m\).

Simple construction for specific parameters

The cleanest construction occurs when \(m=q\) is a prime power and \(N=q^2\). Imagine labeling people by the \(q^2\) points of a \(q\times q\) grid but doing arithmetic in the finite field \(\mathbb F_q\). Each round is characterized by a line of a given slope.

• One round uses the vertical lines.
• The other \(q\) rounds use the lines of slope \(a\): \(y=ax+b,\ \mathrm{where}\ a,b\in\mathbb F_q\).

Lines of one slope partition the grid, so they form a round. Any two points determine one and only one line, so any two people meet exactly once. For prime-power \(q\), finite fields give this construction immediately. This is what is used in the interactive schedule above (\(q=4\)).

For a general candidate \((m,N)\), there is still a universal search method: encode every possible table as a binary choice and require every participant to appear once per round and every pair once overall. This is an exact-cover or integer-programming problem. It can decide modest instances, but it is not a proof that all arithmetically admissible inputs work. At scale, design theory uses recursive constructions—group-divisible designs, frames, and difference families [Beth–Jungnickel–Lenz].

Add Gender back into the picture

Now color every participant one of two types—say women and men (suppose there are \(c\) women and \(N-c\) men). A table of size \(m\) can contain \(0,1,\ldots,m\) women. Let

\[ h_j=\#\{\text{tables containing exactly }j\text{ women}\}. \]

The vector \(h=(h_0,h_1,\ldots,h_m)\) is a histogram of the entire event. It lets us ask questions the ungendered version of the problem cannot ask:

• Can we make \(h_0=h_m=0\), so no table is all men or all women?
• If \(m\) is even, can most tables sit near the balanced value \(j=m/2\)?
• Which histograms are numerically plausible but geometrically impossible?

The first one would be interesting from a dating point of view where you would like to avoid tables of all men or all women. The second is just a stronger version of that, which states can be avoided. Highly biased table where all but one are men, or all but one are women. The third one is just the full mathematical complexity of this problem. The first three moments of these are fixed based on the parameters but there is interesting geometry past these. In mathematical language, asking for \(h_0=h_m=0\) is asking for a weak two-coloring or a two-sided blocking set. This connects our scheduling puzzle to block-design colorings [Rosa–Colbourn], asymptotic weak colorability [Horsley–Pike], and blocking sets in affine planes [Jamison], [Brouwer–Schrijver].

These equations are necessary, but they forget how tables overlap. That missing geometry is where the interesting questions live.

The complete \(m=4,N=16\) experiment

With 8 women and 8 men, there are \(\binom{16}{8}=12{,}870\) ways to assign the two types to the schedule. This is because we can fix the table arrangements and only permute the gender across the sixteen participants. Exhaustive enumeration produces exactly seven histograms:

The two bold rows are the ones that I found interesting. The first has no single-gender table anywhere. The second has sixteen perfectly balanced tables and four single-gender tables. The interactive figure uses the latter construction and drops its four single-gender tables. What remains is unusually clean: sixteen mixed tables over four rounds, every one split \(2+2\), and every possible man–woman pair covered exactly once. That right there is the perfect speed dating schedule!

Explore the full five-round design

Here is the original construction explorer. Unlike the dating view above, it keeps the fifth round because the full design is trying to make every pair meet exactly once. Switch between the two histograms at the top, choose any person, and click a round—or click someone in the tracker to jump directly to the round in which the pair meets.

The questions I was left with

While there are some obvious mathematical questions, some of which are still open in the academic literature, here’s a short list that opens up:

• Schedule existence. For which \((m,N)\) does a perfect pairwise-meeting schedule exist?
• Balanced events. When can a schedule be colored with equal-sized types and no monochromatic table?
• Histogram spectrum. Which vectors \(h\) satisfying the counting equations actually come from a schedule?
• Practical variants. What changes if we care only about cross-type meetings, allow uneven tables, or permit a small number of repeats?

Like many puzzles, I love that something that sounds almost petty (having to hear the same introduction four times) and opens into large bodies of mathematics. Here’s to improved speed dating!

References

1. D. K. Ray-Chaudhuri and R. M. Wilson, “Solution of Kirkman’s schoolgirl problem,” Proceedings of Symposia in Pure Mathematics 19 (1971), 187–203.
2. H. Hanani, D. K. Ray-Chaudhuri, and R. M. Wilson, “On resolvable designs,” Discrete Mathematics 3 (1972), 343–357. doi:10.1016/0012-365X(72)90091-X.
3. D. K. Ray-Chaudhuri and R. M. Wilson, “The existence of resolvable block designs,” in A Survey of Combinatorial Theory (1973), 361–375. doi:10.1016/B978-0-7204-2262-7.50035-1.
4. T. Beth, D. Jungnickel, and H. Lenz, Design Theory, 2nd ed., 2 vols., Cambridge University Press, 1999. Publisher preview.
5. N. S. Mendelsohn, “Intersection numbers of \(t\)-designs,” in L. Mirsky, ed., Studies in Pure Mathematics (Presented to Richard Rado), Academic Press, 1971, 145–150.
6. T. van Trung, Q.-R. Wu, and D. M. Mesner, “High order intersection numbers of \(t\)-designs,” Journal of Statistical Planning and Inference 56 (1996), 257–268. doi:10.1016/S0378-3758(96)00022-5.
7. A. Rosa and C. J. Colbourn, “Colorings of block designs,” in J. H. Dinitz and D. R. Stinson, eds., Contemporary Design Theory: A Collection of Surveys, Wiley, 1992, 401–430. Publisher page.
8. D. Horsley and D. A. Pike, “On balanced incomplete block designs with specified weak chromatic number,” Journal of Combinatorial Theory, Series A 123 (2014), 123–153. doi:10.1016/j.jcta.2013.12.004.
9. R. E. Jamison, “Covering finite fields with cosets of subspaces,” Journal of Combinatorial Theory, Series A 22 (1977), 253–266. doi:10.1016/0097-3165(77)90001-2.
10. A. E. Brouwer and A. Schrijver, “The blocking number of an affine space,” Journal of Combinatorial Theory, Series A 24 (1978), 251–253. doi:10.1016/0097-3165(78)90013-4.

This puzzle comes from the amazing blog by Tanya Khovanova.

The puzzle

You have $2^n$ visually identical coins, each weighing either 10 or 11 gram. You have a standard balance that will weigh groups of coins. Your goal is to determine whether the parity of 10 gram coins (whether it is even or odd), using at most $n$ weighings.

You can start with the smaller problems trying to solve this for 4 coins with 2 weighings, 8 coins with 3 weighings, or even 16/32 coinds in 4/5 weighings respectively.

Visual Solution

The interactive media below lets you try it for $n = {3, 4, 5}$ (8, 16, and 32 coins respectively) with $n$ weighings allowed. As you might guess, the solution is effectively a binary search (glad to be grinding LeetCode right now).

Select the number of coins (8, 16, or 32) and the number of 10 Gram coins. Then click New puzzle (for a new coin assignment) and step through the weighings manually (Measure W₀ button), or hit Run auto to watch the algorithm work automatically.

Formal Solution

The argument is a discrete analogue of the intermediate value theorem: if a quantity starts positive and ends negative while changing by at most 2 each step, it must hit zero somewhere along the way — and hitting zero is exactly what a balanced scale tells us. There are three key ideas:

1. $\textbf{Reduction.}$ Checking the parity of the signed difference $D = L - R$ of any complete weighing (equal coins on each side) suffices to determine the parity of the coin count.
2. $\textbf{Binary search.}$ We construct a sequence of $m+1$ complete weighings where consecutive weighings differ by a single swapped pair, so $\lvert D_{k+1} - D_k \rvert \le 2$. The sequence starts at $D_0$ and ends at $D_m = -D_0$. And then we run a binary search for a sign change in this sequence.
3. $\textbf{Terminal condition.}$ Any balanced weighing along the binary search will conclude even parity. If none balance, two adjacent opposite-sign values must be found and I show that this forces odd parity.

Thus the decision rule is: if any weighing balances, parity is even; if none balance, parity is odd.

Notation and reduction

Let $N=2^n$. Since $N$ is even, the count of 10g coins and the count of 11g coins have the same parity (they sum to $N$), so it suffices to determine the parity of $T$, the number of 10g coins. We work only with complete weighings: those placing exactly $N/2$ coins on each pan. Let $m = N/2 = 2^{n-1}$.

In particular, a balanced weighing ($D = 0$) implies $T$ is even, and if $T$ is odd no complete weighing can ever balance.

Sequence of weighings

We will start by pairing the coins (randomly as the pairing is immaterial) into $m=2^{n-1}$ pairs $(x_1, y_1), \ldots, (x_m, y_m)$. Then, we construct a sequence of complete weighings as follows: for $k = 0, 1, \ldots, m$, define the complete weighing $W_k$ by

So $W_0$ puts all $x$-coins on the left, and each step swaps one pair. For $k \in {0, 1, \ldots, m}$, denote by $D_k$ the signed weight difference of weighing $W_k$. Note that at $W_m$, the coins in each pair have swapped their position and thus: $D_m = -D_0$.

Binary search

Weigh $W_0$. If it balances, we know that $T$ is even and we are done. Otherwise $D_0 \neq 0$ and since $D_m = -D_0$ the signed differences start and end with opposite signs. Next we run a binary search through the sequence $W_0, W_1, \ldots, W_m$ as follows: Maintain an interval of indices $[a,b]$ such that $D_a$ and $D_b$ have opposite signs. Initially this is $[0,m]$. Its length is a power of two.

While $b-a>1$, let $c = (a+b)/2$ and weigh $W_c$. If $D_c=0$, then the weighing balances and the parity is even. Otherwise, $D_c$ has the same sign as exactly one of $D_a$ and $D_b$. Retain the half-interval whose endpoint values have opposite signs:

Thus each additional weighing halves the interval while preserving the opposite-sign condition.

The initial interval has length $m=2^{n-1}$. After $n-1$ midpoint weighings (plus the initial 1 weighing for $W_0$ gives a total of $n$ weighings), either a balance has occurred or the remaining interval has length $1$. In the latter case, we have adjacent indices $k$ and $k+1$ for which $D_k$ and $D_{k+1}$ are nonzero and have opposite signs.

Terminal condition

Once we find two adjacent indices with opposite weight differences, we need to show that $T$ has to be odd. Now, we know that:

where the first equality follows from the single pair swap between $W_k, W_{k+1}$ and the last follows because each weight is 0 or 1.

We complete the proof by contradiction. Suppose $T$ were even. By Lemma 1, all $D_j$ would be even. Then $D_k$ and $D_{k+1}$ are even, nonzero, and of opposite sign, so $\lvert D_{k+1} - D_k \rvert \ge 4$. But we just established that $\lvert D_{k+1} - D_k \rvert \le 2$, a contradiction.

This completes the proof. $\blacksquare$

8 Coins solution

Below is the solution for eight coins by the above method. $n = 3$, $m = 4$, the five weighings are:

Weigh $W_0$; if unbalanced, weigh $W_2$ (the midpoint of $[0, 4]$). Based on whether $D_2$ shares the sign of $D_0$ or $D_4 = -D_0$, weigh $W_1$ or $W_3$. Three weighings suffice.

One of the most interesting privacy technologies that I have come across is called Stained Glass Transform (SGT). This was invented by folks at Protopia AI (their team includes my talented friend and collaborator Sid Roy and in this blog I am looking into their technical paper [1]. The problem it addresses is one that anyone building on cloud LLM APIs encounters: you want the model’s intelligence, but you don’t want the LLM provider to see your prompt/data.

The problem

When you call any hosted LLM endpoint (think ChatGPT, Claude.ai, OpenRouter, HuggingFace), you hand your prompt in the clear to a third-party server, which stores it in their database logs. This is a major concern given the increasingly personal nature of prompts and the mechanics of the data economy. A mechanism to let users benefit from LLMs while preserving the privacy of their input is therefore critical.

Existing solutions

There are a few different ways to resolve this prompt privacy challenge.

• $\textbf{Local hosting.}$ Host the model yourself so the prompt never leaves your environment. Ollama makes this straightforward, letting you run Llama, Mistral, Gemma, and other open-weight models on consumer hardware with a single command. The obvious limitation is compute: a capable model needs a GPU with sufficient VRAM. Beyond that, you forfeit all the infrastructure that comes for free with hosted endpoints: load balancing, auto-scaling, automatic retries, hardware maintenance, and the operational overhead of keeping a model server healthy in production.
• $\textbf{Fully Homomorphic Encryption (FHE).}$ FHE allows computations directly on encrypted data so your prompt is encrypted on-device and the server processes it without ever decrypting it. This Belfort Labs demo is a live in-browser experience that gives a feel for what FHE-based inference looks like in practice. On the open-source side, Zama’s Concrete ML is the leading library tackling the underlying hard cryptographic engineering. The downsides are steep: FHE inference is slower than plaintext, LM endpoints need significant re-engineering to operate over encrypted arithmetic (plaintext-ciphertext), and key management at scale is a non-trivial operational challenge.
• $\textbf{Trusted Execution Environments (TEEs).}$ TEEs (e.g. Intel SGX/TDX, AMD SEV, Confidential Containers) create hardware-isolated enclaves where code and data are hidden even from the host OS and cloud provider. This can be used to perform two-sided privacy where the server cannot see the user’s prompt and the model provider’s weights can simultaneously remain confidential. In practice, the user must still trust the hardware vendor’s attestation, GPU TEE support (needed for performant inference) is relatively new (NVIDIA Hopper is the first generation with production-ready confidential computing), and trust questions around the TEE hosting entity can undermine the privacy guarantees entirely.

Stained Glass Transform (SGT)

The Stained Glass Transform is a novel solution to the same problem with a well-studied and rigorous notion of privacy. The solution involves sending obfuscated embeddings instead of raw text to the LLM provider and letting the provider’s endpoint do the rest.

In other words, it moves the initial preparatory stages used by all LLMs (tokenization and embedding) to the user’s side. Using a trained machine learning model (their secret sauce), the embedding (and thus the prompt) is obfuscated. The key insight, however, is that this obfuscated prompt provides two empirically validated guarantees:

• $\textbf{(Utility preservation)}$ The LLM output on the obfuscated prompt is close to the LLM output on the raw text.
• $\textbf{(Privacy guarantee)}$ The raw text prompt is hard to reverse-engineer from the obfuscated embeddings.

Conventional Flow

Private Flow

User Prompt "How does a black hole form?"

Embedder

d = 2,048

Embedder

d = 2,048

Stained Glass Transform

obfuscate embeddings
using a trained model

LLM

LLaMA 3.2 1B · frozen

LLM

LLaMA 3.2 1B · frozen

Response (conventional)

Response (prompt private)

Token Embeddings

d = 2,048 dims · 7 tokens

LLaMA 3.2 1B

Training the SGT

The SGT paper is well-written and in this post I have simply followed their approach. While I describe my implementation choices such as architecture (which may not be fully detailed in the paper for IP reasons), I encourage the reader to refer to the paper for further details. The high-level idea is that you run a small local network (called the SGT) that takes the embedding sequence and replaces it with a perturbed version. Thus, the server never sees tokens or raw embeddings; it only ever processes the scrambled version. The crux of the work is showing how to efficiently train the SGT to preserve embedding privacy while retaining utility — that is, LLM output quality should not degrade.

Embedding transformation

The transform is stochastic:

The SGT predicts a deterministic shift $\mu_\theta$ and a per-dimension noise scale $\sigma_\theta$. Adding Gaussian noise with a learned variance means no two passes produce the same obfuscated embeddings — which is important for resisting repeated-query attacks.

Architecture

SGT is trained per model and in this blog, I use the model from the paper — Llama 3.2 1B. I chose the following small post-norm transformer encoder as the SGT module placed in front of the frozen LLM:

class SGT(nn.Module):
    def __init__(self, embed_dim=2048, num_layers=2, nhead=8):
        super().__init__()
        enc_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=nhead,
            dim_feedforward=embed_dim * 2, dropout=0.0,
            batch_first=True, norm_first=False,  # post-norm keeps output bounded
        )
        self.encoder = nn.TransformerEncoder(
            enc_layer, num_layers=num_layers, norm=nn.LayerNorm(embed_dim)
        )
        self.mu_head        = nn.Linear(embed_dim, embed_dim)
        self.log_sigma_head = nn.Linear(embed_dim, embed_dim)
        # initialize as identity: mu=0, small sigma
        nn.init.zeros_(self.mu_head.weight);   nn.init.zeros_(self.mu_head.bias)
        nn.init.zeros_(self.log_sigma_head.weight)
        nn.init.constant_(self.log_sigma_head.bias, -2.0)

    def forward(self, x, padding_mask=None):
        h         = self.encoder(x.float(), src_key_padding_mask=padding_mask)
        mu        = self.mu_head(h)
        log_sigma = self.log_sigma_head(h).clamp(-6.0, 3.0)
        eps       = torch.randn_like(h)
        x_tilde   = x.float() + mu + log_sigma.exp() * eps
        return x_tilde.to(x.dtype), mu.to(x.dtype), log_sigma.to(x.dtype)

For Llama 3.2 1B (embed_dim=2048), this SGT has 75.5 M parameters — about 6% the size of the LLM it protects. It runs locally in milliseconds per token; the LLM never needs to move.

Loss functions

Training balances four objectives simultaneously: one utility loss and three obfuscation loss components (refer to the paper for more details — the authors explain the challenges and their choices well). I trained over 40K OpenOrca examples for 5000 steps on a Google Colab T4 GPU (best checkpoint was at step 4500).

Utility — the obfuscated sequence should produce the same distribution of next tokens as the clean sequence. I use KL divergence instead of hard-label cross-entropy, because a 128 K-vocab LM’s probability mass is spread across many tokens. Hard-label gradients are too sparse to compete with the obfuscation losses through 16 frozen transformer layers.

def loss_utility(logits_obf, logits_clean):
    log_p_obf = F.log_softmax(logits_obf.reshape(-1, V).float(), dim=-1)
    p_clean   = F.softmax(logits_clean.detach().reshape(-1, V).float(), dim=-1)
    return F.kl_div(log_p_obf, p_clean, reduction="batchmean")

AbsCosine — push the obfuscated embedding orthogonal to the original. If $\lvert\cos(\tilde{x}, x)\rvert$ is near zero, the nearest-neighbour attack can’t find the original token:

def loss_abscosine(x, x_tilde):
    cos = F.cosine_similarity(x.reshape(-1, D), x_tilde.reshape(-1, D), dim=-1)
    return cos.abs().mean()

Norm penalty — keep obfuscated norms close to clean norms per token, so the LLM’s internal normalizations behave as expected:

def loss_norm_penalty(x, mu):
    clean_norms  = x.float().norm(dim=-1).detach()
    shifted_norms = (x.float() + mu.float()).norm(dim=-1)
    return (shifted_norms - clean_norms).abs().mean()

Mutual information — a minibatch Monte Carlo estimate of \(I(\tilde{x}; x)\) in nats per dimension, computed in float64 to avoid cancellation. This directly minimizes how much information \(\tilde{x}\) retains about \(x\) across the learned distribution, not just pointwise:

def loss_mi(x_tilde_A, mu_A, log_sigma_A, x_clean_B, mu_B, log_sigma_B):
    # H(x̃ | x) from diagonal Gaussian component entropy
    H_comp = (0.5 * LOG_2PIE + log_sigma_A_64).sum(dim=(-1, -2)).mean()
    # H(x̃) ≈ -E[log p_mix(x̃)] via minibatch GMM
    log_prob = -0.5 * (diff.pow(2) / var_B + log_const).sum(dim=(-1, -2))
    H_mix    = -torch.logsumexp(log_prob, dim=1).mean() + math.log(B_B)
    return ((H_mix - H_comp) / (T * d)).float()

The final combined loss uses weights \((\alpha_u, \alpha_\text{acs}, \alpha_\text{norm}, \alpha_\text{mi}) = (2.0, 0.3, 0.05, 0.15)\). Getting these weights right took three iterations — the main failure mode is \(\alpha_u\) so large that the utility loss keeps \(\sigma\) tiny, leaving mutual information high throughout training. The loss curves are below:

Note that the training is probably a reasonable local optimum given that the privacy metrics and utility are worse than those reported in the paper.

Does the LLM output give away the input?

The paper covers simple attack baselines and the same authors also construct a better reconstruction attack called BeamClean [2]. Given that their attack only considers the embedding vector, I was curious to see if a stronger attacker — one that can also see the text output produced by the model — could improve on BeamClean. BeamClean [2] finds the top vocabulary candidates at each token position by cosine similarity to the obfuscated embedding, scores them with a language-model prior, and runs beam search.

I implemented two extensions that use the observed LLM output as an additional signal with regularization to prevent the language model from exploiting quirks in the garbled output (e.g., preferring Does over does for superficial reasons). The results, however, have been mixed and not significant enough to generalize broadly. While the current evidence suggests that BeamClean+output is no stronger than BeamClean alone, I leave it as an open question to rigorously verify.

Takeaways

SGT is a genuinely clever idea. The key insight is that if the embedding layer can be made public, you can separate it from the inference pipeline to achieve strong privacy. This can be a great middle ground where the model owner retains ownership of the model while the user gets prompt privacy.

• It is genuinely surprising to me that a model can be trained to achieve two contrary objectives well: (1) obfuscation and (2) utility preservation. In this regard, SGT feels just as innovative as fully homomorphic encryption.
• I was able to train the model from scratch with limited resources. This is largely a credit to the paper being well-written and speaks to the academic community’s culture of knowledge sharing.
• Fully reproducing the paper’s reported NN-FR of 0.93 likely requires significantly more than 5,000 training steps. My checkpoint is a useful proof of concept that the approach works directionally, though not yet at the privacy levels claimed for production use.
• The SGT approach can also be packaged to provide a solution to the two sided privacy problem of running private evaluation benchmarks. For instance, a model owner with a model $M$ and a benchmark owner with a dataset $D$ would both like to evaluate $M(D)$ under the constraint that $M$ needs to be private from the benchmark owner (for intellectual property reasons) and $D$ needs to be private from model owner (to prevent contamination, overfitting, gamifying external auditing).

References

1. J. Roberts, K. Mylonakis, S. Roy, and K. Kale. “Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transform.” arXiv:2506.09452, 2025. To appear at IEEE S&P 2026. arxiv.org/abs/2506.09452
2. K. Kale, K. Mylonakis, J. Roberts, and S. Roy. “BeamClean: Language Aware Embedding Reconstruction.” arXiv:2505.13758, 2025. arxiv.org/abs/2505.13758

The intersection of LLMs and mathematical research is a fascinating one. Mathematics is objective and verifiable, and today’s frontier models have been trained extensively on its concepts and methods. There is a particular sense of joy and wonder in watching LLMs solve previously unsolved (and genuinely difficult problems), while knowing that their capabilities emerge from mechanisms that are, at their core, surprisingly simple.

To me, this is an extraordinary testament to what humans can achieve collectively through ingenuity, hard work, and engineering. Like many others working in this space, I became interested in testing the limits of LLMs on open mathematical problems through my own lens. This post tells the story of that process.

Introduction

You have 4 cards from a standard deck of cards: Ace, King of Hearts and Spades (A♥, K♥, A♠, K♠). You can point to a card and ask one of two possible questions:

• Is this card Red?
• Is this card a King?

These 4 cards are now shuffled. It is easy to find which card is which by asking 8 questions (each question on each card). However, your goal is to come up with a strategy for asking questions such that you can always find which card is which in 5 or fewer questions. Try your hand at the interactive media below. Drag a question from the right onto any card to learn one fact about it, then click “I know it!” once you know for sure the full arrangement.

This post is about using an AI to solve an open question posed around a scaled version of this simple puzzle. The rest of the post is organized into these sections:

• 16-card version scales the opening 4-card puzzle to 16 cards and then states the open mathematical question.
• The bulk covers the search for optimal strategies: the LLM-agent execution setup, how performance evolved across fifty iterations, and a playable browser version of the game (jump here to see the main result).
• I close with key learnings from the collaboration and connections to related open conjecture by Donald Knuth that was recently disproved [5].

16-card puzzle

This small problem can be extended to larger space and things get interesting there. You have sixteen unique cards (each encoding a 4-bit value \(0,1,\dots,15\)) and four binary properties. When you check property \(b\) of a card, you learn a single bit: bit \(b\) of that card’s hidden value. For instance, property $3$ reveals the most significant bit (MSB) of the card whereas property $0$ reveals its least significant bit (LSB).

It is easy to see that four property checks fully determine any given card. The catch: the cards are all shuffled face-down, so you don’t know which card is which. You can trivially do this in 64 checks (each property check on each card). The puzzle is to recover the whole arrangement (which card is which) in under 50 queries in the worst case. Can you do it? (You can play with smaller variants of the problem below.)

An open question

Underneath the story is an interesting permutation-based puzzle. A hidden permutation $\sigma$ scrambles ${0,\dots,15}$; a query names a (card, property) pair and returns that bit of that card’s hidden value $\sigma(\text{card})$. You stop the instant only one permutation is still consistent with every answer so far. Let $D(16)$ be the $\textbf{fewest number of queries that always suffice}$ even against an adversary who answers every query as unhelpfully as the facts still allow. The question I ask is: what is the exact value of $D(16)$?

$\textbf{Information-Theoretic Analysis.}$ Counting sets a floor. There are \(16!\) possible arrangements and each answer is one bit, so no strategy can ever beat $D(16) \ge \lceil \log_2 16! \rceil = 45.$ And the lazy strategy — check all four properties of all sixteen cards — uses $16 \times 4 = 64$. Somewhere in $[45, 64]$ sits the true answer, but where?

Note that there are many ways to formulate this problem, each connected to a different body of mathematics: as an adaptive bit-query optimal decision tree; as matchings in a bipartite graph, in turn connected to permanent computations; automorphisms to reduce the state-space size; and strategies around dynamic programming or brute-force traversal. Also, despite it being an open problem, I would classify it under the “attentionless” problems coined by Terence Tao.

The search for optimal strategies

I approached computing the optimal strategy from two sides. First, the set of strategies I could reason out by hand (each strategy a provable upper bound) but there was only so far I could go by pen and paper. Then I started with the main exploration: an LLM agent, turned loose on the search with a meta-harness, grinding the best-known strategy. This section lays out the results of that exploration. Before any search, it’s worth walking down the ladder of things a careful human would try (I encourage you to try it yourself) – each strategy gives a provable upper bound. The rest of this post is about the exploration of using AI to attempt this puzzle and learnings along the way.

The execution setup

I built a meta-harness and let an LLM agent run the research loop itself: propose an idea, implement it, run it, read the numbers, propose the next idea — across dozens of iterations. Rules and constraints that were integral in this exploration:

• $\textbf{Resource constraints.}$ All experiments were run on a single MacBook with no clusters/GPU farms on tap. A strategy has to earn its keep inside what a single box can spare which, as we’ll see, is exactly what makes the problem so stubborn. The longest runs were big exhaustive searches run unattended over weekends (and then some).
• $\textbf{One idea, one folder.}$ Every attempt (v1, v2, … v50) is a self-contained directory with its own solver, verifier, and notes. Results are isolated and reproducible; nothing leaks between versions. The failures are kept next to the wins. Only major optimizations and programming efforts spill as dependencies into other directories.
• $\textbf{Strategic steers.}$ Initially the agent required directional guidance and manual steer which eventually reduced with appropriate harness guardrails.

Agent performance vs time

Here’s the whole campaign at a glance – we’ll unpack what the numbers mean below. Each point is one idea the agent tried. A solid dot is a proven worst case – an exhaustive minimax search that finished inside the budget. An error bar is a sampled heuristic: the average over 1000 random shuffles, with a whisker up to the worst case actually observed. The bold staircase is the best guaranteed strategy known at each stage – it only steps down when a new method proves it can always do better. The dashed line is the information floor, 45. Hover a point (tap on mobile), or use ← / →.

The shape of the story: the staircase drops fast through the human-reachable strategies (64, 60, 49, 47), then stalls at 46 – where an exact, structured solver finally proves a guarantee. While the heuristics swarm below it on typical shuffles (averaging ~44.5) yet can never promise it. There’s an independent problem of finding strategies that are good on average (but not worst case). But in this case, I focused on the worst case, i.e., guaranteeing that a certain bound suffices.

Explore the game yourself

I built the actual game in the browser so you can try the strategies yourself. Surprisingly – given how large the exhaustive searches were for the 16 card variant – all of them fit inside this single browser tab without noticeable performance issues. Be sure to explore the Learned policy (a small NN distilled from an expert) and the 3+1 quotient decomposition for the 16 card game.

Two things tend to jump out once you play. First, a typical shuffle collapses to a single answer well before 45 queries. It is useful to remember here that the floor is a worst-case promise, not an average. Second, the greedy strategy is eerily good on average yet still trips into 46 on the nastiest shuffles, which is the whole reason “prove 45” is hard and “prove 46” needed a weekend search. I encourage you to try all the different strategies of the 16-card game (despite their complexity, they run in your browser).

Learnings from LLM-assisted solving

Here are a few learnings from it, each tagged + for an upside or − for a limitation.

Related work

A very closely related problem was originally proposed as a conjecture by Knuth and disproved by Florian Stober and Armin Weiß in 2022 [5].

The problem is for comparison-based sorting (excluding sorts such as radix sort that require no comparisons). Questions ask what is the minimum number of comparisons required to sort a group of $n$ elements. They call this number $S(n)$, and fun puzzles along the way involve showing $S(4) = 5$ and $S(5) = 7$. In general, though, $S(n)$ has no known closed form. The best general algorithm is merge-insertion sort (Ford & Johnson, 1959 [2]), whose comparison count $F(n)$ exactly matches the information-theoretic floor $\lceil \log_2 n! \rceil$ for every $n \le 11$ – which is what might have led Knuth to conjecture it might be optimal for all $n$ [1]. It is not: Manacher [3] showed merge-insertion is beaten for large $n$, and the counting floor itself first falls short at $n = 12$, where $F(12) = 30$ but $\lceil \log_2 12! \rceil = 29$. The smallest genuinely stubborn case was $n = 16$: the floor is $45$, merge-insertion spends $46$, and for decades nobody could prove which was right. Stober and Weiß settled it – $S(16) = 46$ – with an exhaustive bidirectional search on a single workstation, using an incomplete backward search as pruning advice for a complete forward one [5]. (That backward-as-advice trick is the closest thing to a lit exit sign our own $D(16)$ campaign has found; it is exactly what v48 executes.)

Sorting is not (obviously) our problem. It is tempting to reduce one to the other – both identify a hidden permutation of $n$ items through binary, adversarial queries against the very same $\lceil \log_2 n! \rceil$ counting floor. But they reason about different objects. Comparison sorting lives on posets: each comparison refines a partial order, and the adversary’s remaining freedom is the number of linear extensions still consistent with it. Our bit-query game lives on bipartite compatibility: each probe pins one coordinate of one value, and the number of arrangements still consistent is the permanent of a $0/1$ matrix. I could not find a correspondence that carries a bound from one problem to the other – a comparison is not a coordinate reveal, and a permanent is not a count of linear extensions – so the numerical similarity/connection below is, as far as I can tell, two problems related but unrelated at the same time. If anyone can find a real connection between them, I am sure it will make for some insightful new mathematical work.

Here are both sequences against their shared floor. Every $S(n)$ is known; $D(n)$ is settled through $n = 12$, still open for $n = 13, 14, 15$, and conjectured at $n = 16$:

green = a gap above the floor  ·  red = a bit-query gap that differs from sorting's. Every settled $D$ gap is red; whether the still-open $n = 16$ entry matches sorting as we conjecture or rather surprisingly, 45 is sufficient is the whole question now.

Read down the columns and the two problems part ways and then meet again. Sorting hugs the floor: $S(n)$ equals $\lceil \log_2 n! \rceil$ for every $n \le 11$, first steps above it at $n = 12$, and famously stalls one probe over at $n = 16$. The bit-query game leaves the floor almost immediately (a $+1$ gap already at $n = 5$) and then wanders, even reaching $+2$ at $n = 10$. But at $n = 8$ the gap snaps back to $0$. In fact the bit-query game hits the floor exactly at every power of two below sixteen ($n = 2, 4, 8$), because $2^k$ values tile the $k$-bit cube with no wasted codewords and nothing is ever half-asked.

The wandering in between is worth a caveat. When $n$ is not a power of two, the values ${0, \dots, n-1}$ only partly fill the $k$-bit cube: $2^k - n$ codewords go unused, so the game there is lumpier, far less symmetric, and its gaps are tangled up with that encoding waste rather than with anything fundamental. Those non-power-of-two gaps therefore say little about the clean powers-of-two cases: a $+2$ at $n = 10$ signals nothing for $n = 16$. Restricted to the powers of two (2, 4, 8, and 16), the only honest reading of the $D$ column is $0, 0, 0, \, ?$.

While that makes $n = 16$, the next power of two, look like it should land right on the floor at $45$ (the very same “surely it’s tight here” intuition Knuth probably had about sorting). Both intuitions appear to break at sixteen: the best strategy I have found proves the upper bound of $D(16) \leq 46$, one probe above the floor – and if that is optimal, it closely echoes the proven $S(16) = 46$. For me, it would be very surprising if $D(16)$ happens to be 45. Anyway, whether this relation is a coincidence, or the shadow of a deeper mathematical connection I simply couldn’t find, is the question this whole exercise left me with.

Summary

A quick summary of the problem so far. Note that $D(16)$ is the worst case queries to identify a permutation uniquely using bit-queries):

• Proved: $45 \leq D(16) \leq 46$
• Proved: The 3+1 decomposition requires exactly 46 in the worst case.
• Not proved: $D(16)=46$ (alternative open question: whether a fully interleaved 45-query strategy exists).
• Empirical: greedy typically finishes around 45, but this is not a guarantee.
• There are some interesting ML based approaches which approach the problem from an empirical angle (as opposed to provably X questions are required in the worst case).

References

1. D. E. Knuth. The Art of Computer Programming, Vol. 3: Sorting and Searching (2nd ed.), §5.3.1 (minimum-comparison sorting; the merge-insertion conjecture). Addison-Wesley, 1998.
2. L. R. Ford Jr. and S. M. Johnson. “A tournament problem.” The American Mathematical Monthly 66(5): 387–389, 1959. doi:10.2307/2308750
3. G. K. Manacher. “The Ford–Johnson sorting algorithm is not optimal.” Journal of the ACM 26(3): 441–456, 1979. doi:10.1145/322139.322145
4. M. Peczarski. “The Ford–Johnson algorithm still unbeaten for less than 47 elements.” Information Processing Letters 101(3): 126–128, 2007. doi:10.1016/j.ipl.2006.09.001
5. F. Stober and A. Weiß. “Lower bounds for sorting 16, 17, and 18 elements.” arXiv:2206.05597, 2022. Proves $S(16) = 46$. arxiv.org/abs/2206.05597
6. H. J. Ryser. Combinatorial Mathematics (Carus Mathematical Monographs 14). Mathematical Association of America, 1963. Ryser’s inclusion–exclusion formula for the permanent – used throughout the interactive game to count consistent arrangements.
7. L. G. Valiant. “The complexity of computing the permanent.” Theoretical Computer Science 8(2): 189–201, 1979. Proves computing the permanent is #P-complete. doi:10.1016/0304-3975(79)90044-6

Privacy regulation and AI sovereignty create an incentive towards open-weights-only stacks. I wanted to explore how close is the frontier of open-source only models in comparison to closed sourced frontier models (Opus and ChatGPT). I ran a small benchmark study to measure three axes at once: accuracy, cost, and speed.

Evaluation setup

In my open open source setup models, I use the following set DeepSeek-V4-Pro, Qwen3-235B-Thinking, GLM-5.2. I used four ensembling strategies:

• Synthesizer: Several OSS models answer concurrently, then an OSS synthesizer model reads every member’s answer and its reasoning and writes the final answer by deciding which line of reasoning to trust
• Consensus adjudicator: This is shown the vote tally of the responses of multiple OSS models, it defaults to the majority answer and overturns it only when it can name a concrete, specific error in the consensus reasoning thereby fixing the synthesizer’s habit of talking itself into a lone wrong answer
• Self-consistency: Each OSS model answers k=3 times and the final answer is the plurality vote, with no aggregator model reading anything (OpenRouter Fusion also used this)
• OpenRouter Fusion: OpenRouter’s own hosted mixture-of-models router that blends multiple providers behind a single endpoint (turns out to be the cheapest option on all five benchmarks).

No closed model was used anywhere in the ensembling stack, not even as judge. For frontier models, I used Claude Opus 4.8 and ChatGPT 5.5. For the benchmarks, I used GPQA-Diamond (198 questions), AIME 2025 (30), SuperGPQA (500), HLE-math (128), SimpleQA (1000). Finally, I used OpenRouter via Inspect to run the evaluation pipelines, with live per-token pricing for cost.

Comparison with Claude Opus 4.8

I used Claude Opus 4.8 as the reference bar. ChatGPT was generally strictly better on these benchmarks but the ensembles had mixed performance comparison. Scroll through this visual to see the results in a nutshell.

The accuracy finding is more nuanced than a single number lets on: open source wins where benchmarks are dense enough for ensembling to help, and loses where the questions are genuinely at the frontier of difficulty. GPQA-Diamond and SuperGPQA fall in the first category; HLE-math and AIME fall in the second. Ensembling can also have big impact on costs with simple tasks being delegated to local models.

Cost and Speed tradeoff

Accuracy parity is only half the story. The chart below shows the full trade-off space for the models with costs in $ also included.

On cost, open source can undercut the frontier substantially. This to me was the largest insight. Fusion (by OpenRouter) was the cheapest contestant on all five benchmarks: $1.33 vs ChatGPT’s $33.15 on SimpleQA,^† $2.79 vs $23.92 on HLE. The catch is that naive ensembling goes the other direction: self-consistency, which samples every model several times, was the most expensive option of all (up to $66 on SimpleQA). How you ensemble matters more than whether you ensemble.

On speed, open source loses. Flip to vs Speed. A single frontier call returns in seconds to minutes; open ensembles ran ten times longer or more (HLE: ~20 minutes per question for some strategies). Fanning out to three reasoning models and synthesizing is inherently slow. No amount of accuracy parity changes this, and it is the honest weak spot of the approach.

^† SimpleQA Fusion uses OpenRouter's built-in web search, not a tool-free setup — which explains the accuracy spike to 97%.

Closing thoughts

Open source models have truly come a long way. However, frontier models certainly are worth their premium because as the envelope of capabilities increases, the bar of problems users want to tackle also raises higher.

Start with the observation that a table lookup is secretly an inner product. Encode the index as a one-hot vector \(e_i\) (all zeros except a single 1 in slot \(i\)) and then

The lookup is the dot product of a selector with the table. Now the key idea: $\textbf{Parseval’s theorem.}$ Under an orthonormal transform — Fourier, or here a wavelet transform — the inner product is identical in both domains:

So you’re free to compute the lookup in whichever domain is cheaper. The protocol moves the table into the wavelet domain, where the discrete wavelet transform concentrates its structure into a handful of large coefficients (the rest are tiny). The secure inner product against the transformed selector then costs far less work — and by Parseval it lands on exactly the same answer, \(T[i]\).

That’s the whole trick: a lookup is an inner product, an inner product doesn’t care which domain you compute it in, and the wavelet domain is where it’s cheap. While the fundamental idea is simple, there is a lot of mathematics that makes DWT compress a lookup table well. The ability to be able to draw the higher order terms is what provides the compression. And in terms of bit length, this might translate into a lookup table of size $2^{29} \approx$ half a billion entries to a look up of aobut $2^{12} \approx 4000$ entries.

PDF Code Wave Hello to Privacy · PETS 2025

“If you can’t explain something to a first-year student, then you haven’t really understood it.” — Richard Feynman

I was recently asked to explain some of my research to a broad, non-technical audience, and this quote started nagging at me: did I really understand secure computation well enough to make it that simple? Thankfully, a delightful corner of the field came to my rescue.

A modern dilemma

You’ve just met someone wonderful at a party, and you’d each privately like to answer the same question: would I want to swap numbers with this person?

The trouble is that neither of you wants to say it out loud. Admitting “yes” to someone who would have said “no” is exactly the kind of face-losing moment you’d rather avoid. So what you really want is only the combined answer: do we both want to exchange numbers? You’ll swap only if both of you said yes — like a dating app’s match, but with no app (and no company) sitting in the middle. How do you pull that off?

Mathematically, each of you holds a private bit — yes (1) or no (0) — and you want to compute one function of them: their logical AND. The hard part is the privacy requirement: you must learn the value of that AND and nothing else. If the answer comes out “no,” neither of you should be able to tell whether it was one no or two.

Remarkably, you can do exactly this with nothing but a handful of playing cards. Scroll through it below — the cards in the panel respond as you read — and at the end, you can run the whole thing yourself.

What you just watched

This is the five-card trick, the 1989 protocol by Bert den Boer that kicked off a small, delightful corner of cryptography called card based cryptography. The five-card trick is one entry into the field called secure multiparty computation (MPC): a set of techniques for letting people compute a shared answer from private inputs that none of them ever has to reveal. The cards make the magic tangible, but the structure is exactly the same one used in the cryptographic version:

• Commitment — fixing a secret value so it can’t be changed or read (face-down cards; in real systems, cryptographic commitments or secret shares).
• Local computation — operations each party performs on their own (turning a card; locally combining shares).
• Communication — the shared step that blends secrets without exposing them (the random cut; in real MPC, exchanging masked messages over many rounds).

And the closing caveat is real too: removing the trusted referee isn’t free. Practical MPC pays for privacy in extra computation and communication, which is why a huge amount of research goes into making it cheap enough to deploy — for things like hospitals jointly studying patient outcomes, banks detecting fraud across institutions, or auctions where no one sees the losing bids.

If you want to go deeper, the original protocol is B. den Boer, “More efficient match-making and satisfiability: the five-card trick” (EUROCRYPT ‘89), and the modern survey work by Mizuki, Sone, and others on card-based cryptography is a wonderful rabbit hole.

Here’s the catch that never shows up in a complexity analysis. When you train privately, you’re flying blind. You can’t inspect activations, gradients, or intermediate values — that’s the entire point of privacy. So when the model doesn’t learn, you’re left with one ambiguous signal: the accuracy is low.

Low accuracy could mean almost anything: a real optimization difficulty, a subtly wrong fixed-point truncation, an overflow in one layer. In cleartext you’d drop in a print and know in minutes. Under MPC every value is a secret share — there is nothing to print.

For secure computation to achieve it’s dream, we will need contributions not just about raw speed but also about robust, trustworthy primitives and sufficiently debuggability. As systems become increasingly capable at exploration, debugging, and hypothesis generation, the primary bottleneck in privacy-sensitive analytics will shift from executing computations to discovering them. Future privacy-preserving systems will therefore optimize not only for confidential execution, but for confidential exploration: enabling humans and AI agents to iteratively search over vast spaces of models, queries, and hypotheses while revealing substantially less raw data than is required today. The long-term equilibrium is neither unrestricted data access nor fully opaque one-shot computation, but a spectrum of interactive discovery mechanisms that provide just enough information to find useful computations without exposing the underlying data itself.

PDF Code Piranha · USENIX Security 2022