Looped Transformer

Splashery stretches a drumhead over this Splash's outline, twelve inches across, and finds the ways it can ring. Its lowest is 298.8 Hz, 70 cents flat of E♭4, the nearest tine on the comb. The next fifteen are its overtones, and each one draws its own figure: sand gathers on the lines that stay still.

Tap it: the tiles loop through one block, sharper each lap, until 3 + 4 = 7.

A looped transformer runs the same transformer block again and again, feeding its output back in as its next input, instead of stacking many different layers. Each pass through the loop can refine the answer a little more, a bit like checking your work, so a small model can spend longer thinking about a harder problem.

Tap it: five tiles ride three laps around a track through one block. Each pass sharpens every tile one step, from noise to a blocky mosaic, to nearly right, to exact, until “3 + 4 = 7” settles with the 7 in gold. Pick the poster or a 3D model in the Splash tab.

Repeating one block is a valid idea in recurrent transformer designs. Here three laps reveal stored pictures of 3 + 4 = 7; no transformer block or arithmetic inference runs.

1. Simplified

   Transformer computation can repeat across depth.¹

   The single block and returning track illustrate a recurrent-depth architecture.

2. Correct

   The splash ends with the correct arithmetic statement 3 + 4 = 7.²

   The five stored marks are 3, +, 4, =, 7; the last tile images contain only finished bitmap values.

3. Simplified

   The three laps illustrate refinement rather than learned reasoning.¹

   Each lap chooses another stored bitmap level, with no attention, recurrent tensor update, or learned stopping rule.

1. 1Universal Transformers, Dehghani and coauthors, 2018
2. 2Multilayer perceptrons, Dive into Deep Learning authors

Simplified

The noise, coarse, nearly finished, and exact tile images are prepared stages. The lap count is fixed at three and is not chosen by difficulty.

Made in Splashery. Its code is MIT.

This link opens it in the Playground. To put it on your own page, add the element and one script tag. Every option.