Frame-grid lets a model "watch" a video as numbered grids of frames. AuraGraph is its sibling for sound: audio becomes tiles, image panels plus a short text sidecar, that any model with image input can read. Left to right is time, bottom to top is pitch, brightness is loudness. There are no audio tokenizers for these models, so sound has to arrive as something they can see and count.
Try it
Paint a Tile, Hear a Tile toyDraw sound as a picture and play it. Or drop in a recording, see its tile, paint on it, and hear it rebuilt from the picture alone. Lo-fi knobs melt the tune or smear the rhythm.
AuraGraph app v0.2Drop in a recording, get tiles a model can read (the machine view 24 blind readers picked), data tiles AuraLoom can play, and a READ-ME-FIRST for your AI friend. One file, offline, nothing uploaded.
What we've found so far
- Models can read tiles. Blind reads of real songs, graded against human ears, found voices, hits, ladders of overtones, bass lines, tempo and structure. FINDINGS · blind reads · outside test
- A tile can be played back. From the picture alone, 94% of the melody returns, frame for frame. It sounds like a cover, not the recording. INVERSION-1
- Melody lives on the pitch axis, rhythm on the time axis. Coarsen one and the other survives. What makes a tile readable as music is exactly what makes it playable.
- Which drawing do models read best? Plain gray, with the grid burned in: 95% for the models that can read tiles, against 83% for the old colour map. Readers fool themselves the same way whatever the drawing: they bend a real mark to fit the question. LEGIBILITY-1
- Where it could go: a model that sings by drawing, a diffusion model that paints music, music as a PNG. SPEC-SEEDS
Developed in public
Everything in the repo is working notes, written as we went, with the mistakes and corrections left in. The Python prototypes are in prototype/; freely licensed reference recordings (with their licences) are in samples/reference/. Tiles of copyrighted songs are not published in full, because they play back. The reasoning is in INVERSION-1.