Byte pair merges
How real tokenizers invent their own word pieces from raw letters.
Byte pair encoding builds tokens by noticing which neighbours show up together most and gluing them into one piece. Repeat that a few times and common chunks like "th" or "ing" become single tokens.
Write bpeMerge(text, merges): start with text split into single characters. Then do up to merges rounds. In each round, count every adjacent pair of tokens and pick the most frequent. On a tie, pick the pair whose first appearance is earliest. If the best pair appears fewer than 2 times, stop early. Otherwise, walk left to right and replace each occurrence of that pair with the two tokens joined together (an occurrence that overlaps one you just merged is skipped). Return the final array of tokens.
Call show(tokens.join(" ")) at the start and after each merge, and the stage prints the text getting chunkier line by line.
Example: bpeMerge("aaab", 1) returns ["aa", "a", "b"].
Tests you can see
bpeMerge("aaab", 1)["aa","a","b"]bpeMerge("lowlowerlowest", 2)["low","low","e","r","low","e","s","t"]bpeMerge("singringking", 3)["s","ing","r","ing","k","ing"]
More tests are hidden. The server runs them all before a time counts.
The idea behind this, in a two-minute live pageCall show() while your code works and the run plays back here.
Tab indents · Cmd/Ctrl+Enter runs