Warm-up tokens · strings

Split into words

Before a model reads anything, the text gets chopped into pieces.

A tokenizer breaks text into small pieces called tokens. The simplest one splits on everything that is not a letter or a digit.

Write tokenize(text): text is a string. Return an array of lowercase tokens, where a token is a run of letters a to z or digits 0 to 9. Everything else (spaces, punctuation, apostrophes) only separates tokens and is dropped. Empty text gives [].

Call show(token) each time you find a token and the stage prints them one per line.

Example: tokenize("Hi, there.") returns ["hi", "there"].

Tests you can see

tokenize("Hi, there.")["hi","there"]
tokenize("Version 4 shipped in 2023's spring")["version","4","shipped","in","2023","s","spring"]
tokenize(" the cat ")["the","cat"]

More tests are hidden. The server runs them all before a time counts.

The idea behind this, in a two-minute live page

Call show() while your code works and the run plays back here.

javascripttokenize()

function tokenize(text) {
  // lowercase, then split on anything that is not a-z or 0-9
  return text.split(" ");
}

function tokenize(text) {
  // lowercase, then split on anything that is not a-z or 0-9
  return text.split(" ");
}

Tab indents · Cmd/Ctrl+Enter runs