This dialog is used to create the enriched transcript of the 'Let's build the GPT Tokenizer' video

Raw Transcript (from Scribe)

TIP: To make markdown images visible to AI you can add #ai anchor tags to image urls like [](image_path#ai)*

TIP: To visualize the auto-cropped version of an image you can change the image url from [](image_{idx}.jpg) to [](image_{idx}_cropped.jpg)*

TIP: add_tscript_msg will add the full transcript as a single note msg, then you can ask questions about the transcript overall, such as the external resources that are mentioned, a high level summary, speakers, etc... Once you are done don't forget to hide that message to save token space.

add_tscript_msg(scribe_dst, yt_video_id='zduSFxRajkE')

Introduction to Tokenization

[00:00] Andrej Karpathy: Hi everyone. So in this video, I'd like us to cover the process of tokenization in large language models. Now, you see here that I have a sad face, and that's because, well, tokenization is my least favorite part of working with large language models. But unfortunately, it is necessary to understand in some detail because it is fairly hairy, gnarly, and there are a lot of hidden foot guns to be aware of. And a lot of oddness with large language models typically traces back to tokenization.

A Jupyter Notebook open in a web browser. The first cell is a Markdown cell with the text 'Tokenization :('

[00:25] Andrej Karpathy: So what is tokenization? Now, in my previous video, "Let's build GPT from scratch," we actually already did tokenization, but we did a very naive, simple version of tokenization.

A YouTube video page titled 'Let's build GPT: from scratch, in code, spelled out.' by Andrej Karpathy.

[00:36] Andrej Karpathy: So when you go to the Google Colab for that video, you see here that we loaded our training set. And our training set was this Shakespeare dataset. Now in the beginning, the Shakespeare dataset is just a large string in Python. It's just text. And so the question is, how do we plug text into large language models?

A Google Colab notebook titled 'Building a GPT'. The screen shows the beginning of the Shakespeare dataset text.

[00:57] Andrej Karpathy: And in this case here, we created a vocabulary of 65 possible characters that we saw occur in this string. These were the possible characters, and we saw that there are 65 of them. And then we created a lookup table for converting from every possible character, a little string piece, into a token, an integer.

[01:18] Andrej Karpathy: So here, for example, we tokenized the string "hi there" and we received this sequence of tokens. And here we took the first 1,000 characters of our dataset and we encoded it into tokens. And because this is character level, we received 1,000 tokens in a sequence. So token 18, 47, etc.

Python code in a Colab notebook showing the tokenization of the first 1000 characters of a dataset into a sequence of 1000 integers.

[01:41] Andrej Karpathy: Now, later we saw that the way we plug these tokens into the language model is by using an embedding table. And so basically, if we have 65 possible tokens, then this embedding table is going to have 65 rows. And roughly speaking, we're taking the integer associated with every single token, we're using that as a lookup into this table, and we're plucking out the corresponding row. And this row is a, uh, is trainable parameters that we're going to train using backpropagation. And this is the vector that then feeds into the transformer, um, and that's how the transformer sort of perceives every single token.

Python code defining a BigramLanguageModel class. The line 'self.token_embedding_table = nn.Embedding(vocab_size, n_embed)' is highlighted, showing the creation of an embedding table.

[02:19] Andrej Karpathy: So here we had a very naive tokenization process that was a character-level tokenizer. But in practice, state-of-the-art, uh, language models, people use a lot more complicated schemes, unfortunately, for, uh, constructing these, uh, token vocabularies. So we're not dealing on a character level, we're dealing on a chunk level. And the way these, um, character chunks are constructed is using algorithms such as, for example, the byte-pair encoding algorithm, which we're going to go into in detail, um, and cover in this video.

Tokenization in GPT-2 and Llama 2

[02:52] Andrej Karpathy: I'd like to briefly show you the paper that introduced byte-level encoding as a mechanism for tokenization in the context of large language models. And I would say that that's probably the GPT-2 paper. And if you scroll down here to the section "Input Representation," this is where they cover tokenization, the kind of properties that you'd like the tokenization to have. And they conclude here that they're going to have a tokenizer where you have a vocabulary of 50,257 possible tokens. And the context size is going to be 1,024 tokens. So in the, in the attention layer of the transformer neural network, every single token is attending to the previous tokens in the sequence, and it's going to see up to 1,024 tokens. So tokens are this fundamental unit, um, the atom of, uh, large language models, if you will. And everything is in units of tokens, everything is about tokens. And tokenization is the process for translating strings or text into sequences of tokens and, uh, vice versa.

A screenshot of the GPT-2 research paper, highlighting the 'Input Representation' section which discusses Byte Pair Encoding (BPE), a vocabulary size of 50,257, and a context size of 1024 tokens.

[03:55] Andrej Karpathy: When you go into the Llama 2 paper as well, I can show you that when you search "token," you're going to get 63 hits. Um, and that's because tokens are, again, pervasive. So here they mentioned that they trained on 2 trillion tokens of data and so on. So we're going to build our own tokenizer. Luckily, the byte-pair encoding algorithm is not, um, that super complicated, and we can build it from scratch ourselves and we'll see exactly how this works.

A screenshot of the Llama 2 research paper. The text 'Our training corpus includes a new mix of data from publicly available sources, which does not include data from Meta’s products or services. We made an effort to remove data from certain sites known to contain a high volume of personal information about private individuals. We trained on 2 trillion tokens of data as this provides a good performance-trade-off.' is visible.

The Weirdness of Tokenization

[04:20] Andrej Karpathy: Before we dive into code, I'd like to give you a brief taste of some of the complexities that come from the tokenization because I just want to make sure that we've motivated it sufficiently for why we are doing all this and why this is so gross. So, tokenization is at the heart of a lot of weirdness in large language models, and I would advise that you do not brush it off. A lot of the issues that may look like just issues with the neural architecture or the large language model itself are actually issues with the tokenization and fundamentally trace back to it.

[04:50] Andrej Karpathy: So, if you've noticed any issues with large language models can't, you know, not able to do spelling tasks very easily, that's usually due to tokenization. Simple string processing can be difficult for the large language model to perform natively. Uh, non-English languages can work much worse, and to a large extent, this is due to tokenization. Sometimes LLMs are bad at simple arithmetic, also can trace be traced to tokenization. Uh, GPT-2 specifically would have had quite a bit more issues with Python than, uh, future versions of it due to tokenization. There's a lot of other issues. Maybe you've seen weird warnings about a trailing whitespace. This is a tokenization issue. Um, if you had asked GPT earlier about "SolidGoldMagikarp" and what it is, you would see the LLM go totally crazy and it would start going off about completely unrelated tangent topic. Maybe you've been told to use YAML over JSON with structured data. All that has to do with tokenization. So basically, tokenization is at the heart of many issues.

A Jupyter Notebook with a Markdown cell titled 'Tokenization :('. It contains a bulleted list of issues in LLMs that are caused by tokenization, such as problems with spelling, string processing, non-English languages, arithmetic, and Python code.

Visualizing Tokenization

[05:50] Andrej Karpathy: I will loop back around to these at the end of the video, but for now, let me just, um, skip over it a little bit. And let's go to this web app, um, the tiktokenizer that vercel.app. So I have it loaded here. And what I like about this web app is that tokenization is running sort of live in your browser in JavaScript. So you can just type here stuff, "hello world," and the whole string re-tokenizes.

A web application called 'Tiktokenizer'. The left pane is a text editor, and the right pane shows the tokenized output. The text includes examples of English, arithmetic, Korean, and Python code.

[06:15] Andrej Karpathy: So, here what we see on the left is the string that you put in. On the right, we're currently using the GPT-2 tokenizer. We see that this string that I pasted here is currently tokenizing into 300 tokens. And here they are sort of, uh, shown explicitly in different colors for every single token. So for example, uh, this word "Tokenization" became two tokens, the token 30,642 and 1,634. The token " is" is token 318. Be careful, on the bottom you can show whitespace, and keep in mind that there are spaces and, uh, slash n, new line characters in here, but you can hide them for clarity. The token " at" is token 379. The token " the" is 262, etc. So you notice here that the space is part of that, uh, token chunk.

[07:16] Andrej Karpathy: Now, so this is kind of like how our English sentence broke up, and that seems all well and good. Now, now here I put in some arithmetic. So we see that, uh, the token 127 plus and then token 6, space 6, followed by 77. So what's happening here is that 127 is feeding in as a single token into the large language model, but the, um, number 677 will actually feed in as two separate tokens. And so the large language model has to, uh, sort of, um, take account of that and process it correctly in its network. And see here, 804 will be broken up into two tokens. And it's all completely arbitrary. And here I have another example of four-digit numbers, and they break up in a way that they break up, and it's totally arbitrary. Sometimes you have, um, multiple digits, a single token. Sometimes you have individual digits as many tokens, and it's all kind of pretty arbitrary and comes out of the tokenizer.

[08:15] Andrej Karpathy: Here's another example. We have the string "Egg." And you see here that this became two tokens. But for some reason when I say, "I have an egg," you see when it's a " an egg," it's two tokens, it's, sorry, it's a single token. So just "Egg" by itself in the beginning of a sentence is two tokens, but here as a " an egg" it's suddenly a single token for the exact same string. Okay? Here, lowercase "egg" turns out to be a single token, and in particular, notice that the color is different, so this is a different token. So this is case sensitive. And of course, uh, capital "EGG" would also be different tokens, and again, um, this would be two tokens arbitrarily. So for the same concept, "egg," depending on if it's in the beginning of a sentence, at the end of a sentence, lowercase, uppercase, or mixed, all this will be, uh, basically very different tokens and different IDs. And the language model has to learn from raw data from all the internet text that it's being trained on that these are actually all the exact same concept. And it has to sort of group them in the parameters of the neural network and understand just based on the data patterns that these are all very similar, but maybe not almost exactly similar, but very, very similar.

[09:30] Andrej Karpathy: Um, after the egg demonstration here, I have, um, an introduction from OpenAI's ChatGPT in Korean. So, "mannaseo bangawoyo," uh, etc. Uh, so this is in Korean. And the reason I put this here is because you'll notice that, um, non-English languages work slightly worse in ChatGPT. Part of this is because, of course, the training dataset for ChatGPT is much larger for English than for everything else. But the same is true not just for the large language model itself, but also for the tokenizer. So when we train the tokenizer, we're going to see that there's a training set as well. And there's a lot more English than non-English. And what ends up happening is that we're going to have a lot more longer tokens for English.

[10:17] Andrej Karpathy: So, how do I put this? If you have a single sentence in English and you tokenize it, you might see that it's 10 tokens or something like that. But if you translate that sentence into, say, Korean or Japanese or something else, you'll typically see that the number of tokens used is much larger. And that's because the chunks here are a lot more broken up. Uh, so we're using a lot more tokens for the exact same thing. And what this does is it bloats up the sequence length of all the documents. So you're using up more tokens, and then in the attention of the transformer, when these tokens try to attend each other, you are running out of context, um, in the maximum context length of that transformer. And so basically, all the non-English text is stretched out from the perspective of the transformer, and this just has to do with the, um, training set used for the tokenizer and the tokenization itself. So it will create a lot bigger tokens and a lot larger groups in English, and it will have a lot of little boundaries for all the other non-English text. So if we translated this into English, it would be significantly fewer tokens.

[11:24] Andrej Karpathy: The final example I have here is a little snippet of Python for doing FizzBuzz. And what I'd like you to notice is, look, all these individual spaces are all separate tokens. They are token 220. So, uh, 220, 220, 220, 220, and then " if" is a single token. And so what's going on here is that when the transformer is going to consume or try to, uh, create this text, it needs to, um, handle all these spaces individually. They all feed in one by one into the entire transformer in the sequence. And so this is being extremely wasteful, tokenizing it in this way. And so, as a result of that, GPT-2 is not very good with Python. And it's not anything to do with coding or the language model itself, it's just that if you use a lot of indentation using space in Python, like you usually do, uh, you just end up bloating out all the text, and it's separated across way too much of the sequence, and we are running out of the context length in the sequence, uh, roughly speaking, what's what's happening. We're being way too wasteful. We're taking up way too much token space.

Improving Tokenization: GPT-2 vs. GPT-4

[12:29] Andrej Karpathy: Now, if we also scroll up here, we can change the tokenizer. So note here that GPT-2 tokenizer creates a token count of 300 for this string here. We can change it to cl100k_base, which is the GPT-4 tokenizer. And we see that the token count drops to 185. So for the exact same string, we are now roughly halving the number of tokens. And roughly speaking, this is because, uh, the number of tokens in the GPT-4 tokenizer is roughly double that of the number of tokens in the GPT-2 tokenizer. So we went from roughly 50k to roughly 100k.

[13:01] Andrej Karpathy: Now, you can imagine that this is a good thing because the same text is now squished into half as many tokens. So, uh, this is a lot denser input to the transformer. And in the transformer, every single token has a finite number of tokens before it that it's going to pay attention to. And so what this is doing is we're roughly able to see twice as much text as a context for what token to predict next, uh, because of this change. But of course, just increasing the number of tokens is, uh, not strictly better infinitely, uh, because as you increase the number of tokens, now your embedding table is, uh, sort of getting a lot larger. And also at the output, we are trying to predict the next token, and there's the softmax there, and that grows as well. We're going to go into more detail later on this, but there's some kind of a sweet spot somewhere where you have a just right number of tokens in your vocabulary where everything is appropriately dense and still fairly efficient.

[13:57] Andrej Karpathy: Now, one thing I would like you to note specifically for the GPT-4 tokenizer is that the handling of the whitespace for Python has improved a lot. You see that here, these four spaces are represented as one single token for the three spaces here, and then the token " if." And here, seven spaces were all grouped into a single token. So we're being a lot more efficient in how we represent Python. And this was a deliberate choice made by OpenAI when they designed the GPT-4 tokenizer. And they group a lot more whitespace into a single character. What this does is it densifies Python, and therefore, we can attend to more code before it when we're trying to predict the next token in the sequence. And so the improvement in the Python coding ability from GPT-2 to GPT-4 is not just a matter of the language model and the architecture and the details of the optimization, but a lot of the improvement here is also coming from the design of the tokenizer and how it groups characters into tokens.

The Tiktokenizer web app showing a Python FizzBuzz code snippet. The tokenizer is set to 'cl100k_base' (GPT-4). The indentation spaces are grouped into single, larger tokens, unlike the GPT-2 tokenizer.

From Strings to Integers: Unicode and Encodings

[14:56] Andrej Karpathy: Okay, so let's now start writing some code. So, remember what we want to do. We want to take strings and feed them into language models. For that, we need to somehow tokenize strings into some integers in some fixed vocabulary. And then we will use those integers to make a lookup into a lookup table of vectors and feed those vectors into the transformer as an input.

[15:20] Andrej Karpathy: Now, the reason this gets a little bit tricky, of course, is that we don't just want to support the simple English alphabet. We want to support different kinds of languages. So this is "annyeonghaseyo" in Korean, which is hello. And we also want to support many kinds of special characters that we might find on the internet, for example, emoji. So, how do we feed this text into, uh, transformers?

A Jupyter Notebook cell showing a Korean phrase '안녕하세요' (hello in Korean!) and an emoji '🤗'.

[15:43] Andrej Karpathy: Well, what is this text anyway in Python? So if you go to the documentation of a string in Python, you can see that strings are immutable sequences of Unicode code points. Okay, what are Unicode code points? We can go to Wikipedia. So Unicode code points are defined by the Unicode Consortium as part of the Unicode standard. And what this is really is that it's just a definition of roughly 150,000 characters right now. And roughly speaking, what they look like and what integers, um, represent those characters. So this is 150,000 characters across 161 scripts as of right now. So if you scroll down here, you can see that the standard is very much alive. The latest standard 15.1 is September 2023.

A screenshot of the Wikipedia page for 'Unicode'. It describes Unicode as a text encoding standard defining 149,813 characters and 161 scripts.

[16:31] Andrej Karpathy: And basically, this is just a way to define lots of types of characters, like for example, all these characters across different scripts. So, the way we can access the Unicode code point given a single character is by using the ord function in Python. So for example, I can pass in ord of 'h', and I can see that for the single character 'h', the Unicode code point is 104. Okay? Um, but this can be arbitrarily complicated. So we can take, for example, our emoji here, and we can see that the code point for this one is 128,000. Or we can take "an," and this is 50,000. Now, keep in mind, you can't plug in strings here because, uh, this doesn't have a single code point. It only takes a single Unicode code point character and tells you its integer.

A Jupyter Notebook showing the use of Python's ord() function to get the Unicode code point for the character 'h' (104), an emoji '🤗' (128075), and a Korean character '안' (50504).

[17:24] Andrej Karpathy: So in this way, we can look up all the, um, characters of this specific string and their code points. So ord(x) for x in this string, and we get this encoding here. Now, see here, we've already turned, the raw code points already have integers. So why can't we simply just use these integers and not have any tokenization at all? Why can't we just use this natively as is and just use the code point?

A Jupyter Notebook cell showing a list comprehension that applies the ord() function to each character in a string containing Korean text and an emoji, outputting a list of their integer Unicode code points.

[17:52] Andrej Karpathy: Well, one reason for that, of course, is that the vocabulary in that case would be quite long. So in this case, for Unicode, this is a vocabulary of 150,000 different code points. But more worryingly than that, I think, the Unicode standard is very much alive and it keeps changing. And so it's not kind of a stable representation necessarily that we may want to use directly. So for these reasons, we need something a bit better.

[18:16] Andrej Karpathy: So to find something better, we turn to encodings. So if you go to the Wikipedia page here, we see that the Unicode Consortium defines three types of encodings: UTF-8, UTF-16, and UTF-32. These encodings are the way by which we can take Unicode text and translate it into binary data or byte strings. UTF-8 is by far the most common. So this is the UTF-8 page. Now, this Wikipedia page is actually quite long, but what's important for our purposes is that UTF-8 takes every single code point and it translates it to a byte string. And this byte string is between one to four bytes. So it's a variable-length encoding. So depending on the Unicode point, according to the schema, you're going to end up with between one to four bytes for each code point.

A screenshot of the Wikipedia page for UTF-8, showing a table that maps Unicode code point ranges to their corresponding byte-length representation in UTF-8 (1, 2, 3, or 4 bytes).

[19:02] Andrej Karpathy: On top of that, there's UTF-8, uh, UTF-16, and UTF-32. UTF-32 is nice because it is fixed length instead of variable length, but it has many other downsides as well. So the full kind of spectrum of pros and cons of all these different three encodings are beyond the scope of this video. I'd just like to point out that I enjoyed this blog post, and this blog post at the end of it also has a number of references that can be quite useful. Uh, one of them is "UTF-8 Everywhere Manifesto." Um, and this manifesto describes the reason why UTF-8 is significantly preferred and a lot nicer than the other encodings and why it is used a lot more prominently, um, on the internet. One of the major advantages that's just to give you a sense is that UTF-8 is the only one of these that is backward compatible to the much simpler ASCII encoding of text. Um, but I'm not going to go into the full detail in this video. So suffice to say,

Introduction to UTF-8 Encoding

[20:00] Speaker A: Suffice it to say that we like the UTF-8 encoding. And, uh, let's try to take this string and see what we get if we encode it into UTF-8.

Jupyter Notebook cell showing a Python list comprehension to get the Unicode code points for a string containing Korean and English text.

[20:08] Speaker A: The string class in Python actually has .encode, and you can give it the encoding, which is, let's say, UTF-8. Now, what we get out of this is not very nice because this is the bytes, this is a bytes object, and it's not very nice in the way that it's printed. So I personally like to take it through a list because then we actually get the raw bytes of this, uh, encoding.

[20:31] Speaker A: So this is the raw bytes that represent this string according to the UTF-8 encoding.

Comparing UTF-8, UTF-16, and UTF-32

[20:37] Speaker A: We can also look at UTF-16. We get a slightly different byte stream. And here we start to see one of the disadvantages of UTF-16. You see how we have zero, zero something, zero something, zero something. We're starting to get a sense that this is a bit of a wasteful encoding. And indeed, for simple ASCII characters or English characters here, uh, we just have this structure of zero something, zero something, and it's not exactly nice.

Jupyter Notebook cell showing the UTF-16 encoding of the string, resulting in a long list of integers with many zeros, indicating inefficiency.

[21:02] Speaker A: Same for UTF-32. When we expand this, we can start to get a sense of the wastefulness of this encoding for our purposes. You see a lot of zeros followed by something. And so, uh, this is not desirable.