Transcript of "LLM 만들어 봤어요?" - 파트3 인트로
홍정모
0:01Recently, a student mentioned being0:03 asked during a job interview if they0:05 had ever built an LLM. They said they0:08 were quite flustered because it wasn't0:10 a particularly large company, nor was0:12 it a firm that developed AI. One could0:15 say that interest in AI is so intense0:17 that such a question is asked in an0:19 interview. However, on the other hand,0:21 it makes one wonder if hiring standards0:24 have become excessively high. But there0:28 is no need to worry too much. Thanks to0:32 AI, building an LLM has become much0:35 easier. From now on, you will no longer0:37 be afraid of such questions; instead,0:39 you will feel a sense of welcome. GPT20:41 is commonly regarded as the prototype0:44 of modern LLMs. In Part 3, we will0:47 study LLMs in greater depth by building0:50 one ourselves. Even modern LLMs, which0:54 demonstrate amazing performance, are0:56 actually rooted in a very simple idea.0:59 It involved probabilistically1:01 predicting the next word based on the1:04 preceding words. It was implemented as1:07 a statistical model nearly 30 years1:10 after Claude Schenert's idea in the1:12 1940s. It was a method of creating a1:15 table by counting word combinations one1:17 by one. The Anggram we built ourselves1:21 in Part 1 was exactly this. However,1:23 this method had a fundamental1:25 limitation. No matter how fast1:27 computers and GPUs become, it is1:29 impossible to count every single word1:31 and phrase that exists in the world. We1:34 had fallen victim to the famous curse1:36 of dimensionality. Here, a remarkable1:38 shift in perspective created a1:40 breakthrough. Instead of stacking words1:43 to fill the table, we taught the neural1:46 network to finish the next word. This1:48 is the Neural Probabilistic Language1:51 Model (ML) that we studied in Part 2.1:53 The word bedding that naturally occurs1:56 during the neural network's training1:58 process broke the curse of2:00 dimensionality and opened the first2:02 door leading to the current LM. And2:05 finally, in 2017, the famous2:07 Transformer model appeared. Transformer2:11 means a converter—that is, a device2:14 that changes something. What does it2:17 change? It is the word bedding we saw2:20 in the Neural Language Model. It2:22 changes it into a more appropriate2:24 expression depending on the context at2:26 hand. In a way, the idea itself is2:29 simple. However, this served as the2:32 fuse, and from that point on, the2:34 technology we commonly call AI began to2:36 develop at a truly terrifying speed.2:41 Then, in 2019, just two years after the2:44 Transformer paper was published, OpenAI2:47 released GPT2. Here, the 'T' in GPT2:50 stands for Transformer. A single piece2:53 of writing, born from the regrets of2:55 Transformer, turned the world upside2:58 down. The famous story of the unicorn3:01 begins right here. Not only those who3:04 read the paper, but also those who3:06 later actually tested it, were left3:09 speechless. This was because no one had3:12 anticipated that AI would be able to3:15 write this well. There were even3:17 experimental results showing that3:20 people believed it was real. Because it3:22 was so convincing, praise quickly3:24 turned into concern. What if this is3:27 used to endlessly churn out fake news?3:30 Eventually, OpenAI postponed the3:32 release of the large model, deeming it3:34 too dangerous. The narrative of "too3:36 dangerous artificial intelligence"3:38 began right here. However, looking back3:41, the reality was different. In just3:44 six months, two graduate students3:45 replicated the exact same model using3:47 only the publicly available information3:49. OpenAI did not open up its training3:52 code or data, you know. While it is3:56 true that GPT 2’s performance was3:58 outstanding by the standards of that4:01 time, fortunately, looking back now, no4:04 instances of abuse at the level feared4:06 have been reported. So, let’s take a4:09 look at just how impressive its4:12 capabilities were. The most intriguing4:15 demo at the time was an article about4:17 discovering a unicorn. We often refer4:20 to very fast-growing startups as4:23 unicorns; the name of this non-existent4:26 animal carries the meaning that they4:28 are extremely rare and hard to find.4:32 However, the research team conducted an4:34 experiment with GPT2 as if someone had4:37 actually discovered a unicorn. They had4:40 it read a prompt stating that a herd of4:42 unicorns had been found in the Andes4:44 Mountains and that they were even4:45 speaking English. Upon hearing this,4:49 the GPT2 model followed up without4:51 hesitation and wrote the article. "An4:53 unknown being. The mystery has finally4:57 been solved." It unfolds naturally,4:59 just like a science news report written5:02 by a seasoned professional journalist.5:04 Here, we can easily spot the signature5:07 of a Transformer. It independently5:10 identifies and utilizes keywords like5:12 the Andes Mountains from the context5:15 provided by a human. It also5:17 consistently repeats the name of Dr.5:20 Perez, a fictional character it created5:23, throughout the code. This means that5:26 when generating the next word, it5:28 deeply examines the given context and5:30 utilizes it appropriately, which5:33 significantly improves the quality of5:35 generation. This is called "Attention."5:39 The paper you are currently viewing is5:41 the very paper that first introduced5:43 Transformers to the world. The title5:45 suggests that nothing else is needed5:46 except attention. In a way, it even5:49 appears provocative. It signifies a5:53 structure built solely on attention,5:55 stripping away the complex methods of5:58 the past. Now, let me introduce the six6:02 highlights we will study in Part 3.6:06 First is generating actual results at6:09 the level of GPT2. We will directly6:12 train a model with 800 million6:14 parameters (0.8 B) and properly study6:17 how the Transformer structure is6:19 reflected in the generated results. The6:22 second is, of course, attention. There6:25 was a time when simply being able to6:26 implement a Transformer model using6:27 PyTorch was enough to be recognized as6:31 a skill. Now, however, we are in an era6:34 where AI performs simple coding much6:36 better. Rather than getting bogged down6:39 in complex details, we aim to6:40 understand the fundamental mechanisms6:43 as easily and intuitively as possible.6:45 For example, because Korean and English6:48 have different word orders, words are6:50 often connected in a staggered manner.6:52 We will see firsthand how Attention6:55 unravels this. The third topic is6:58 dynamic inbedding. Even the same word7:00 can have significantly different7:02 meanings depending on the context. If7:04 you understood word inbedding in Part 27:07, this is a very interesting moment7:10 where you can instantly realize the7:13 origin of the name "Transformer."7:16 Induction Head is the part I find the7:18 most interesting. We will explore the7:22 secrets behind how Antropic, the AI7:25 company you are all familiar with,7:28 analyzes LMs. Fifth is "Blocking." This7:33 refers to a sudden realization. It7:35 means that if you keep studying7:37 persistently even when you don't7:39 understand something, your skills will7:41 suddenly skyrocket at some point. You7:43 can witness that moment of realization7:46 firsthand in Part 3. Finally, there is7:49 the Gcloud usage guide. In the past, I7:51 used to get a headache over commands to7:54 send to the cloud and lose sleep7:56 monitoring whether training was going7:58 well. Now, however, we can entrust all8:01 complex and tedious tasks to AI. This8:05 series was designed as an AI-native8:07 course where we build AI using AI. Even8:10 as I developed this course, AI8:12 technology advanced rapidly, ushering8:15 in an era where we can study much more8:18 easily yet deeply. From environment8:21 setup to implementation, training8:23 monitoring, and experimental result8:25 analysis, most of the barriers that8:28 once made it difficult to build LLMs8:30 have crumbled. Does this mean human8:32 work has completely disappeared? Is it8:35 really enough to just issue commands8:37 with determination? Unfortunately, that8:39 is not the case. Even AI gets confused8:42 by cutting-edge technologies like8:44 building LLMs. Just as great power8:46 comes with great responsibility, a8:48 moment of carelessness could lead to8:50 recklessly wasting expensive GPU usage8:52 fees. You must clearly understand the8:55 big picture and hold onto it firmly. It8:58 is no exaggeration to say that if you9:01 possess the skill and confidence to9:03 steer the direction you desire at any9:06 time without being swayed by AI, AI can9:09 amplify your capabilities tenfold or9:11 even a hundredfold. We are living in an9:17 era where a small idea—finishing the9:19 next word—can grow into a massive9:22 current known as the AI revolution. I9:25 believe this lecture will serve as a9:27 journey for you to find your own9:29 unicorn. Now, let's begin the9:32 preliminary study with "Building from9:34 Scratch" Part 3: Transformers.
1,378 words · 222 lines
![[선데이 라이브] 당신도 AI 시대에 살아남을 수 있다!](https://i.ytimg.com/vi/D95UC9U5ShA/hqdefault.jpg)
