Transcript of But what is a neural network? | Deep learning chapter 1
3Blue1Brown
0:04This is a 3.0:06It's sloppily written and rendered at an extremely low resolution of 28x28 pixels,0:10but your brain has no trouble recognizing it as a 3.0:14And I want you to take a moment to appreciate how0:16crazy it is that brains can do this so effortlessly.0:19I mean, this, this and this are also recognizable as 3s,0:22even though the specific values of each pixel is very different from one0:27image to the next.0:28The particular light-sensitive cells in your eye that are firing when you0:32see this 3 are very different from the ones firing when you see this 3.0:37But something in that crazy-smart visual cortex of yours resolves these as representing0:42the same idea, while at the same time recognizing other images as their own distinct0:47ideas.0:49But if I told you, hey, sit down and write for me a program that takes in a grid of0:5428x28 pixels like this and outputs a single number between 0 and 10,0:59telling you what it thinks the digit is, well the task goes from comically trivial to1:04dauntingly difficult.1:07Unless you've been living under a rock, I think I hardly need to motivate the relevance1:10and importance of machine learning and neural networks to the present and to the future.1:15But what I want to do here is show you what a neural network actually is,1:18assuming no background, and to help visualize what it's doing,1:22not as a buzzword but as a piece of math.1:25My hope is that you come away feeling like the structure itself is motivated,1:28and to feel like you know what it means when you read,1:31or you hear about a neural network quote-unquote learning.1:35This video is just going to be devoted to the structure component of that,1:38and the following one is going to tackle learning.1:40What we're going to do is put together a neural1:43network that can learn to recognize handwritten digits.1:49This is a somewhat classic example for introducing the topic,1:52and I'm happy to stick with the status quo here,1:54because at the end of the two videos I want to point you to a couple good1:57resources where you can learn more, and where you can download the code that2:00does this and play with it on your own computer.2:05There are many many variants of neural networks,2:07and in recent years there's been sort of a boom in research towards these variants,2:12but in these two introductory videos you and I are just going to look at the simplest2:16plain vanilla form with no added frills.2:19This is kind of a necessary prerequisite for understanding any of the more powerful2:23modern variants, and trust me it still has plenty of complexity for us to wrap our minds2:28around.2:29But even in this simplest form it can learn to recognize handwritten digits,2:33which is a pretty cool thing for a computer to be able to do.2:37And at the same time you'll see how it does fall2:39short of a couple hopes that we might have for it.2:43As the name suggests neural networks are inspired by the brain, but let's break that down.2:48What are the neurons, and in what sense are they linked together?2:52Right now when I say neuron all I want you to think about is a thing that holds a number,2:58specifically a number between 0 and 1.3:00It's really not more than that.3:03For example the network starts with a bunch of neurons corresponding to3:08each of the 28x28 pixels of the input image, which is 784 neurons in total.3:14Each one of these holds a number that represents the grayscale value of the3:19corresponding pixel, ranging from 0 for black pixels up to 1 for white pixels.3:25This number inside the neuron is called its activation,3:28and the image you might have in mind here is that each neuron is lit up when its3:32activation is a high number.3:36So all of these 784 neurons make up the first layer of our network.3:46Now jumping over to the last layer, this has 10 neurons,3:49each representing one of the digits.3:52The activation in these neurons, again some number that's between 0 and 1,3:56represents how much the system thinks that a given image corresponds with a given digit.4:03There's also a couple layers in between called the hidden layers,4:06which for the time being should just be a giant question mark for4:09how on earth this process of recognizing digits is going to be handled.4:14In this network I chose two hidden layers, each one with 16 neurons,4:17and admittedly that's kind of an arbitrary choice.4:21To be honest I chose two layers based on how I want to motivate the structure in4:24just a moment, and 16, well that was just a nice number to fit on the screen.4:28In practice there is a lot of room for experiment with a specific structure here.4:33The way the network operates, activations in one4:35layer determine the activations of the next layer.4:39And of course the heart of the network as an information processing mechanism comes down4:43to exactly how those activations from one layer bring about activations in the next4:48layer.4:49It's meant to be loosely analogous to how in biological networks of neurons,4:53some groups of neurons firing cause certain others to fire.4:58Now the network I'm showing here has already been trained to recognize digits,5:01and let me show you what I mean by that.5:03It means if you feed in an image, lighting up all 784 neurons of the input layer5:08according to the brightness of each pixel in the image,5:11that pattern of activations causes some very specific pattern in the next layer5:16which causes some pattern in the one after it,5:18which finally gives some pattern in the output layer.5:22And the brightest neuron of that output layer is the network's choice,5:26so to speak, for what digit this image represents.5:32And before jumping into the math for how one layer influences the next,5:36or how training works, let's just talk about why it's even reasonable5:40to expect a layered structure like this to behave intelligently.5:44What are we expecting here?5:45What is the best hope for what those middle layers might be doing?5:48Well, when you or I recognize digits, we piece together various components.5:54A 9 has a loop up top and a line on the right.5:57An 8 also has a loop up top, but it's paired with another loop down low.6:01A 4 basically breaks down into three specific lines, and things like that.6:07Now in a perfect world, we might hope that each neuron in the second6:11to last layer corresponds with one of these subcomponents,6:14that anytime you feed in an image with, say, a loop up top,6:18like a 9 or an 8, there's some specific neuron whose activation is6:22going to be close to 1.6:24And I don't mean this specific loop of pixels,6:26the hope would be that any generally loopy pattern towards the top sets off this neuron.6:32That way, going from the third layer to the last one just requires6:36learning which combination of subcomponents corresponds to which digits.6:41Of course, that just kicks the problem down the road,6:43because how would you recognize these subcomponents,6:45or even learn what the right subcomponents should be?6:48And I still haven't even talked about how one layer influences the next,6:51but run with me on this one for a moment.6:53Recognizing a loop can also break down into subproblems.6:57One reasonable way to do this would be to first6:59recognize the various little edges that make it up.7:03Similarly, a long line, like the kind you might see in the digits 1 or 4 or 7,7:08is really just a long edge, or maybe you think of it as a certain pattern of several7:13smaller edges.7:15So maybe our hope is that each neuron in the second layer of7:18the network corresponds with the various relevant little edges.7:23Maybe when an image like this one comes in, it lights up all of the7:27neurons associated with around 8 to 10 specific little edges,7:31which in turn lights up the neurons associated with the upper loop7:35and a long vertical line, and those light up the neuron associated with a 9.7:40Whether or not this is what our final network actually does is another question,7:44one that I'll come back to once we see how to train the network,7:47but this is a hope that we might have, a sort of goal with the layered structure7:51like this.7:53Moreover, you can imagine how being able to detect edges and patterns7:56like this would be really useful for other image recognition tasks.8:00And even beyond image recognition, there are all sorts of intelligent8:04things you might want to do that break down into layers of abstraction.8:08Parsing speech, for example, involves taking raw audio and picking out distinct sounds,8:12which combine to make certain syllables, which combine to form words,8:16which combine to make up phrases and more abstract thoughts, etc.8:21But getting back to how any of this actually works,8:23picture yourself right now designing how exactly the activations in one layer might8:27determine the activations in the next.8:30The goal is to have some mechanism that could conceivably combine pixels into edges,8:35or edges into patterns, or patterns into digits.8:39And to zoom in on one very specific example, let's say the hope8:43is for one particular neuron in the second layer to pick up8:46on whether or not the image has an edge in this region here.8:51The question at hand is what parameters should the network have?8:55What dials and knobs should you be able to tweak so that it's expressive8:59enough to potentially capture this pattern, or any other pixel pattern,9:03or the pattern that several edges can make a loop, and other such things?9:08Well, what we'll do is assign a weight to each one of the9:11connections between our neuron and the neurons from the first layer.9:16These weights are just numbers.9:18Then take all of those activations from the first layer9:21and compute their weighted sum according to these weights.9:27I find it helpful to think of these weights as being organized into a9:31little grid of their own, and I'm going to use green pixels to indicate9:34positive weights, and red pixels to indicate negative weights,9:37where the brightness of that pixel is some loose depiction of the weight's value.9:42Now if we made the weights associated with almost all of the pixels zero9:46except for some positive weights in this region that we care about,9:50then taking the weighted sum of all the pixel values really just amounts9:53to adding up the values of the pixel just in the region that we care about.9:59And if you really wanted to pick up on whether there's an edge here,10:02what you might do is have some negative weights associated with the surrounding pixels.10:07Then the sum is largest when those middle pixels10:10are bright but the surrounding pixels are darker.10:14When you compute a weighted sum like this, you might come out with any number,10:18but for this network what we want is for activations to be some value between 0 and 1.10:24So a common thing to do is to pump this weighted sum into some function10:28that squishes the real number line into the range between 0 and 1.10:32And a common function that does this is called the sigmoid function,10:35also known as a logistic curve.10:38Basically very negative inputs end up close to 0, positive inputs end up close to 1,10:43and it just steadily increases around the input 0.10:49So the activation of the neuron here is basically a10:52measure of how positive the relevant weighted sum is.10:57But maybe it's not that you want the neuron to10:59light up when the weighted sum is bigger than 0.11:02Maybe you only want it to be active when the sum is bigger than say 10.11:06That is, you want some bias for it to be inactive.11:11What we'll do then is just add in some other number like negative 10 to this11:15weighted sum before plugging it through the sigmoid squishification function.11:20That additional number is called the bias.11:23So the weights tell you what pixel pattern this neuron in the second11:27layer is picking up on, and the bias tells you how high the weighted11:31sum needs to be before the neuron starts getting meaningfully active.11:36And that is just one neuron.11:38Every other neuron in this layer is going to be connected to11:42all 784 pixel neurons from the first layer, and each one of11:46those 784 connections has its own weight associated with it.11:51Also, each one has some bias, some other number that you add11:54on to the weighted sum before squishing it with the sigmoid.11:58And that's a lot to think about!11:59With this hidden layer of 16 neurons, that's a total of 784 times 16 weights,12:06along with 16 biases.12:08And all of that is just the connections from the first layer to the second.12:12The connections between the other layers also have12:14a bunch of weights and biases associated with them.12:18All said and done, this network has almost exactly 13,000 total weights and biases.12:2313,000 knobs and dials that can be tweaked and turned12:27to make this network behave in different ways.12:31So when we talk about learning, what that's referring to is12:34getting the computer to find a valid setting for all of these12:37many many numbers so that it'll actually solve the problem at hand.12:42One thought experiment that is at once fun and kind of horrifying is to imagine sitting12:47down and setting all of these weights and biases by hand,12:50purposefully tweaking the numbers so that the second layer picks up on edges,12:54the third layer picks up on patterns, etc.12:56I personally find this satisfying rather than just treating the network as a total black13:01box, because when the network doesn't perform the way you anticipate,13:04if you've built up a little bit of a relationship with what those weights and biases13:09actually mean, you have a starting place for experimenting with how to change the13:13structure to improve.13:14Or when the network does work but not for the reasons you might expect,13:18digging into what the weights and biases are doing is a good way to challenge13:22your assumptions and really expose the full space of possible solutions.13:26By the way, the actual function here is a little cumbersome to write down,13:29don't you think?13:32So let me show you a more notationally compact way that these connections are represented.13:37This is how you'd see it if you choose to read up more about neural networks.13:40Organize all of the activations from one layer into a column as a vector.13:48Then organize all of the weights as a matrix, where each row of that matrix corresponds13:50to the connections between one layer and a particular neuron in the next layer.13:58What that means is that taking the weighted sum of the activations in14:02the first layer according to these weights corresponds to one of the14:05terms in the matrix vector product of everything we have on the left here.14:14By the way, so much of machine learning just comes down to having a good14:17grasp of linear algebra, so for any of you who want a nice visual14:21understanding for matrices and what matrix vector multiplication means,14:24take a look at the series I did on linear algebra, especially chapter 3.14:29Back to our expression, instead of talking about adding the bias to each one of14:33these values independently, we represent it by organizing all those biases into14:38a vector, and adding the entire vector to the previous matrix vector product.14:43Then as a final step, I'll wrap a sigmoid around the outside here,14:46and what that's supposed to represent is that you're going to apply the14:50sigmoid function to each specific component of the resulting vector inside.14:55So once you write down this weight matrix and these vectors as their own symbols,15:00you can communicate the full transition of activations from one layer to the next in an15:05extremely tight and neat little expression, and this makes the relevant code both a lot15:10simpler and a lot faster, since many libraries optimize the heck out of matrix15:14multiplication.15:17Remember how earlier I said these neurons are simply things that hold numbers?15:22Well of course the specific numbers that they hold depends on the image you feed in,15:27so it's actually more accurate to think of each neuron as a function,15:31one that takes in the outputs of all the neurons in the previous layer and spits out a15:36number between 0 and 1.15:39Really the entire network is just a function, one that takes in15:43784 numbers as an input and spits out 10 numbers as an output.15:47It's an absurdly complicated function, one that involves 13,000 parameters15:51in the forms of these weights and biases that pick up on certain patterns,15:55and which involves iterating many matrix vector products and the sigmoid15:59squishification function, but it's just a function nonetheless.16:03And in a way it's kind of reassuring that it looks complicated.16:07I mean if it were any simpler, what hope would we have16:09that it could take on the challenge of recognizing digits?16:13And how does it take on that challenge?16:15How does this network learn the appropriate weights and biases just by looking at data?16:20Well that's what I'll show in the next video, and I'll also dig a little16:23more into what this particular network we're seeing is really doing.16:27Now is the point I suppose I should say subscribe to stay notified16:30about when that video or any new videos come out,16:33but realistically most of you don't actually receive notifications from YouTube, do you?16:38Maybe more honestly I should say subscribe so that the neural networks16:41that underlie YouTube's recommendation algorithm are primed to believe16:44that you want to see content from this channel get recommended to you.16:48Anyway, stay posted for more.16:50Thank you very much to everyone supporting these videos on Patreon.16:54I've been a little slow to progress in the probability series this summer,16:57but I'm jumping back into it after this project,16:59so patrons you can look out for updates there.17:03To close things off here I have with me Lisha Li who did her PhD work on the17:07theoretical side of deep learning and who currently works at a venture capital17:10firm called Amplify Partners who kindly provided some of the funding for this video.17:15So Lisha one thing I think we should quickly bring up is this sigmoid function.17:19As I understand it early networks use this to squish the relevant weighted17:23sum into that interval between zero and one, you know kind of motivated17:26by this biological analogy of neurons either being inactive or active.17:30Exactly.
But relatively few modern networks actually use sigmoid anymore.17:34Yeah.
It's kind of old school right?17:35Yeah or rather ReLU seems to be much easier to train.17:39And ReLU, ReLU stands for rectified linear unit?17:42Yes it's this kind of function where you're just taking a max of zero17:47and a where a is given by what you were explaining in the video and17:52what this was sort of motivated from I think was a partially by a17:56biological analogy with how neurons would either be activated or not.18:01And so if it passes a certain threshold it would be the identity function but if it did18:06not then it would just not be activated so it'd be zero so it's kind of a simplification.18:11Using sigmoids didn't help training or it was very difficult to18:15train at some point and people just tried ReLU and it happened18:20to work very well for these incredibly deep neural networks.18:25All right thank you Lisha.
3,357 words · 286 lines
