Deep Learning Full Course 2026 [FREE] | Complete Deep Learning Tutorial For Beginners | Simplilearn

Key Takeaways

[music] Hey everyone, welcome to this deep learning course by SimplyLearn. Today, deep learning is powering some of the most exciting technologies around us from face recognition and self-driving cars to voice assistance, image generation, medical diagnosis, recommendation systems, and modern AI chatbots. But deep learning is not just about big models or complex algorithms. At its core, it's about building neural networks that can learn from data and improve its training and make intelligent predictions. And that is exactly what we'll be learning in this course. We will start by understanding what deep learning is, how it differs from traditional machine learning, and why neural networks are so important in today's AI world. Then we'll be breaking down the core building blocks of neural networks which is neuron layers, weights, biases, activation functions and how data flows from input to output. We'll also explore the perception activation functions like sigmoid relu and softmax and how neural networks are trained using forward propagation loss functions backward propagation gradient descents and weight updates. After that, we'll move into hands-on modeling building with TensorFlow, Keras and also PyTorch where you will learn how to create, train, and test deep learning models step by step. We'll also apply deep learning to image classification using data site like MNIST and fashion and understand how models recognize digits and clothing items from images. Along the way, we'll also cover optimization techniques like SG momentum, adaptive learning rate and also Adam optimizer along with methods like dropout and batch normalization to reduce. By the end of this course, you'll understand how deep learning modes are best trade and optimized for real world task. So get started and understand how neural networks actually if you're interested in mastering the future of technology professional certificate course in generative AI and machine learning is a perfect opportunity for you can't put this even provides hands on expertise in cutting edge like geni machine learning and tools such as classes learning a prestigious produces for support Are you find out or is it your answer below? >> Start with our introduction. So I'm hoping to come here are just the background deep learning. So some of the um achievements that really happened in recent times um we're talking you know less than 10 years old and some of these achievements that are uh have brought deep learning to the forefront and really popular uh um field. Um so we'll understand some of the applications and also some of the challenges of deep learning. Deep is not perfect. It has some drawbacks and some challenges that make it um difficult at times. So we'll talk about that. Um and then we'll talk about some of the frameworks. Kind of already mentioned uh PyTorch and TensorFlow as the two main ones. Just talk about that a little bit. Um and then talk about what are the main steps of any deep learning project that uh we would want to follow. I think what you're going to see there is they mostly mimic the same kind of steps we were doing in machine learning things like uh data preparation um then kind of building a model training it um evaluating it on test data those kind of steps are mostly the same it's just the details of those are going to be drastically different you know that training process is going to be more involved I think with with deep learning um building the model is a little more hands-on with deep learning as well because you're literally going to be architecting and constructing a neural network. It's much different than just instantiating a um a model from scikitlearn which is pretty easy to do. Um there's going to be more hands-on work involved in in building these deep learning uh networks that are going to be our models. Um so we'll talk about that process. Okay. All right. Let me skip that for the moment. All right. So let's start with just AI in general. So the reason to start here is just to see where deep learning kind of fits into the rest of the stuff we've so far such as machine and uh where has fit in um where we have AI as this pretty broad umbrella um that is um basically involving machines, computers um trying to perform tasks that normally would have to do. Um and what we just got finished was machine learning, right? So machine learning is primarily having the machine learn from data how to make a decision, right? How to complete some task whether that's prediction or maybe like an unsupervised case um doing things like compression, dimensionality reduction or finding patterns and clusters. It's learning from data how to do that, right? That's machine learning broadly. um where deep learning fits in is really a subset of machine learning because we're still learning from data. Uh we still are doing that. It's just we're using a specific tool to do that which is going to be a neural network. Um so that's why in this diagram you see deep learning as kind of a circle embedded inside of the machine learning circle because it's it is kind of a sub uh field of machine learning. you're still learning from data. You're just doing it in a very particular way. Um, so it's kind of like a specialization. That's why I said earlier not everybody studies deep learning because you can get pretty far without deep learning. You still have this whole field of machine learning. And you can you can still do a lot with that. Um but this is certainly kind of a specialization is drilling down into working with neural nets in particular as your model. Um it turns out that working with neural nets is really powerful for many reasons. Um and what we're going to see is that deep learning really unlocks uh working with specific types of data that we have not worked with so far. And you know we already have a sense of that things like images, things like text, we really haven't worked with too much or at all. Um but deep learning really unlocks the ability to do that. Um so you can see that here. All right. So so this is this is where so we're studying deep learning. We're going to be in this course. This is where it kind of fits in. It's just a subset of machine learning because at the end of the day, we're still learning from data. It's just going to be using a particular type of model which is a neural net. And more so than just a basic neural net, it's often going to be what we will um term as a deep neural net. Meaning it will be a pretty complex model that has uh a lot of layers to it um as we will see. But it is just a subset of machine learning. Okay. So, historically, how has this evolved to get to that point? Um, well, we had originally the the very first model of a neuron was proposed in 1943 um by McCulla and Pitts. So, they produced a paper kind of um proposing a potential model of the of the brain and the neurons in the brain uh in 1943. So, obviously a long time ago. Um and at the same time some other things going on. We had the touring test which you may have heard of basically um testing the intelligent ability of a machine. Um and so we had that going on as well. Obviously other things going on from the 40s to 50, but the big one that's relevant to deep learning is the very first proposed model of the neuron that was as old as 1943. It was more of an idea then. Obviously, they didn't have computing power to do anything with that. Um, it was more just the idea, the concept was kind of as old as that of just modeling an actual neuron. Um, it wasn't until the 50s that we see the actual um term artificial intelligence being adopted. um as well as the very first very basic neur uh neural network model the perceptron uh that was invented in 1957. At the same time we have some programming languages being invented uh to take advantage of machines um not Python yet obviously but um you know some older languages that end up inspiring the uh generation of Python later on. But still some some uh high level languages uh that are invented. But this perceptron is really relevant. Actually we're going to study perceptron in the next lesson as kind of the first um [clears throat] model the first basic neural network model that uh um will propel you know uh deeper and deeper networks uh built off of the idea of kind of a perceptron. So we'll talk about perceptron quite a bit. Are perceptrons still being used today? Um perceptron itself is probably not used too much but the idea of the perceptron is very present in so many network. It think of it like the building block of any neural network um was founded on the perceptron idea. So the the perceptron itself like just a single neuron. So, so we're going to learn about this, but the perceptron is basically the model of a single neuron. Um, a single neuron not quite used too much. Uh, it's very basic, but it forms the building block for nearly all neural networks uh that are around today. So, I would say yeah, I would say it's it's inspired pretty much any neural network is inspired by the perceptron. So it's a very very important uh idea and concept and we're going to spend some time talking about it and studying it uh as kind of our foundation for neural nets and then we'll build on that by kind of stacking uh stacking multiple neural multiple perceptrons essentially together uh till we get a larger and larger network. Yeah. So the thing is the idea has been around a long time uh but there hasn't been the you know computing power to really take advantage of it until recently and when I say recently probably last 15 20 years um hasn't been right there hasn't been the computing power available to really unlock the capabilities of these and make make uh larger and larger networks that can solve harder and harder problems. So we don't see that uh we don't see that until you know recent first job was a key punch operator. I think it's the uh don't know what key punch is. I think it's the um like the binary uh the bit uh cards that [snorts] uh fed the original kind of computers or older computers and you had to literally punch out the also this time in the '60s was the original chatbot Eliza um 1966. Now it's nothing like generative AI right this is this chatbot is all rules driven so nothing datadriven nothing like that certainly no neural net nothing that we see today with these transformerbased models and GPTs and such but it is a chatbot in the sense of you can input text and then it can respond with text basically it's all rule driven though so it had to be pre-programmed uh how to respond to things but that was you know the first kind of AI chatbot Okay. So then we see kind of a uh lull in the uh 70s to 70 to 80. Um interest in AI dropped significantly. Um funding from from government sources was insufficient. So there's kind of a lull period here. Now that's not to say that um there wasn't anything going on. you know, there's still statistical research going on and and advancements in the machine learning community, but nothing really relevant to um neural networks uh at this period. Um but still, you know, some of the some of the developments in the in the statistics community are still relevant during this time uh to machine learning, just not necessarily deep learning. But there's a rebound of that in the 80s. Um, and you may be able to guess why. Maybe due to geopolitical things going on in the 80s like the Cold War. Um, we see a revival in uh research and dedication to AI. Um, we also start to see some communities uh develop um that are uh dedicated to the advancement of AI. So the Stanford hosts the inaugural uh conference the AAI um which is kind of the uh inspiration for some of the conferences we see today like nuripss and and things like that. So in the 80s there's kind of a revival um and then there's a kind of a um interesting period of time in the late uh late 90s. So kind of early I would say even early 90s to early 2000s in this period um we see a significant investment in AI from the big players like like Facebooks and Netflixes of the world and Google of course Twitter um they start investing in AI quite a bit um 1997 we have the IBM deep blue if you've ever heard of that um beat the uh world chess champion Um so that was a really cool kind of AI achievement but this is this period's called out here in particular because AI is particular deep learning had not advanced that far even by this point. However there are some significant investments in this period that lead us to kind of where we are today. Um I would say two of them that are not necessarily listed here but will be coming up are where it says companies like Facebook, Twitter and Netflix and Google not mentioned but should be there. They really start investing in AI. I want to call out Facebook and Google in particular because they build the frameworks that we use today. So Facebook invented PyTorch around this time and Google invented TensorFlow around this time. They did those independently. Now they are mostly open sourced so anybody can use them but they were in-house tools to work with neural nets. Everybody wanted to work with neural nets. They had their own in-house tools uh like PyTorch and TensorFlow to kind of build and manage neural nets at the time. And this all started right around this period 2006 or so. Um you know not that long ago relative to uh history only about 20 years. Um not that long ago and we see kind of those foundational uh libraries being developed and obviously there's been a lot of advancements since then. So in uh the period after that um this is kind of where the deep learning golden age is like really the 2010s onward um 2012 is called out here. We're going to talk about there's a really big event in 2012 that happens which is the Alex net uh convolutional net is is uh is published and studied. It's a really big computer vision model. Um uh so um a lot of really amazing achievements from from this period really 2010s onward in deep learning and it kind of coincides with um hardware being more available um neural nets being more readily available with those frameworks being more mature and developed like PyTorch and TensorFlow um more people have access to them. They're open sourced so more people can work with them and and really get their hands on it including the research community. So there's a reason it kind of blows up in the 2010s onward and even even today. Uh it's so I'll tell you why. It's because um a lot of generative AI is powered by the transformer and the transformer was not developed really until 2017. So that's less than 10 years ago. Um, but the really the computing power and data availability to work with something like a transformer has not caught up until really like the last five or so years. So that's why it's so recent. The model that they're all built off of is less than 10 years old. Um, the transformer um, but it's more like just other things like hardware and data availability. We've caught up to that. uh um and and now we can have these really massive language models that power generative AI uh because compute power is relatively easy to to manage and um data is pretty easy to get your hands on too but even but that's what I'm saying the model is not that old that powers it um so what are some of our motivations for deep learning um so there's a number of topics that we're going to study. You know, they're all listed here. We're going to start with the perceptron. Um we're going to to talk about these topics later on that have to do with like uh the back propagation really having to deal with how to train a neural net. Um so we'll talk about how how they get trained. We'll talk about the earliest model of a neuron which is going to be the perceptron. So we'll certainly cover that. Um then we can stack perceptrons together in what what is known as a multi-layer perceptron. That's kind of the next level. Um that's kind of like a miniature deep neural net. It's it's not very deep, but it does have multiple layers to it. Um and then we have the kind of in more recent times these very advanced networks like these that we're going to study. um convolutions mainly being used for images and recurrent neural networks mainly being used for text. So we're talking about the motivation one of the motivations one of the primary motivations for talking about deep learning is really this this idea that um deep learning is going to unlock our ability to work with text data and our ability to work with image data. There are so many problems out there that work with this kind of data which in the community is known as unstructured data. Right? So, so far we've really only done problems with structured data and structured meaning that it's basically a matrix that has rows and columns with really well- definfined features, right? That are rows and columns like our like our housing data that has square footage and bedrooms and bathrooms. That's all very well structured. Text is not structured at all. It can be variable length. You can have paragraphs. You can have whole documents. It's unstructured. Um images, same thing. They can be many different resolutions. They have a little bit more structure to them in the sense that they are matrices somewhat, right? They have um rows and columns of pixels, but still they're considered unstructured data. um and and uh so neural nets and deep learning will unlock the ability to work with those. There's so many problems that work with images like object detection, image recognition, um you know, and that extends to so many applications. And then working with text, obviously generative AI, working with text or even doing things like sentiment analysis or language translation, a lot of things work with text. So we need these advanced neural nets to be able to deal with that kind of data. Um the the models that we've studied so far just are not capable of working with that data very effectively at all like logistic regressions or decision trees or even things like an XG boost. They're going to struggle on that unstructured data. They're not really going to be able to process it. Neural net on the other hand is going to be able to. That's one of our motivations for studying it. is it unlocks this whole set of applications for us that we couldn't do otherwise namely working with text and image data. So some particular tasks there that we were that we are going to study. So things like image recognition. Um now the very basic example that I probably have said before but you know something like being able to predict if an image is a cat or a dog or if an image is um a cat, dog, a giraffe or a wolf or whatever it is, right? So image recognition tasks and that really extends into so many applications like object detection. Um so working with images really huge deal. We're going to devote a lot of time to it. Um but we really need neural nets to be able to do that effectively as we are going to find out. Um things with language like understanding and processing human language. We have these transformers that do this. so effectively, but it wasn't always that way. And it was nearly impossible to do um with basic uh models. Just nearly impossible to understand language at the level that we see today. If you think about the the amount of language understanding it takes to have these um things like GPT and clot and stuff, it's just immense. And they're but they're really huge neural networks that are doing that and they're learning against so much data. But we really need deep learning for that. Um and other applications things like speech recognition you know we haven't talked about it but another type of unstructured data is um audio right so audio data being able to do things like recognition of audio translation into text that's a really hard problem basically impossible without neural nets uh to be able to recognize speech and audio. Same thing with video. I know it's not listed here, but video certainly unstructured. We need uh neural nets to be able to process those um as well. So basically any type of unstructured data, we're going to use neural nets to be able to process them. Um a lot of things like um Alexa devices for instance have been powered by RNN neural networks to uh basically recognize your commands, your voice commands. Um, that's all neural network powered, right? To be able to recognize a voice command, translate that into some text and then use that text for uh creating uh tasks or doing things or or generating new text to be able to respond. So this speech recognition part is all um neural net driven namely RNN's recurrent neural nets um and and in modern times RNN/transformers we see transformers kind of taking over this space as well a lot of language stuff is being taken over by transformers but traditionally they've been RNN's so um let me give you some breakthroughs. So for example, you know, we talked about this period of 2010s. One of the uh breakthroughs in deep learning was in 2012 the Alex Net paper which I encourage you to go look up and and read a little bit about. Um so so the Alex net is a type of convolutional neural net and it introduced really the convolution as a preeminent operation on image data to be able to classify images into different categories. Um so it it this network architecture using convolutions um blew away the competition in the imageet challenge which was a um basically taking a bunch of images. So you can see here like the training data has 1.2 million images testing data has more has 1.5 million images. Your goal is to categorize those images into 1,000 different categories. So, so think of it as like 1,000 different animals, not just cat or dog, but a thousand different uh animals. And um the best methods at the time um things like basic neural nets or maybe even like an XG boost were blown away by this convolutional net um by more than 10% which is astronomical amount in in research um of this at this kind of level. So this was a huge breakthrough and what we see from there on is convolutions becoming kind of the default um building block of anything working with images. So like object detection, it's all going to use convolutions and it was all inspired by this Alex Snap paper. Um and it was for the first time really like people realizing like hey these neural nets can really work on unstructured data very effectively. Um because the methodologies to work on these images prior to this was to basically try to generate a bunch of structured features which was not very effective. going from like an image into a flattened vector and trying to generate features out of that to pass through like an XG boost um was the strategy but it got surpassed immensely by this convolutional net and then we see just an explosion of computer vision progress with people using uh people using uh convolutions. So, one of the things we're going to study in this course is convolutions. They're really important for computer vision. If you're ever going to do anything with images, uh any kind of neural nets processing images is going to use convolutions in some fashion, right, in the architecture. So, uh we'll devote some time to studying convolutions for sure. But this was a huge breakthrough at the time. And again, 2012, that's really not that long ago. It's it seems it's hard to believe that this was 2012. Um and you know that is kind of a long time somewhat but really not really not really in historical terms. So you know we'll talk about some more breakthroughs. That's just one of them. Uh however there are challenges uh that were encountered along the way. Um obviously like a lack of research funding was a big challenge. Um probably bigger like bigger than that definitely are these next two which are data and hardware. These are the two things that stalled deep learning progress for a long time. Hardware mostly because neural nets and their structure this is something we're going to learn about benefit immensely from advanced GPU hardware. Um it's primarily because the computations that a neural net does benefit a lot from the parallel parallelization that a GPU can provide at the hardware level. Um and so only until recently have we had a lot of advancements in hardware such that GPUs are readily available. wasn't always that case. And also, it wasn't always the case that that the code, the Python code could take advantage of a GPU, but it it can today pretty easily um with our frameworks like PyTorch and TensorFlow. They they make it easy to take advantage of the GPU hardware. And so that's why we've seen an explosion in the last 1015 years because GPUs are more readily available and can be taken advantage of by the code. Um, same thing with data. Data is more readily available, right? So, there's a lot more of it, a lot more it's it's cheaper to store it, a lot easier to collect it. So, it wasn't always that case. And what the cases with neural nets is you need a lot of data to train a neural net more than what we've used so far for the structure data problems um that we've dealt with. Um so the the the expensive nature of storing data and getting access to it was also prohibitive to a lot of progress in deep learning but that's that's been kind of fixed right it's now a lot cheaper than ever and consequently we've seen a big explosion right last 1015 years let's talk about deep learning then uh a little more in particular so we just covered the fact that um deep learning is really a subset of machine learning focusing on using um neural nets and often what are going to be deep neural nets which are going to be neural nets with uh a lot of layers or a lot of um complex structure to it um to be able to uh learn from data. So learning from data is machine learning right but using neural nets to do that is deep learning. Um, so that's where we're at is using neural nets in particular, moving away from just basic machine learning into more specialization here with neural nets. So um what's nice about neural nets is that it can work with structured data but as I've said it can unlock the ability to work with unstructured data from many different domains such as imagery, audio, video, text um to make predictions uh usually um and the um uh this has unlocked a lot of progress in domains like computer vision so object detection, self-driving cars, those kind of things NLP which is dealing with language so language tasks like sentiment analysis translation text generation um even working with audio data like speech recognition a lot of progress in those fields um and deep learning is going to be able to surpass a lot of the machine learning techniques that we've seen because they have a very uncanny ability to learn complex patterns um One of the things that is um that gives that the ability is how complex the model is. But the other thing is it's going to use a lot of data. So it's going to have a lot of examples to go off of. It needs that in order to extract those complex patterns amongst imagery, text, audio, video, right? It's going to require a lot of data. So that's something we'll see when we start doing examples. Okay. So just to compare the two. Um so we're talking about deep learning and machine learning and remember deep learning really being a subset of machine learning. Um machine learning is a little bit more broad um where we have you know unsupervised learning supervised a lot of the problems in deep learning are still supervised meaning that there's still a label like if you have an image there's still a label of what that image is. Is it a cat? Is it a dog? Is it a giraffe? Right? or one of those thousand categories like an imageet uh challenge. So there's still a supervised nature to a lot of these deep learning problems. It's just the model that we're using is now a deep neural net, right? Instead of a decision tree, instead of a random forest and also the data is inherently different, right? It's going to be imagery, it's going to be text, potentially audio, video. So, we're moving away from structured data and really focusing on unstructured use cases with things like images and text. Um, one of the things that's true that we've seen so far is especially with machine learning course we just finished, we saw that it's really important to get those features right. You know, we have to do some data prep to get our features correct, get rid of any null values. um maybe even engineer some features like scaling it or adding two features together. Um it's really important to do that. What we're going to see with deep learning is it's not that important to do that. It's not that important to do feature engineering at all because then the network's going to be so powerful at picking up on patterns from the original data like the original images. um we're not going to need to do a lot of manual feature engineering with with deep learning. So that's going to be one kind of advantage here is not a lot of manual uh feature engineering and that's actually the the imageet uh challenge we just talked about with AlexNet uh convolutional net winning that challenge that was the thing everybody was doing was trying to do a lot of manual feature engineering so that they could use like a a classical machine learning model like a XG boost um But doing that wasn't as effective as just a really good neural net that can just extract patterns from the raw data very effectively um using something like convolution. Okay. So deep learning as I've said really excels in those kind of tasks like image recognition, NLP, speech uh recognition. Um so we've talked about that. Uh it NLP remember is short for natural language processing. So you'll probably hear me use that word NLP especially when we get to later on and we're studying things like transformers and RNN's those are just language tasks right so think of text working with text text generation sentiment analysis uh speech or language translation I should say um so processing natural language which is really text right for for our use cases um deep learning often requires really large data sets. So much more data than you see in a machine learning technique um and also more computational resources. So this is think GPU most deep learning is going to benefit from using a GPU whereas machine learning doesn't really use that at all. Like if you do a decision tree, an SVM, a logistic regression, you're not using a GPU at all. You don't need to you it will still be pretty effective um without that. And uh whereas deep learning a lot of models will take forever. Just to give you some context like the um GPTs of the world, these transformers that are you know neural network based um they train on data that's about the size if you took all of the text from Wikipedia and more. So all the text from Wikipedia um plus maybe all the text that would be in an encyclopedia collection. Think of a data set that big as a prototypical training data set for something like a GPT and think of thousands of GPUs working together. Even in that case, it can still take days to train a model. Days, not minutes, not hours, days. Because the models are so big and there's so much data being processed to train it. um that is uh how much time it takes to train those kind of models. So there's substantial amount of resources um which is why there's so many uh news articles written about you know Nvidia and partnering with these uh generative AI companies like OpenAI and Microsoft etc. Um, so a lot of resources are are needed. Now, for the networks we're going to build in the in the program, not so much. We'll we'll still explore using a GPU and we're actually going to practice using a GPU. Um, we will have access to free GPUs in collab. So, that's really nice. Or even the SimplyLearn lab environment has free access GPUs. Um so we will practice using it and it will benefit us for things like image recognition working with imagery or RNN's or transformers even um they'll benefit us but just as a rule of thumb deep learning requires more data more resources to to train computational resources. If you have a Windows machine, sure. Um, you can use the Nvidia GPUs. If you have a Mac, um, like a MacBook, um, even like one of the M1, M2, M3, M4 series, those have built-in GPU capabilities that, uh, we can take advantage of. So, I'll share those resources with us when we get to building our models. But even if you have like one of those M1, M2, M3, M4 Max with the the silicon chips, they can use their internal GPU to train models. Um, and I've done that before and it's really nice. It speeds up the training a lot. Um, we can also use Collab. So, Collab offers free GPU access, uh, which is nice. Um, so we can use that too and we will. So we'll practice that uh as when we get to it. And then finally, deep learning. You know, we're going to be using neural nets that uh are potentially very deep in the sense that they have a lot of layers. So we haven't studied what a layer is yet. That's okay. Just think of a really complex model with a lot going on with it. That's going to be um you know, way more complex than something like a decision tree or SVM or random forest even. So that's what we're talking about here. to train deep learning, we uh often will use GPUs uh because they're going to be really effective. Most traditional machine learning methods that are not deep learning can get away with just using a CPU because they have very simple algorithms that are that are training that do not require using the um underlying GPU resource that can really speed up uh uh deep learning. So I wanted to take us through some of the successes that have been recent, some of the foundational kind of moments um and there's been growth in a lot of areas. So working with audio, computer vision, even reinforcement learning that has been um powered by deep learning. Um most reinforcement learning algorithms prior to deep learning were based on what are called kind of tabular methods. um where they were uh keeping track of um uh rewards and and uh data in in a more structured format. Uh but deep learning has since made a lot of progress in simplifying that and and uh making it much more powerful and and even in reinforcement learning. We'll talk about one of those uh cool moments there. Um so back to the Alex set, you know, we talked about this in 2012. This was the champion in the renowned uh imageet challenge. Um this is where again they used convolutions to really uh power um learning imagery and being able to classify images. Um and this this was so groundbreaking that uh a lot of moments in deep learning over the years have been referred to as kind of a imageet moment or Alexet moment um because how big of a breakthrough this really was. So 2012 um these this convolutional model was kind of created. Now we'll study this architecture later on of what this actually means, what these uh layers are doing, what is a convolution. Um we'll study all of that when we get into computer vision. But this was a huge breakthrough in working with image data. Then in 2013 we have a really important breakthrough of working with text known as uh the first kind of word embedding model which is really really important for where we are today because word embeddings play a really critical role in working with text. uh because these are um models that map text into numerical vectors which is really critical for um working with language modeling and every transformer really has a word embedding at the very front of it to take raw text or tokens and essentially um make those into a mathematical vector so the neural net can work with it. The very first uh model there that was a big breakthrough this is 2013 um was a model called wordtovec uh which uh you may have heard of. This was uh used for uh pre-processing text essentially to to map it from raw word or tokens into uh vectors. So really really critical for any kind of language model to do this. And so this this has been around for a little bit longer than the language models, but this was a really important breakthrough for them for for generative AI. So every gener like GPT cloud every everybody has a word embedding model that's at the very front of their uh transformerbased u text generation model. So this is this is a huge breakthrough. Um then we have the very first kind of uh language models known as sequence to sequence. Now this is not using a transformer. So this is using RNN's to try to do NLP tasks like a language translation from uh German to English or French to English or or English to French whatever it is. Um, and they tried to to make uh NLP, so doing things like translation, language generation, sentiment analysis, um, they tried to make it a lot better and it did. So, at the time, RNN's were kind of the best language models around, and this is 2014, but they quickly get surpassed by Transformers not too long after that. And that's where we are today. Everything is kind of transformerbased but at the time RNN's were kind of the best uh for language. We will study RNN's uh later on in this course. Recurrent neural nets RNN's which are just a flavor of deep learning that uses a special architecture. So here it is again just some different variations. And there's a lot of different variations on the RNN architecture um that attempts to predict um again what word should be next in the sequence using the RNN. We do that very well today with transformers rather than RNN's but at the time these were the best around. Okay. Right around the same time in the image generation uh field we have GANs as a big breakthrough. So what what are GANs? They are uh generative adversarial networks. Um these are models that generate images, fake images. Um and they do it through a competing uh adversarial networks that um one is a generator that builds images, the other is a discriminator that's supposed to predict if that image is real or fake. and they kind of boost each other up so that the generator by the time the training is done is so effective at generating fake images, they kind of mirror real images. This actually led to the explosion in deep fakes. So deep fakes became a real issue. There's still issues today, but the the first issue the first time it was an issue was really when GANs were uh prominent. They were developed in 2014. So again, not too long ago, um, most models that generate images today are actually not GANs. They're what are known as diffusion. So like Dolly and and the Google banana models that you can um put in some text and generate an image. Um, they're not GANs, but GANs were the earliest kind of image generation models that were out there and really effective. And some of them are still used today for sure but majority of image generation models are are what is known as diffusion which we will study later on when generative AI but this was a huge breakthrough in using neural nets to generate images right that idea of generating images through neural nets this was a big breakthrough here 2014 okay and then on the reinforcement learning front um we have the Alph Go developed by Google and its Deep Mind team. First uh kind of machine program to defeat a professional Go player. So, if you've never heard of it, Go is a board game that is thought to be more complex than chess cuz the board is a lot bigger. Um it has more complex rules to it. um and more uh more states in terms of the the the way that the pieces can be placed on the board. There's a lot more variability in in the pieces uh in their positions on the board. So, it's thought to be a more complex game and it was mastered by reinforcement learning using deep learning. So this is the first time where deep learning really um benefits a reinforcement learning model so much that it's able to beat uh go professional go players. Um this was a huge breakthrough 2016 about 10 years ago. Um huge breakthrough especially in reinforcement learning. This is where deep you know deep learning being able to you be used in RL big deal. It wasn't always that way. Okay. Now to the big one which is 2017. This is what I alluded to earlier with transformers. So this is where um the very first transformer was created. We're going to study transformers later on in the course, but transformers changed everything when it came to language tasks. So I said RNN's were used prior to transformers around 2014 to do a lot of language tasks. Transformers come along a few years later and really revolutionize everything. And all of the generative models today like Chad GBT, like Gemini, like Claude, they're all based on Transformers. They are just massive extensions of the Transformer idea. Um, these large language models, these LLMs are all based on the Transformer idea. This was a huge breakthrough. I remember myself um I had just gotten into data science, machine learning in the industry right around here like 2016 and I remember reading this paper at the time. Uh and how important it was. It was just a huge breakthrough. Uh the attention all attention is all you need. Highly encourage you to go and read that if you can. Um it's it's amazing. Now, it might not make sense yet. By the time we get to transformers, our lesson on transformers later on in this course, probably make more sense, but um really fascinating breakthrough. This is 2017, not that long ago, right? Really not that long ago. And not too long after this, we see the explosion in generative AI, right? Especially for text. So, those are just to name a few. Obviously there's been other advancements over the years but those are some of the biggest um and uh what we've seen is uh um deep learning has really come a long way. There's so many cool algorithms. There's uh there's also been an explosion in in implementation uh um because we have these um frameworks readily available that are open source that anybody can use. So more it's it's more and more people have access to these libraries and are able to develop their own algorithm in the community. So the research community has benefited a lot and we've seen because of that an explosion in AI tools and uh AI models because of the ease of use of these deep learning um frameworks and and all the advancements that have come over the years. So a lot of cool uh libraries, a lot of efficient algorithms out there and we're going to study those like we're going to study TensorFlow, we're going to study PyTorch so we can build our own networks. But the community has benefited greatly and has been part of the reason there's been an explosion recently is because of how easy it is to get access to those libraries and start building your own neural nets and trying things out really easy. Okay, so that brings us to uh an important question which is why should we study deep learning? I said before that, you know, it's an advanced topic. Not everybody studies it. It's a bit of a specialization within machine learning and a lot of people just get through data science and machine learning and kind of stop there. So, what reason do we have as learners to actually study deep learning? I'm going to talk about that. Um, one of the key reasons is that deep learning, as I said before, unlocks problems across many different domains that were that are difficult but are necessary in those domains. So, I'll give you one example is healthcare. There's so many image related examples in healthcare that are really vital that neural nets help with significantly. So think about analyzing images and image data of the body or of um the brain of um potential diseases. Those can be more readily analyzed with deep learning. Um and so healthcare has seen a lot of advancements in analyzing that kind of data with with deep learning, autonomous vehicles of course, robotics, um even like fraud detection, there's a lot of advanced methodology there or even like time series forecasting has benefited from uh neural nets. So there's so many advanced use cases that neural nets can benefit um because it can work with text, because it can work with image data. Um it just unlocks a lot of possibilities. Um so so that's one reason is it just gives us many more problems we could potentially solve in in different domains. It's worth studying it there. Another thing is we have so much more access, right? So, um, GPUs are more readily available. Um, cloud environments have GPUs readily available. We're going to see that in Collab, like we are going to be able to work with GPUs in Collab for free. But certainly, like if you're paying for it on a on a cloud provider like GCP, AWS, Azure, they all have GPUs that are relatively cheap. um considering that we're actually using you know advanced hardware there. Um so because of that it is because it's so readily available um we have a better chance at utilizing deep learning more than ever because GPUs are so so much uh easy to get your hands on than ever before. Um, so it's not to say it's perfect. Like, you know, we were joking earlier about the in Nvidia um 5090. You know, there are some physical hardware elements that are expensive um and not easy to get your hands on, but like cloud resources for GPUs um are readily available pretty easily. We're going to see that with Collab in particular. Um, another thing is, uh, you know, GPUs. I I forgot to mention this, but let me go back to the slide actually is, um, one of the things that Nvidia really powered was this uh, CUDA framework. This is something that I mentioned a little bit earlier. This allows the Python library to interact with the GPU and be able to offload operations to it. instead of having to keep everything on the CPU which is much slower for the neural net operations that that it's doing and its layers. Um and so in uh there was kind of this um framework to interact Python with the hardware this CUDA um library and um because of that uh the libraries that manage neural nets like PyTorch and TensorFlow have integrated the CUDA framework and have the ability to easily manipulate operations on the GPU um which makes working with uh neural net so much easier. easier. So you can take advantage of a GPU that's just a long way of saying you can take advantage of GPU really really easy with the modern framework. So with PyTorch with TensorFlow they can interact with the GPU so easily because of things like the CUDA framework um makes it so much easier. So the the CUDA is we don't ever see it. it's abstracted away with the framework like PyTorch or TensorFlow and we're going to see that we are going to build models in PyTorch and TensorFlow that will take advantage of the GPU. So we're going to see this directly how easy it is to truly take advantage of that hardware and it's going to speed up our operations a lot. It's it's going to make it so much faster to to do that processing. Okay, so we just said GPUs uh play a critical role um and they're more readily available than ever. So that's another reason to kind of learn deep learning is we have access to these resources. Um and it's it's really because the GPU can do parallel math operations so efficiently, way more than a CPU. Um so they they really we're going to see the effects of this you know when we do especially working with um computer vision and convolutional networks working with a GPU then becomes almost necessary it'll speed up operations so much like if we didn't use it we'd be sitting there for minutes maybe hour waiting for the model to train but if we use a GPU it'll take seconds it'll really speed it up a lot okay one of the other reasons we we've kind of hinted at this is just a low barrier to to entry. So, so this is just to say why not learn it if you're going to learn machine learning. There's so many problems that use neural networks and why not learn deep learning because it's never been easier to do. Um there's hardware available with GPUs more than ever. We talked about that. There's user really extremely userfriendly frameworks like TensorFlow and PyTorch to help us build neural networks and they're open- source. So they're easily we can pip install it, get it into our environment and start building neural nets and training them really really easy and we already have all the fundamentals we need to really understand model building, model training, um model evaluation. we have all that fundamentals from working with uh Python, working with uh machine learning that we've already done, data science. So, it's pretty easy to get into at this point. So, why not, right? It it's just going to help us uh expand our the breadth of our skills and problems that we can solve. And like I said, open sourcing these like TensorFlow and PyTorch was a big deal. It made it so much more accessible for the community to build their own neural nets. Prior to that, it was really limited to these big companies. You had to be in one of them in order to really start building neural nets. But now anybody can because these are free and open source like TensorFlow and PyTorch. Okay. So, another key reason and you may be wondering this especially because of generative AI um taking over uh a lot of development these days but still there's a need for trained deep learning experts and practitioners. So many um companies still work with deep learning models and need deep learning expertise um to be able to like tweak the training, set up the training, set up the model, um make the adjustments if the training doesn't go well, really understand the model. Even if you're using generative AI, you still need to have an understanding of how the modeling works and how it functions in order to effectively use it. So there's still a need for deep learning expertise regardless of AI and generative AI. So it's a good opportunity. It's a it's a good skill to have. Um, I can say this like beyond just having data science and machine learning experience going through this program, I think having the deep learning experience is just another really um beneficial thing to have if you're trying to get into the industry. It it really unlocks more potential for you, I think. And again, not everybody studies it. So you're getting a uh kind of uh um advancement above the competition potentially by learning deep learning. Any questions about any of that? Hopefully that makes sense to us. We feel motivated to learn deep learning. Again, it's an advanced subject. Not going to be the easiest subject. There's going to be challenges along the way. Um by no means is it easy, but I think it's worthwhile. And there's still a lot of problems. You know, one of the things that we didn't mention on there that I think is important to call out is so much of generative AI is deep learning based. So, it's worth having the expertise in deep learning in general to be able to apply those skills to generative AI and really understand the the kind of models there like a transformer, right? you if you understand how it works effectively, you can better work with it. All right, let's talk about some applications. So, um there's many, as I said before, there's many different uh fields that really benefit from uh deep learning and working with neural nets. Um, so here's some of them, just a few, you know, um, NLP being a huge one in terms of processing text, being able to understand human language, that's a big one. Uh, self-driving cars, using computer vision for object detection, that's something we'll talk about and study. Um, not only like detection of objects, but classification of them. So, think of like, okay, here is an image and inside of it, there's a pedestrian over here. There's a there's a a sign like a a stop sign. There's a tree over here. There's another car over here. Um, here's the road uh um divider. Um, so so many objects that are that you can detect in the environment and really classify what they are. And that's all powered by deep learning. Um, same thing with audio processing, transforming speech to text. That's how we talked about earlier. That's how things like Alexa and those voice assistants really work. It's translation of audio into text. Um, and then, uh, manipulating images as well. So, generative AI with images. So much of generative AI is all deep learning based transformers um diffusion models that uh generate images GANs all of these generate and manipulate images they're all based on deep learning so much of generative AI is deep learning based okay now there are some limitations uh one of the things to realize about deep learning we've talked about it a lot so far is a significant ificant amount of data is usually required. Meaning that deep learning may not be the best for tasks that have um little amounts of data. And so what is little? Um usually we're going to be dealing with uh in the tens of thousands if not more. That's like a that's on the very low end of maybe images or text examples in order to train a moderatelysized neural net. uh maybe even on the smaller end of a neural net. The bigger the neural net that we have, the more complex of a model we have, the more data we need. Um so this is this is like for the problems we've dealt with so far, throwing a neural net at it would likely be overkill and would not be effective. So it's not like we can just throw a neural net at our housing data and expect it to predict the price very effectively. it doesn't work like that. We would need a lot more data for that to be effective. Um, and we would probably need to tweak the neural net uh a good amount to get an effective model there. Um, the neural neural nets have a tendency to overfit. So they they need a lot of data to avoid that overfitting. They need a lot of examples. Um, and that can be sometimes difficult to come by. um the amount of data that's required to be collected. Generative AI like GBT requires so much data. That's why they're using people's chats to continuously build a data set so that it can keep training it. Um so much data is required there to train a really effective model especially one that big. Um just requires a huge data set. Um let me give you an example. So consider you know a model that's we've we've used this example before right just differentiating or classifying cats and dogs. So this image is a cat this image is a dog and etc. Um, in order to to do this effectively, we would need many different images of cats and dogs. And and likely those images would have to be of different size cats and dogs, different breeds, um, from different angles, potentially from different lighting and brightness of the image. Um, many different variations on those, and they'd also have to be labeled, right? So, it's it's still a supervised learning problem. Um so you know gathering that data set may be an expensive endeavor may not be trivial. So um that is kind of a a cost burden of doing deep learning is the data. It's going to be a lot. Um you may need a really massive data set and that can be difficult to acquire in some instances not all but in some cases that may be difficult to acquire. So it's just a it is not a limitation so much as it is a caution that if you're going to use deep learning you generally need a lot of data and so you have to be prepared to collect a lot and process a lot in order to um have an effective model. The other thing is hardware. We've talked about this. Now, it it's again not necessarily a huge limitation because we do have access to GPUs, but you may need to pay for it. Um, and you may need advanced hardware. The more complex your model is, the more you're going to benefit from, you know, uh, a more complex hardware like a GPU. um it is more readily available than ever, but it doesn't mean that it's always going to be free and always going to be easy to uh acquire. So there may be some work there to use that or to acquire that hardware um for a really complex model. Now that being said, we are going to work with free GPUs for the examples we will do, but the examples we will do will be relatively uh moderately complex examples. They won't be um you know massive models that would require a much more significant hardware investment to to be able to run. So I used the example before um something like a Jad GBT is going to be using thousands of GPUs. That's a significant investment. If you're just using one for free from like Collab to do simple examples, that's one thing. But for a really big network, um you're going to want to use a significant amount of GPU resources um which may be costly, right? So it's just something to be aware of. Okay. So one of the other limitations I mentioned earlier is that deep learning because there's um it's such a complex neural network it can be susceptible to overfitting. Um in fact they are very susceptible to overfitting can easily overfit especially if there's not a lot of training data. Um and that's just the nature of the neural net. it it's so complex of a model that it can effectively memorize a training data very easily if you don't have guardrails in place. And that's one of the things we're going to study in that lesson on optimization of the training process. How to avoid overfitting. There's certainly things we can do and we're going to study to prevent a neural net from overfitting, but they are very prone to it naturally very prone to it because of their complexity. Right? Remember, remember we talked about overfitting. The more complex a model is, the more likely it is to overfit. The more basic a model is, the more likely it is to underfit. So, neural nets fall in that category of being really complex. So, they are very prone to overfitting. So, something we have to kind of uh be on the lookout for. We'll study some techniques to help us uh overcome overfitting or prevent it. All right, another one that is very underrated and a lot of people don't think about when it comes to deep learning is the explanability of deep learning is very very limited. So when we studied machine learning and we did things like logistic regression or a decision tree that was super explainable because we like in logistic regression you get coefficients on every feature right you get those betas so we know exactly which features are important because they're going to have higher coefficients right and same thing with like a decision tree you get an actual tree structure so you know that you know the path that leads to a prediction and so they're really readily explainable. The issue with neural networks is they're often a black box and and what that means is you put data into it, you get a result out, but it's almost impossible to explain every little thing that happened that led to that prediction. Very difficult to explain with neural nets because they're such a complex model. they don't have a very natural way of um having like a single coefficient. Most um to to give you some context, most modern neural nets like like a uh GPT um that's generating text is going to have in the billions, tens of billions, hundreds of billions if not trillion weights. uh which are you know think of those betas it's going to have like a trillion of those betas. So it's impossible [snorts] to say what a single contribution to the prediction is. Um now this is an active area of research um in the research community is people are trying to figure out good ways to um break down how a neural net's making a prediction. But it still is near is pretty much not possible relative to other machine learning techniques. Um they're just they're really complex models. You put something in, you get something out. It's hard to say what happens in between. You know the architecture, but it's hard to say what exactly contributes to that final result. [snorts] It's hard to hard to trace that back. Um so uh deep learning can be very difficult for explanability and and this actually has an effect in like regulated environments. Let's say you're in the finance industry and you're making um like loan decisions based on a model. Um you probably don't want to use a a neural net because it's going to be hard to explain how you predicted your like uh loan or not loan decision. um you you probably want to use something that is more explainable like logistic regression or a decision tree which would those you can directly see the contribution of their features right um leading to a prediction whereas a neural net you're not going to be able to see that it's just too complex of a model okay so that's another kind of drawback of deep learning now there on the opposite end of that there are cases that we don't care about the explanability at all so you think about like a self-driving car. We don't care the explanation of why this is a stop sign, why this is a pedestrian, why this object. We just wanted to do that really fast. It needs to be real time. It needs to be really fast. We don't care about an explanation in that case, right? We just need it to make an accur accurate prediction. So, there are certainly cases where we don't care about that. But in a lot of like regulated industries, you may care about that. uh if you're if you're using a model to make some type of decision um you may care about the explanability of it in which case neural nets are not ideal for that. All right, let's talk about the well before we do this before we talk about the frameworks any questions about uh some of those limitations. So recapping those a lot of data is needed a lot of hardware resources are needed typically especially for complex networks and then uh there's a real lack of explanability which could be prohibitive in certain situations not every situation but certain uh modeling scenarios. Let's talk about some of the uh the frameworks then that we will use. So these are going to be uh basically our libraries and tools to um help us develop neural networks. Um basically they um they allow us to design neural networks. They allow us to train them. They allow us to debug them like how they're working or what kind of uh evaluation we're doing, if they're producing any uh any bad results. Um and also for deployment. Now, we won't really focus on deployment uh too much. That's not that's going to be out of the scope for us a bit. We're going to focus on um how to design and kind of train them. But these Python frameworks will allow us to build and train and evaluate uh neural network neural networks. Um and this is huge like like I said earlier, it wasn't always this case, right? You really had to be part of these large corporations that were doing this like a Facebook or a Google in order to have access to the tools to build neural networks. and now they're readily available like PyTorch and TensorFlow. So the main the main uh frameworks that we're going to study are uh PyTorch and TensorFlow. Now caris is on this screen because what we're going to learn is caris is really um embedded into tensorflow. It was actually developed by Google, the same people who made TensorFlow. It's really just a part of it. It's it's an interface to core TensorFlow. So when we hear the word caris, we should really be thinking TensorFlow. It's the same thing. Um it's actually technically built on top of TensorFlow and allows us to interface with TensorFlow. Um but so so these these were built by Google and then um PyTorch was built by Facebook and now Meta. Um but at the time Facebook um and they are open source now so they're widely available and of of course we will learn how to use them to build neural nets and uh uh train them. Um they're both widely used. So both widely used community. Um most people ha just have developed a preference for one or the other just based on either what they originally learned to begin with or um what they just naturally like. Um but the the truth is there's uh a lot of models out there like um even like an open-source GPT that exists or Llama they're developed in both frameworks. They have a version that's developed in PyTorch. They have a version that's developed in TensorFlow. And it's they do the same things. They mostly just have syntax differences. That's the only only thing about it is they primarily just have syntax differences. Um but again, it's worth our time to study both because both are so widely used. Um and honestly, both are still maintained even though they're open source. Both of these companies still maintain uh groups of developers that are maintaining these libraries. So there's still people within Google who actively manage TensorFlow and there's still a group within within Meta now that's actively managing PyTorch. Um and they they do work on it all the time but it's also open source. So contributions can come from anywhere really, but there's dedicated kind of people within each of these uh companies that are still maintaining these frameworks because they're so critical, right? They're so critical to doing deep learning um and doing neural networks. [snorts] So we're going to study them uh pretty deeply. So caris, let me talk about caris. it is um going to be a a interface in Python to the TensorFlow library. So TensorFlow itself, so TensorFlow was created first and then Caris came along a couple years later um to interact with TensorFlow. TensorFlow is kind of the underlying library that Caris is built on top of and it just it's much much much more user friendly to manipulate and manage neural networks. So most people gravitate towards using the caris library within TensorFlow um to building if they're going to use TensorFlow, they're going to be using caris as as the interface to TensorFlow. So it's just built on top of that um by the same Google group who made TensorFlow. Um and it's it's the advantage of it rather than using the original TensorFlow is so much easier to get started with and build neural nets and so much easier to manage them. It's a really nice interface to TensorFlow. So when we build models in TensorFlow, we're really going to be using Caris, okay, as that um kind of library on top of uh TensorFlow um to to build uh to build neural nets. So we will study that. Um is there an advantage of using caris directly? Yeah, that's what it is. It's it's a really userfriendly interface. So it's designed to make uh building TensorFlow models so much easier cuz the truth of it is like core the original TensorFlow is kind of ugly. It's kind of ugly and like messy to deal with. So um people when they're building TensorFlow models gravitate towards using caris as the interface to tensorflow because to like core TensorFlow is really messy. Um TensorFlow is as we'll learn later is is basically a graph library for building like a tensor graph to to uh mimic a neural network a graph of uh computation. But Caris makes it so much easier to to work with TensorFlow. So most people gravitate towards caris than just you usually you won't ever use the original TensorFlow unless you're maybe doing specific research that would require it. You're mostly always going to use caris to build your models and train them. [clears throat] So that's what we'll do. Okay. So underlying caris is TensorFlow which came first. um from Google. Uh it's you know obviously within Python it's open source now um and this was the original library they built inhouse to to manipulate and manage neural networks. Um you know caris came along as the interface to tensorflow uh to make working with TensorFlow so much easier. Um so that's something that we'll see and actually TensorFlow has some really cool tools uh uh as part of it that we will uh work with later on. They have tools for like visualizing the training process which is really nice. Um so we'll work with those. Okay. So on the other end of it is PyTorch that uh that Facebook developed now Meta um who still maintains it but it you know has been open sourced. um same kind of thing they were trying to work with neural networks and needed an in-house uh you know in-house library to do that and so they developed PyTorch at the time um you know and since it has been open source to the community um and it's it's very similar to TensorFlow honestly has the same kind of features uh as TensorFlow um it's just a different syntax um and uh so that we're going to work with both as I said we're going to work with PyTorch and with TensorFlow um in order to build models and and uh train them. So as I said in the community it's kind of a 50-50 split on which one people have a preference for. I personally um like PyTorch. I I I actually learned TensorFlow first. Um but found myself liking PyTorch a little bit more. Um but it's preference honestly. It's preference. And I've seen um it it really just depends on which what you like better. And then often times like where you're working also determines, you know, maybe everybody has a standard for okay, we're all going to use TensorFlow or oh we're all going to use PyTorch. um you see things like that but um honestly if you know one of them you will be well verssed in the other and and we are going to work with both so we'll get exposure to both just syntax differences really um between the two so what I wanted to do was wrap up this introduction so we've covered a lot in the introduction but I wanted to wrap up specifically with um the life cycle of a deep learning project which I think we're going to see mostly overlaps with the same steps we were doing for machine learning. Um it's just there's going to be more um you know there's going to be more details that are that are involved here when we do training when we do uh model building. Um but mostly the life cycle this is honestly this is the same as machine learning from a high level right it's mostly the same we have some type of data collection and data prep phase which is kind of what this is um so so this is like our data prep that we do for machine learning we're used to doing that um first usually now prior to that there may be some planning that has to be done uh to collect data. And in deep learning, that may actually be a significant amount of work to um go out and gather a bunch of images or process a bunch of images or text data um you know and label it appropriately. That may actually be non-trivial, but uh you know, all the examples we will do will assume we already have that data. Um but other than that it's mostly the same steps in term at a high level of like once we have our data prepped we go ahead and and um train we build and train our model. Um obviously the details of doing this with a neural net are going to be much different than with a basic model like logistic regression or something. Um but once we train the model we then evaluate right we evaluate it on some test data. um evaluate and see how it performs and that and the reason there's an arrow backwards is based on that evaluation we may need to make make some tweaks. So you know at each one of these it could be very iterative um in the training like if we're not if we're underfitting we may need to go back and revise our data. We may need to revise our model even. Um so it can certainly be iterative. Um and then after you have a model you're very satisfied with in the evaluation phase, that's where you can get into kind of um deployment of it and then like monitoring it and m maintaining it. These two things we're not going to be focused on too much because again it's going to be a little bit out of scope for us. Um, we're mostly going to focus our attention on these two phases, training and evaluating of neural nets. Okay, but let's talk a little bit more about these steps. So, the planning uh as I said, this is where um the primarily whenever you're doing deep learning, this third bullet is is really um critical. So I would lean on this bullet as really critical as so planning really meaning what resource we going to need in terms of data what kind of data are we going to need like images text you know what are we working with here not only data but hardware like can we get access to like are we going to need a GPU um can we get access to that um for especially for deep learning that's going to be really critical to know your resources ahead of time um and be able to have those available especially the hardware um like a GPU. So usually there's some thought put into that from a planning perspective um in terms of the resources. Uh then there's the actual collection. So depending on the problem, this could be very different, right? We may be physically taking a bunch of images. We may be downloading a bunch of images or or grabbing them from some database. Um we may have IoT sensors that we're collecting sensor data from or like our self-driving car is taking a bunch of imagery sensor data. Um so that could be different. um audio, you know, audio or video. We could be collecting for examples. We need to collect that and we also need to label it, right? We need to have adequate labels. Um and like think of the cat and dog images, right? We need to take every single image and provide a label with it. Um if it's a cat or it's a dog and we also need to gather those images, right, in the first place. So there's going to be data prep that goes on here. No different than before. Okay. Then we're going to talk about the training phase. So one of the things about the training phase is that's not really shown here but is definitely a prerequisite is we first need to build obviously we need to build I would say build slasharchitect um architect uh the model right the neural net neural net so obviously we need to do that and that will feed into the rest of the training process Now I um we'll talk about this later. I'm not too concerned about these steps. These are just the steps of the training that we're going to learn. Um so this this will really go backward and iterate through these steps. But training of a neural net is very unique. It basically involves sending a bunch of data through the network, figuring out what your error is, and then going back and updating the network. That's essentially what all these steps do. And then you iterate through that on your data over and over and over again until your network uh learns how to effectively make predictions. Um so it's a very iterative process which is what makes it take a long time. You're going to iterate through all of your data and you're going to do that on a number of steps which could be thousands of times that you're doing that. Um, and it's really the same thing over and over again of again sending data through computing how far off you are in the error using that amount of error to go back and update all your your model. Um, this process is where that gradient descent is going to come into play. So we kind of touched on this in the recommendations with that matrix factorization learning. um we are going to study this in detail quite a quite a bit more here in neural nets because that's really what's going to power neural net training is um gradient descent. So again these things I don't expect us to know right now just giving you a preview of the training is a lot more involved. Um there's there's a significant amount of computation involved here in like sending data through, generating predictions, computing error, updating all of our weights in our network, which could be billions and billions and billions of weights. So this this back propagation step could be significantly computationally intense um in the gradient descent. And you're doing this many many many times in iteration uh across all your data. So it is significant the amount of resources that are spent in training a neural net. All right. So there's that's the training and again we're going to have a lot more to say about these in future lessons. So let's put these terms on hold until we learn more about neural nets. I'll we'll cover what forward propagation means, what loss means, what back propagation means. We're going to study all of those. So a lot more to come on those but for now just think of the training phase as we have to iterate through having our network learn and adjust learn and adjust learn and adjust over and over and over again until it learns effectively. Um so as I said the model is goes through iterative uh performance um evaluation essentially in terms of how much error it's generating and so we keep an eye on that and continue to train it over the course of the training phase which may be lots of iterations and you can go backwards at this point. So based on the training, you may go back and adjust your model. You may go back and adjust your uh parameter like your hyperparameters of the training which is going to be a few. You may go back and adjust your data which is what this slide's saying is that we may have to go back and collect more data. We may have to double check our labels. Um there could be a lot of reasons that we need to go back and make adjustments. It could be we change the architecture as well like we actually change the model itself the neural net. Um so going back to this diagram there's a lot of iteration that happens. So you know based on the training we may go back and alter our data we may alter our model based on the evaluation we may go back and um alter our training hyperparameters. a lot that we you know may be modifying during the course of the training. So the training is going to be more intense I think in neural nets than what we've seen to this point with usually what we've seen to this point is we just run a fit and everything happens for us right especially with scikitlearn right we just run fit and boom we have a model that's trained like a logistic regression or uh decision tree right fit and it's done this is not going to be as simple there's going to be a little there's going to be more setup more moving parts involved olved um more computationally intense for sure to do to do the training. Okay. But either way we slice it, there's still going to be an evaluation phase, meaning that you know even if we train, we still need to compute metrics to see if our model has performed well enough on the training data and on this testing data. So from a supervised learning perspective, that step will basically be the same as what we've seen before. Once we've trained the model, we're going to apply it to a hold out test set of data um and decide whether or not the model's performing well. Is it overfitting? Is it underfitting? We can tell that on a test data just like we did with machine learning. That will not change. So what we're going to do is still have training data and a chunk of data that's probably smaller but still test data that we will train the neural net on this and then apply it on this to evaluate. Right? So we're still going to do that even we did that in machine learning. We're still going to do that here in deep learning. Um and the idea there is to make any adjustments um that we may need based on the performance we're seeing. This is what we just said. So still an evaluation phase that's more or less the same. Obviously just a different model this time. It's a neural net. Okay. Then if everything checks out there, we can deploy it. And there's many different ways to do that. Not going to get into them in this course. But uh suffice to say that once we have done the evaluation, it's kind of ready to go and we would do a lot of um kind of packaging up and infrastructure and things to kind of host the model. Um many different ways to do that, but uh the goal here is to be able to actually use it to make predictions in in some some way, right? So maybe we're using it in like an object detection inside of a self-driving car or something. Um, and of course once you have that model out there, you're going to keep monitoring it and maybe make adjustments as you see the predictions coming in. So one of the goals that of even building the model in the first place is of course we're going to apply that to new data that has never been seen before and that's where things can go off the rails. So you really got to monitor the performance as you are predicting on new data, right? And see how it's performing. So um you may want to retrain it, make adjustments, maybe in the extreme maybe you have to go back and uh change the model up. Um but you should be keeping an eye on the performance if you are using that model on new data. So uh just to wrap up here on this introduction lesson. So that's everything we've covered so far is just background information about neural nets. So we have not talked about what a neural net is technically from like a math perspective. We haven't covered um you know how we train it. Um that's all to come. This is just background, right? So we've talked about deep learning being a specialized subset of machine learning that particularly focuses on using neural networks. We've talked about that. Um, and we've also talked about it's really well suited for unstructured data like images, text, audio, video. Um, so it's really really good at that. Uh, however, it requires usually a lot of resources to function. So it requires a lot of data, a lot of computational power like a GPU to be effective. Um, and then in terms of the life cycle, mostly follows the same steps as machine learning. It's just those steps are going to be different because we're dealing with a different kind of model, right? A neural net and go to lesson three. So, we're going to start our study of neural nets and particularly with the most basic kind starting with a perceptron really like a simple model of a neuron and the idea is to build up from there into uh bigger and bigger neural nets. Um so what we're hoping to learn here is basically get into more of the details of how a neural net functions. So structure, functionality, um including starting with a very basic perceptron and then expanding that a little bit into multi-layer perceptrons, um and then talk about some of the um advancements beyond there that are out there like a deep neural net that that extends that um things like CNN's, RNNs, we'll mention as well. and then talk about some of the activation functions that are uh present in neural nets. Now, it it won't make sense this moment when I say activation functions, but they'll make more sense when we get into uh what's involved in a typical neural neural net, especially modeling a neuron. Um there's things like activation functions that are really critical. We'll study some of the most popular ones um like relu sigmoid softmax um and then we'll talk about some of the uh some of the issues that can pop up during training in particular um things like vanishing exploding gradients. Um let's start with modeling the neuron, right? So let's start there. Um so obviously the models that we're going to build are networks of neurons and we have to start with kind of the most basic model of a neuron and it's very much inspired by the biological neuron that is uh within the brain. Um and uh so not that we need to know all of these things about it from a biology perspective, but just to know that the model of the neuron is obviously based on how a biological neuron kind of functions. Um so from this perspective you know there's inputs that are sent through they go through the various um nerve cells and then out comes some type of output signal that flows through to the rest of the network. That's going to be um very much how the neural networks will function is there'll be kind of input data flow through this collection of neurons and then out on the other end comes this output data. Um so we'll see that as we go. Um [sighs and gasps] so in a biological neuron there's these various components like the cell synapse being a connection between two cells axons transmitting the output dendrites receiving input. We're going to have equivalent components in the artificial neuron model that we are going to take a look at such as the perceptron. So in the artificial neuron we have a similar setup. Now this is really really critical to understand is that the artificial neuron is very much similar to a biological neuron in which case we take all of the inputs and combine them together to generate some type of output. Now the basic model of a neuron is going to do this really important process of taking the inputs to the neuron. So those um could be let's call them x1, x2 and x3 or more generically however many there are xn um and then waiting those with different weights. Okay, waiting those with different weights and then basically taking a weighted sum. So we have something like w1 uh x1 + w2 x2 plus blah blah blah plus wn xn. Now what does this remind us of? Hopefully this reminds us of something like a linear regression, right? It's very similar to that. Yes, perfect. Linear regression. Perfect. um that that kind of does um remind us of a linear regression. The difference is that the neuron is actually going to process this weighted sum through what's called an activation function. So every neuron is going to be activated or deactivated based on the this flowing through some activation function. So you take that that uh you take that linear sum you pass it through an activation and often this activation. So this f f is called an activation function which gets its name from the biological neuron of you know a neuron activating or not like or some people call that firing right a neuron fires um it activates. So this this f is called an activation function and essentially what the activation function is doing mathematically is applying some nonlinear function. So it's typically the activation is going to be nonlinear in nature. It's going to be some type of function that is nonlinear. Okay. and it's going to um it's going to be applied to that weighted sum of inputs to that neuron and then produce some sort of output which is the activation value. Um it's you know producing some type of output. Um so in this case the neuron model this is the this is the model of the neuron is really just applying a activation to some weighted sum of inputs flowing into this neuron. Um so in this way it's kind of like the biological neuron where we had input signals flowing input signals flowing in and then some type of output signal coming out. Right? It it's basically that and then what the neuron is doing is doing a weighted sum through an activation. That's all that the neuron is doing is doing the weighted sum summation of weights times inputs. So you're have you have your signals X you generate an output which is um the weighted sum through the activation and you get this output signal uh Y here training the model with lots of data determines yes exactly so during the course of um doing the computations um the neuron will will be active or not active um it it's really not a matter of binary like active or not active. It's more of what is this value that comes out of this activation function? That's the output, right? What is that value that comes out of the activation? This is this is this value here. It's that weighted sum of inputs through an activation is really the model of the neuron, right? So, what is that output is really what we care about. The reason we care about that is because I want you to think about where this is headed in a in a network. So in a network of neurons, what we're going to have is a collection of these guys. So we're going to have many more neurons that would fill up a layer of neurons. So this is just one. So we'll have a lot of these guys. And each one of these neurons could produce a collection of outputs, right? they could produce a collection of outputs um from their layer and uh um so and then what you what happens is these outputs become the input to the next layer. They the signals keep flowing, right? They go into the next set of neurons, right? And then those um it what actually happens is you start to build this network. So not only does this go like this, but there's many more connections. So like this will become an input into this. This will be an input into this. This will be an input into this. And this will be an input into this. And same thing over here like this will be an input into this. This will be an input into this. This will be an input into this. And this would be an input into this. So everything from the previous layer which is coming out of that neuron all those neurons will flow forward into the next layer of neurons and on and on and on however many layers you might have. So you can have a significant amount of these layers of neurons in a really deep network. You might have lots and lots and lots of layers. So we're going to study that. We're going to stud study that as we go along. But that's really what we're building up to. Yeah, that's that's what we're building up to in a neural network. That's why it's called a network. It's really laid out in this structure of neurons sending data forward into the next layer of neurons and on and on and on until you eventually reach a final output. Now, I want to give you a basic use case of this neuron. Yeah. Yeah. That's really what it is. Interconnected layered outputs. Yep. I want to give you a basic use case of this output because it may be um it may be a little vague on what that output is. Um most of the time for a neural network this output will generally be a probability. So this will generally be some type of probability especially for a classification. probability will usually be a probability. So especially for a classification. So let's say um let's say we were predicting uh if it if an image like with with a brain scan is cancer or not cancer. This would be like a probability of this would be like a probability of cancer that we generate like in a binary case, right? Or spam or fraud, right? Generally represents some type of probability. It doesn't have to like if we're predicting a regression type of problem like let's say we were predicting the price for our housing price data. This this could be a price, right? This could be like a full value uh some some floatingoint number price right that that uh that could be possible but it's a single output from this neuron that could represent um typically for classifications be a probability but could be a price like in a regression problem. Okay. So again just to recap this because I think this is really really critical to understand moving forward in neural nets is what does a neuron do in the network? Every neuron is processing a weighted sum a weighted sum of inputs w2x2 wn xn. It's a weighted sum of inputs through an activation function f. That's what every neuron is doing. And that produces this output value from the neuron, which is that activated value, right? Some nonlinear function. And by the way, when we train the neural net, what we're actually learning during the training process, what gets updated during the training against real data are these weights. These weights get learned similar to when we do linear regression, right? We learn all those betas. Same thing here. We learn all of these weights. Now, do you guys see how if we have a really big network, there's going to be a lot of weights. So imagine we had a really big network with lots of neurons and they're sending data like there could take take an image which is um an image could be like uh a th00and by a,000 resolution. Um that's so a th00and by a,000 is going to be our input. Um, and then essentially what we're going to do is take those, pass them through this network, um, and generate some type of classification of like cat or dog, some probability of like might be cat, it might be dog, might be giraffe. Um, so maybe the output's actually three. So we have three final outputs like cat probability, dog probability or giraffe probability. Um, but there can be a lot of weights because we're going to end up with this really massive network where everything is connected together, right? So there can be a lot of neurons, um, a lot of weights that are that are connecting layers to each other. So these networks can grow massively with the amount of weights that have to be learned. Um now in this simple model of a neuron there's only three weights um only three weights that we would have to learn. So pretty simple but in like a chat GBT there's 100 billion weights that would be pretty conservative actually um their opensource model their transformer network open source GPT is around 20 billion weights okay so just to give you an idea there's 20 billion weights that would have to be learned So you can see why substantial resources would be taken to train a model like that. There's going to be 20 billion of these things that have to be learned. But this is the most basic form of neural net computation you can possibly have. Just a single neuron that's taking a weighted sum of inputs and passing it through an activation. So these inputs could be features, they could be pixels in an image, right? Many different things. Um but you're just taking the weighted sum of those passing through activate passing it through some activation um and generating an output. One of the questions that we should have at this point is what is the activation? It it the activation is actually something that we can change. We can set what we want it to be depending on the problem. So some activations are going to be going to be pretty popular in some cases. In other cases, we'll use other activations. We'll learn which activations. We'll actually talk about it later in the lesson. Um what this function usually looks like um in terms of taking that weighted sum, passing it through some type of function. That function might change from neuron to neuron in the network. Um but uh right now we're just keeping it generic as some function some nonlinear function that this weighted sum goes through. So in that case you know it's different than a regression. A regression would just take this and be done with it. Right? It would just take this weighted sum and that's it. That's the that's the answer. But for a neuron there's one extra component to that which is taking that and passing it through a function [clears throat] right um so so a little bit more involved there so uh everything I just said um is kind of depicted here where we are taking the um the the weighted sum. Now there may also be a bias. I forgot to mention that. Um similar to what we had in linear regression, right? There could also be a bias. So what we're really doing here is taking some function. So this this may represent the function that we're doing some step function um and generating an output um which kind of mimics the biological idea of an activation of the neuron. Right? So if the signal is big enough, if it reaches some mathematical threshold, um then the output signal is generated, meaning that neuron is activated and there's some some output that's generated. It's going to be the same here. We're going to be taking a weighted sum and then passing that through some activation f. Um so uh basically what we could do is think of this as some intermediate value Z is the weighted weighted sum and then we're taking the activation of that Z and generating uh an output value Y [clears throat] right which which the activation is going to be something like this uh maybe a nonlinear step function maybe an exponential um it just depends it depends on the situation we're going to use different activations So that is the artificial neuron model um weighted sum through an activation. That's actually it for the neuron. So some terminology in terms of the difference between the artificial model weighted sum through an activation um and the actual biological model or the biological neuron. Um the nucleus is known as kind of the node which is the node is just this um the node really is just like a a graphical kind of node here that uh we could imagine as part of a network. So a network's going to have lots of nodes uh to it. Um we have our inputs which are coming from usually a previous layer in the network. Um but you know in this in the case of a simple neuron like this think of it like our features right our features like our our bedrooms bathrooms um square footage from our housing data things like that. Um we have weights which are kind of like our synapses in the biological sense and then we have axon which is the output. So there are kind of mirrors to the biological definitions. Okay. All right. Let me go let me go back to this. Any questions on this? This is really important to understand. Again, weighted sum through an activation. That's the model of the neuron that we're going to take and run with for the very first network we're going to define, which is the perceptron. basically is this and again the weights will be learned during the training process. So the weights will be adjusted um so that we get um good outputs when we use our labeled training data, right? Um so if we want to end up with with cat or we want to end up with dog, we need to go back and make adjustments if we have errors there to these weights. And there's going to be an algorithm that dictates how much we should update these weights. Um which is going to be the, you know, gradient descent essentially. We'll talk more about that. All right. So, let's expand on this idea from a single neuron into a network of neurons. So, a neural network then is just the extension of this idea where now we have interconnected layers of neurons which is basically our computation model for the overall uh brain. basically a whole network of biological neurons working together. Um uh each um each network consists of uh basically an input layer and an output layer and then a bunch of potential layers in between. So you can have as many hidden layers. The these layers between the input and output are known as hidden layers. And there might be many of them, potentially many of them. It depends on how deep the network is. A really deep network. Depth is really based on how many hidden layers you have. So if you have very few of them, it's going to be a shallow network. If you have a lot of them, that's going to be a deep network. So this each one of these blue circles is a neuron. So this is a neuron and you have many neurons connected together but every neuron's doing the same thing. It's taking this weighted sum of of of inputs and passing it through some activation and generating an output right that um goes into the next layer. Okay. So just to go back to this definition again the network is this whole thing. So, it's all of these neurons connected together and a a collection of neurons um that are kind of forming an input prior to another layer. Um it this is basically one layer, right? This is a layer of neurons and there and when we build our networks, we can actually control how big that layer is. So, how many neurons are in there? we can actually specify that when we architect our network. Um that's something we'll be able to do. Um and how many outputs do we want? So if we're the outputs are usually going to be relative to how many classes are we predicting or how many outputs do we generally want. So in like let's say we're predicting between is this image a cat or a dog. Um we we would only have these two outputs. So this would be probability of dog and this would be probability of cat let's say um or if it was like uh if it was um if we had a giraffe then that would be a third member of the output layer so I would need to generate a third for a giraffe right or in the case of imageet they needed like in the imageet remember they're predicting a thousand different categories of items So they need an output layer that has a thousand neurons in it. So they need a thousand uh neurons in the output because they're predicting a thousand potential categories of images, right? So that output layer should match what your final prediction is going to be. So if you're predicting 10 categories, you need an output layer that has 10 neurons which are going to represent the probabilities of each class. If you're predicting a regression, you only need one output like the price or the temperature or whatever it is, right? You only need that one output. Or if it's binary like fraud, not fraud, cancer, not cancer, you may only have one output, which is the probability of uh of of the um like cancer or fraud or whatever it is. So how does a neural network process data and make a prediction? So it's kind of what I've been saying. Inputs go through the very beginning layer and then um every neuron that's in that layer receives inputs, assigns a weight, passes that through an activation to generate an output. So every neuron generates an output and the outputs from that first layer are then forwarded to the second layer for processing in the same exact way. So you take a weighted sum of all those inputs from that layer, generate an output, pass that along to the next layer and on and on and on. So this network um you know could have many many many layers. So there could be another layer here that's another what we would call hidden layer which is anything between the input and output and what we should realize is there's going to be contributions from every neuron that flow into each one of these neurons right so there's a lot of connections there every connection is weighted right there's a weight every every connection contributes some weight to that neuron and we process as usual we process process a weighted sum through an activation of all of these guys. It's just that their inputs are going to come from this previous layer. Hopefully that makes sense. The inputs are going to come from the outputs of those neurons from the previous layer and on and on and on. So in this way data we would say is kind of flowing forward or what we would call forward propagation. we forward propagate because it's it's literally like a propagation like a signal propagation. So forward prop or forward propagation because data is flowing from input all the way to the output going through all these neurons. So that that's how we make a prediction is we forward propagate inputs through the network and generate an output. Right? So that's known as forward propagation is sending data through generating a prediction u which could be a prediction could be probabilities right there's 80% probability this image is a cat 20% probability this image is a dog so therefore our final prediction is um cat for instance so this could be 08 this could be 2 and this would be 08 probability it's a cat and 2 probability it's a Right? Something like that. And we we're generating that by taking our image and flowing it through forward propagating it through this network generating these probabilities. That's just an example. So, so this term is important to know forward propagation. That's just sending data forward through the network. um which means that it goes from layer to layer to layer eventually to the output um where every single layer contributes their outputs as inputs to the next layer uh beyond it. Right? Okay. So where we're going to start uh with in terms of these networks, the very first one that we've actually already been studying is the perceptron. So the perceptron is what is exactly what we've been studying. It's just a model of a neuron of a single neuron. Um we will graduate from that into multi-layers of neurons. So um multi-layer perceptron. Um so more than one layer, more than one neuron for sure, but potentially more than one layer. And then we'll graduate from that into deeper networks that have many layers to them. Uh and then and then once we do that, we'll be in a good position, especially once we learn how to train these guys. Um we'll spend a lot of effort on training these and optim optimizing that training process. Um and building it within our code like PyTorch or TensorFlow, we will then start to apply variations on these architectures that involve convolutions. So the neurons will go from simple weighted sum through an activation to a completely different type of computation that the neurons are doing which would be like a convolution. So in in the um just to go back a step to put this in perspective in this picture um we're doing what we would call perceptron style calculations right now meaning we're just taking a weighted sum passing it through an activation and sending that forward through the network. When we get into computer vision and working with text and RNN's, these neurons will now be way more advanced. The neurons will do things like convolve or do convolutions which will be um a very sophisticated computational operation that will require a GPU or else will take forever to train any models. Um, so we will graduate from these very basic neurons. So right now we're just talking about a basic neuron that's doing this weighted sum through an activation and we'll graduate into convolutions and then we'll graduate into RNN. So doing recurrent calculations uh which mimic kind of sequence memory um for for learning how to work with text sequences like sentences. Um, so a lot to cover obviously before we get there, but right now we're starting with the most basic model of a neuron, which is this perceptron kind of style calculation. Uh, just sending a weighted sum through an activation. That's it. But eventually these neurons will become more and more complex. Um, like doing convolutions for image data or doing RNN's for uh text data. and eventually transformers. Um so so these neurons will do even more advanced things like attention in a transformer. Talk about the uh you know the perceptron um being the simplest type of network. That's where we're going to start. It's mainly used for binary prediction. So because like we can just produce that one output which could represent a probability. So the perceptron was the simple model of the neuron and it was originally used for binary classification problems. So that's where we'll start is the same thing kind of for binary. Um and the perceptron's really only good if your data is linearly separable. We'll talk about what that means and why that's really only the case. Otherwise we need to go to more advanced like more layers. So it the perceptron is going to be kind of limited in the sense that it's a single neuron. It's not a network. So it's not going to model too many problems that well. Um but it is the foundation for building on that and generating many many layers of neurons um that that we will see for larger networks. So, and that's where we go with the MLP, which is multi-layer perceptron. Um, basically multiple layers of these uh perceptrons of these neuron models. Um, and uh it can handle a little bit more complexity by learning nonlinear things. Um, because it has multiple layers passing those inputs through these nonlinear activations, it can learn more effective. However, the MLP is not necessarily a deep network. So, it's not going to have that many layers of perceptrons. Maybe only a handful. It's not going to be that deep of a network. Um, so itself will be kind of limited and it especially won't work well for like images or text, but for maybe certain problems like our housing data might work okay. But we'll certainly graduate from perceptron to multi-layer perceptron as a little bit bigger of a network. And then we'll work our way up to deep neural networks which are going to have many hidden layers. So this we we've had this image before. Here it is again where you know a deep network's going to have um this is the depth here. How many hidden layers do you have? A deep network is going to have many many many layers. Okay. So, um again like modern generative AI neural networks are going to have thousands of layers, thousands of these hidden layers here. Um and and notice how they're all connected because you know the inputs the sorry the outputs of one layer become the inputs to the next layer. So you can see connections from each of these um contributing to that neuron. This neuron is taking a weighted sum of those and then generating an output and then there's connections from that output to every other uh neuron and on and on and on. Right? So this big network you get um a certain number of outputs uh a certain number of outputs that um mimic what you're trying to predict. So again there's like this could be a thousand for the image net problem. Um let's say we're doing cat, dog or giraffe. This would only be three. We're sending an image through forward propagating it generating these probabilities which is our prediction. [clears throat] That's kind of how it works. Is it always left or right or can it bounce back? So it is left to right for the networks that we're talking about. The only um time you get the circle like is a recurrent neural net and RNN that is the so the circle you're describing is the essence of a recurrent connection. So it has the ability to do to do this which is kind of like uh mimicking a memory of a sequence sort of like going back and and remembering what our previous uh inputs were. Um, otherwise it's going to flow forward. It's going to be forward propagation left to right. But in an RNN, you would get connections like this, like you're thinking about. I think that's what you were talking about, those recurrent connections. We would get those in an RNN. Yeah. So, not Yeah, we won't have those yet. Not until we get to RNN's later in the in the lessons. Really good question though. So, and then the the the so we'll we're going to study this. We're going to work our way up to this and then as I said once we learn how to build these guys these deep neural networks and really work with them in PyTorch and TensorFlow we'll graduate to CNN's the convolutional neural nets which was um this was um motivated by mimicking kind of like the visual reception of the of the eyes and the cortex um using the convolutions to uh to accomplish Uh convolution being a um a type of computation will basically replace those neurons with more advanced computation. And the um visual reception is kind of mimicked by the convolution allowing us inside of images to pick out things like objects, things like shapes, edges, lines, um human uh features like faces, allowing us to learn those features of an image. Um so the convolutional net, again, this was that Alex net 2012. um really revolutionized being able to learn uh how to predict on image data. So we will learn about CNN's after we cover uh deep neural nets. Uh it stands for I'm not sure what it means in this case actually. Yeah, not not 100% sure. I'd have to uh look that up. I think they're just different um sections of the brain. I think they're they're short for like different uh sections that process like uh visuals. There we go. Yeah, they were different regions. Inferior temporal cortex. Nice. Okay, that makes sense to me. Okay, so then talking about RNN's really quick. Um, so RNN's are a lot, you know, networks that we're going to study later on in one of our lessons after we cover deep networks that will handle sequential data, right? So this is going to be things like language because as I said, that's a sequence of words one after the other in a in a sequence. Um, so we will handle text with RNN's. Also um a a very popular use case for RNN's especially these days is like forecasting. So time series data like uh sales or revenue day by day by day. You can you can build forecast with neural uh RNNs um because they're really effective at sequence learning um mainly because they um allow for these recurrent connections. Uh so they kind of have this short-term memory which is effective for some language tasks um and for forecasting. Unfortunately that short-term memory is is too short. It's not able to um learn the type of context that we see in like a a chat GBT for instance with a very long prompt or a lot of text which is what we see in transformers. So RNNs is kind of limited relative to transformers on language tasks mainly because that short-term memory is just not nearly what it needs to be to generate text or do effective translations, things like that. So we're going to learn about that when we cover RNN's. Um, one of the things that I mentioned that's certainly true is when we uh work with neural nets, we will be able to customize basically everything about that network. How many layers there are, how many neurons are within each layer, what those activation functions are. So you can think of like, you know, this is [snorts] going to be completely customizable from the same with input and output. How many uh inputs are are there? How many outputs should there be? Like if I'm predicting three or 10 classes or maybe two or maybe just one in a binary case. Um basically everything about this is going to be customizable when we build our models. And and that's just what we would call the architecture of the network, right? How many layers there are, how many neurons there are within each layer. And you can see that like when you have a deeper network with a lot of different uh layers here. So like this example on the right, you can see this is going to have a lot more um this has a lot more weights to it that have to be learned. So it's going to take more data. It's going to take longer to train a model like this because there's so many more connections, right? there's so many more weights um that have to be learned during this process um than than if you had fewer layers. So um that's a trade-off, right? Maybe you need more and more layers to learn a really complex problem that may be necessary, but you're going to pay the price when it comes to computation because there's just going to be more weights that you have to keep track of, right? and and learn. Um, so this is something we'll see when we get into TensorFlow and PyTorch. We're going to see how do we build this? What's the code that generates this kind of network? That's something we need to learn, right? What's the code that does this? That's something we're going to learn. It's it's actually going to be relatively easy inside of TensorFlow and PyTorch. They make it easy because those libraries are so good at building it. Um, so not only that, like we'll control the architecture, but we'll also control the training and the evaluation. So all of that we'll be able to set up um inside of our our, you know, PyTorch or TensorFlow when we get to that. Okay. So one of the things that's true about neural nets is they typically require more data. And um there's uh kind of a graphic here to show that like if you have um even a small neural network that's very shallow. So small being like number of hidden layers um you know it can still outperform traditional machine learning. So think of like logistic regression, decision tree, random forest, xg boost. It can still outperform all of those. um it's just going to require more data and the larger the model, the more data that you're going to need. So you can kind of see that on this graph, right? Um the large neural network has a lot of potential to perform really really well. Um but it needs the most data. It's the larger that the network is, the more data it needs to to function properly. Um [clears throat] now these are relatively close. So just a shallow network isn't and especially in the small data um down here it it's not until this point that um the the even the shallow neural network kind of surpasses traditional machine learning. So again for small data um it may be desirable to just continue to use traditional techniques like what we've studied but as we get larger and larger data size um if we're working especially with unstructured like images text we're going to need neural nets and um you know usually we'll start small and work our way up to larger neural networks if we need them you know in terms of the architecture. which is how many layers do we have, how many neurons do we have, those kind of things. Okay, so the the depth of the network like how many layers we have is often um relative to the task. So if we're doing, you know, working with images like in an image classification or like object detection or like in a in a transformer for a generative AI use case, that's going to be a much bigger network than maybe a simpler modeling task like predicting the price on our housing data where it's very structured, has a certain number of features to it. Um we may not need as deep of a network. Um so this is again where we will uh we will you know practice building networks and seeing when that makes a difference like how much we change that architecture uh how big of a difference does that make on our modeling problems. We're going to see that as we get into it. [snorts] But here's a good, you know, this is a good picture here on the uh this one has two layers and it produces two outputs. Um so it takes four input features, produces um two outputs which could be probabilities, let's say. Um and you can see like these these symbols are exactly what I've been describing as you take a weighted sum and you pass it through an activation. That's what the sigma and the f is. So the sigma is like the weighted sum of the inputs to that neuron. Every one of these circles is a neuron, right? So this is a neuron. This is a neuron. This is a neuron. Neuron. Neuron. Neuron. Neuron. And um technically all these other circles are neurons as well. These are just output neurons and these are input neurons. So they're not really doing much. they're just processing the input or holding the output. Um, but but they're going to be a weighted sum through an activation for all these neurons, right? Weighted sum through an activation. That's exactly what what this is meant to represent. Sigma weighted sum through the f through the activation. Um so let's talk about the perceptron uh as a a model that is based on that neuron that we've studied so far. Um basically one and the same. So what is what do we mean by perceptron exactly? So I've been using the word perceptron a little bit. Let's actually define what it is. Um essentially it is a very basic neural network that is designed for binary classification. So think spam not spam right fraud not fraud. So it has only two choices. Um and essentially what it is is the neuron such that it has a very simple activation. This is the activation right here which is that um you get a a classification of one if your weighted sum times your your inputs plus your bias is positive. So this has an activation function like this which is basically just checking is your weighted sum uh is your weighted sum times your inputs or weighted sum with your inputs um greater than zero. So if it's greater than zero, we're going to output for that neuron a one. If it's negative or equal to zero, we're going to output zero. So it's in that way, it's just taking that model of the neuron. So you're taking all of your inputs, your X, X1, X2, X3, X, XN, [snorts] however many you have, and you're feeding all of them into the single neuron, which is taking a weighted sum and uh taking a weighted sum of those inputs and then passing it through an activation, which is going to produce a a zero or a one, right? um a y which is zero or one depending on this activation which is um was it positive or was it zero or negative right is essentially what this activation is. So it's a very simple model used for binary classification that's basically just a neuron um just a single neuron um taking that weighted sum uh passing it through the activation. This is the activation activation function here which is basically just checking is it positive you get a one output. uh is it um negative or zero you get a zero output right for the for the neuron that's all it is so perceptron a very simple model um that is based on this neuron calculation the thing that's unique about the perceptron is it's just using this particular activation um to see you know do we get a one or a zero for a binary classification so what's interesting about the perceptron is um you know It's uh using the basic idea of a neuron in every sense. It's taking a weighted sum through an activation. Uh and what gets learned in this perceptron is the same thing that we've talked about learning in any neural network, which are these weights, right? These weights are going to be what gets updated through the training of the perceptron. We need to talk about how does that happen? What kind of adjustments do you make? Um but essentially we're the algorithm is going to learn those weights of that neuron um so that it can produce good uh outputs for your data, right? You want highquality predictions that line up to your your like spam not spam fraud not fraud kind of uh uh predictions. So it's exactly like this picture. uh it's just the output again is going to be a binary zero or a one and that activation is going to be particularly checking if it's positive or if it's negative or zero. So it's exactly this picture we just talked about from earlier with the neuron. Um for that reason you know the weighted sum through an activation is often just referred to as a perceptron style computation because that's essentially what it is. the perceptron is just using that particular activation of seeing if that weighted sum is positive. So it's that very particular activation. Um again I'll draw it here. um you know we we do this uh weighted sum and we're passing it through the the activation and the activation is such that um if this weighted sum is positive then you get um then you get a a one um if it's uh else basically you get a zero as the output right is that's the activation function there. So the things that need to be updated and learned to have a an effective perceptron is obviously these weights. These play a really critical role uh in determining what the output is going to be in any prediction. So again what does this look like? Um it's very much similar to linear regression. We talked about this earlier and for that reason this model only works really effectively when the data is basically what we would call linearly separable. So if our classes like let's say we just had two features X1 and X2 um if we had let's say these red points belong to one class and then these blue points um or let me draw them in green belong to a different class over here. If you know if we could draw a straight line through here and say everything on this side belongs to yes and everything on this side belongs to no those binary classes. Um then uh then this model would be effective because it's essentially going to find that line and compare it to zero. um essentially as if we shifted this line to be kind of at the origin and we're checking like does this result end up positive meaning it's on this side of the line or does it end up negative meaning it's kind of on this side of the line um which would be a zero. Um so the perceptron is really only good to be used because it's so simple just a single neuron. It's really only good in these situations where the data is kind of what we would call linearly separable. Um uh which is not always the case, right? That's typically not the case. Um which is why this model we need to expand this to have many neurons with nonlinear activations to model like real world problems. Okay. So again the perceptron uh we get a zero or one. We're going to adjust the weights during the training process. Um they get updated kind of in a gradient descent style uh calculation that happens to adjust those according to the error. Um and the perceptron really only works well uh when the when the data is linear separable as we just talked about. Okay. So it's everything here that we just talked about weighted sum through an activation. It's just this prediction is now we know what it is for a perceptron. We know that this is going to be a zero or a one and we know what this activation is going to be. It's it's the you know checking if that weighted sum is positive or negative or zero. Right? So we know what these components are now for the perceptron. So everything we've talked about is is here. nothing really changes except a particular activation for this binary case and it's just a single neuron. So that's the other thing we've been talking about a network of neurons. The perceptron is so basic. It's just a single neuron. All right, just a single one. It doesn't have a layer of multiple neurons. Um it's just a single one. So very basic. So the components there are again what we've been talking about. They we have a set of inputs. We have a set of weights. We have a bias using for shifting um which is important for checking relative to zero. Um we have a summation function that uh just takes the weighted uh sum and then we have an activation which is exactly what's producing our zero or one. Right? the activation everything is there that we typically will have in any neuron and certainly in a network of neurons we're going to have lots of these guys. So the perceptron is a good model of a single neuron um in the sense that it takes a weighted sum takes an activation and produces an output. Now what would happen with this that output in a in a larger network is we know that data would flow forward to the next layer right we know that um in a larger network but in the perceptron we really only just have this right we just have some weighted inputs going into that single neuron um but typically that data would flow forward in what we've said is forward propagation. Um another word for this for uh type of network where data only flows forward in a forward propagation manner is um sometimes known as a feed forward feed forward network. So, so feed forward is basically just a network of perceptron neurons, right? That that perceptron being um their computation is just a weighted sum through an activation and it's just flowing forward. The output goes into the next layer. That output goes into the next layer and on and on. So this is sometimes known as just a feed forward network because we do later on when we have RNN's we'll have the ability to have a recurrent connection that goes backwards. Um but right now you know these especially with perceptron style calculations in these neurons um everything here is just what we call feed forward. Um everything just forward propagates. nothing nothing has a kind of feedback connection yet until we get to RNN's. So this is another term you may hear is a is what we call a feed forward network. It's really just um whenever you hear the word feed forward network, you should think of perceptron style computations that just flow forward weighted sum through an activation going forward. That's all passing their output as the input to the next layer. So some of the features of this is that information only goes one way. Um it goes straight it goes forward propagated through never touches a node twice. It has no memory of the input um which is what we would have in an RNN. We would have memory because we care about sequences. We care about data that's prior to our current um value that we're processing in the sequence. Um so it h so the the feed forward network which is the most basic type that perceptron style computation data just flowing forward basically has no ordering no sequential memory nothing like that because it cannot go backwards just every all the data is um flowing forward. Okay. So what we're going to do now is just look at the extension of the perceptron. So given that we have a perceptron. Remember the perceptron is just a single neuron that has inputs. So we have inputs flowing into a single neuron which is generating a single output. That's perceptron is just this. It's not even a network really. It's just inputs uh weighted some of those going forward into an output. However, if we stack together multiple layers of these guys in terms of these neurons, we can generate what's called a multi-layer perceptron, which is starting to be a very basic network, right? It's not a full deep network in the sense there's not many many layers, there's not many many neurons. It's very limited. So multi-layer perceptron is designed to be um pretty limited um in terms of its structure. It's not a huge network. It's mainly just a small collection of maybe two, three, maybe four at most perceptron layers. Um and and we generate an output which could be one or two or three outputs. depends on what we're trying to predict. Um but uh it's think of just stacking these neurons together in a simple collection of layers and we're starting to build a network that is the multi-layer perceptron. Um now every neuron in the MLP sometimes this is known as MLP multilayer perceptron. um every layer in this multi-layer perceptron is just the simple weighted sum through an activation. So this is still every single neuron still functions the same weighted sum through the activation, right? It's it truly is that weighted sum through an activation. Um so nothing really changes there. Every neuron is like that. Okay. And all the data still feeds forward. It's just now we have multiple neurons and multiple layers, maybe two or three or four of them. Okay, so a perceptron um is decent except it fails at the separable uh things that are not linearly separable. So a very basic example people like to give of where the neuron model of a perceptron really falls apart is like a simple um what's called an xor which is uh the like the exclusive or operation between uh um like two binary values like minus1 and one minus1 and minus1. Um if you take the exclusive or between the two you can see what their values are. Like the exclusive or between minus1 and minus1 is still minus1. Um the exclusive or between minus1 and one is one. Um and then um what's interesting is you know this is not uh separable. You would have to in order to separate these two classes you would actually need two lines or basically something nonlinear. Uh because you would need um you would basically need to draw something uh kind of like this um in order to separate the two uh and say like everything on this side is uh belongs to one class and everything on this side belongs to another class and that's highly nonlinear. So th and this is just one example of a function that would break the the you can think of many more like you could have um like let's say um going back to the example we could draw circles here and here and then draw um the red [clears throat] ones um like this one could be on this side and this one could be on this side and again this makes it really difficult to draw a single line like you can't draw a single line through there and say, well, I can perfectly separate everything on this side and everything on this side. That's not going to be possible with with a function like this with data that's like this. Um, so that's just to say the perceptron is not perfect. It's going to struggle on data that is not linearly separable, which is what this means. meaning you can draw a perfect line or a hyper plane and all the data is on one side of it and all the other classes are on the other side of it for a binary classification. Um you you most likely need a multi-layer perceptron because you need nonlinearity, right? You need this this kind of picture here in order to better separate it which is which is nonlinear. So that's just to say a single neuron by itself is kind of limited. That's where you need layers of neurons to more effectively model problems. That's all we're saying here. Okay. So I have a demo for you guys and this is just going to use um so it turns out that perceptron a single neuron is actually available within scikitlearn. So we don't even need to use tensorflow yet or pietorch yet in order to just work with perceptron. Um, so there's actually just to show you that quickly. There is a a demo. So this 3.04 demo inside of um, so let's um let's practice building just a perceptron. Now again would we typically use a perceptron? No, we would typically use a a uh neural net that has more than just a perceptron which is a single neuron. So, but the perceptron is an important building block to larger and deeper networks because it that fundamental idea of taking a weighted sum through an activation is really critical and it's it shows up in a lot of networks. So, it it is an important building block but by itself is not necessarily an effective model especially in those cases we just saw where it's not linearly separable. But let's see how we use the perceptron here. Um what you can see is look at where it comes from. It's from scikitlearn in the linear model family. Now we've seen other things from there too like the uh logistic regression right we've also seen linear regression there but for classification we've seen logistic regression from that linear model. Here is perceptron. It's considered to be in this linear model family because we're taking that weighted sum which is linear but then passing it through an activation which is uh checking if it's positive or or negative, right? Um so let's load let's run that to do our imports. Let's load our data. So we're going to load that spam based data. I do need to put it in here. So that's loaded and we can check the uh head of it to see the first five rows of course. So you can see there's some uh some data. Now this the spam is is whether or not the email was spam or not. So it's got a it's all a bunch of data about the uh about the email. And then it has a classification category of spam which is the last column which is a one or a zero for not spam. Right? So that's that this is a classification problem. Uh and that and this data is just a bunch of features about different words. So for instance this is a word frequency of the word all the word address the word make the word hour over remove. So, it's got a bunch of word frequency features and then it's got some capital capital capitalization features, um characters like the frequency of the pound symbol, frequency of the dollar symbol, frequency of exclamation point, um parentheses. So, it's got a bunch of features about text uh within the uh email. Okay. All right. So what we're going to do is um just have a simple check here. If there are any nles, we're just going to fill those with zeros. So um obviously we could do a more robust check here. Um in fact, let's let's check uh you know, we know how to do data um isisnull and then dot sum. So we can see that across all the columns. Uh if we have any um doesn't look like we have any. So this yeah it looks like we don't have any nles. So this should be relatively uh this shouldn't do anything essentially cuz this should fail. There are none. So this should be false. It shouldn't have to fill in anything. So that that's fine. That's just going to produce our data. All right. And then this is the this is the important part. This is the important part of grabbing our features and our labels, right? So this is grabbing our features and our labels, right? Features and labels. So this is grabbing every row and every column but the last one. That's what I look does, right? Do you guys remember that for our data frame? This is grabbing every single row but then every column but the last. So everything up until the last which is the minus one index. That is all of our features. And this is grabbing the very last column all all rows. That's what the colon means. All rows. And then only the last column which is that's the label right? This is the lab the spam label uh column is the last and we saw that in the in in this uh head of the dataf frame right spam is the last one every other column before it is is a feature. So we are just separating those out. Pretty typical separating those out and then we're doing what we usually do with that is train test split, right? Pretty standard where we pass in our features and our labels and we do a train test split. Um and we can choose whatever test size. I think point4 is a little big. We could lower that to maybe 3 or 0.25 or 0.2. Um but I'll just leave it for that for now. Um, but this is our usual train test split. So, nothing that interesting here. Just generating our our training data and our testing data. It's pretty standard, right? Nothing that uh out of the ordinary there for a supervised problem, which this certainly is. Even though we're using perceptron, which is a neuron model to do it, still supervised. Let's scale our data. So, we're going to scale all of our features. Um that's uh still something we're going to do here. Um using this perceptron uh scaling all of our features here. So scaling our training features, scaling our test features. So we train both of those or sorry use both of those and then we can fit our perceptron. So look how easy it is to initialize a perceptron. Um and and uh basically the perceptron is um just a single neuron model, right? So that's all it is. And it's going to use um uh it's going to train. So this uh this ffit I should call this out. This ffit is going to train the neuron which is just the perceptron just a single neuron. It's going to train this by learning the weights through a gradient descent uh algorithm. So it's going to run a gradient descent in the background to learn what those optimal weights are. Now we haven't studied gradient descent in detail. We will as part of like when we get to the part on uh coming up shortly on learning how to train neural nets. We're going to learn a lot about this because every single neural net trains through gradient descent algorithm one way or the other. It does gradient descent. So it's not going to be any different for the perceptron because it is just a neuron. It's a very basic. It's not a network per se, but it is a neuron. The network is just a huge collection of neurons. So, um they're all going to train through gradient descent. So, we'll we'll learn more about that as we go. But this ffit is going to run gradient descent on our um our our features and our labels here for the training data. And it's going to the perceptron is going to adjust all of its weights um according to this data. [snorts] Okay. So we get a perceptron that has been fit. Um and then we can make predictions. So we can pass in our training data to get a collection of predictions for our training set. And then more importantly, we want to evaluate that on our test set. So we pass in our test features to our model and dopredict. So so far this is just behaving like any other scikitlearn model, right? Nothing that interesting. But it is cool that the perceptron exists inside of scikitlearn as kind of like any other model. Okay. So we can see the accuracy here um on our data and we get about an 89% accuracy on the training and about an 88% on the test. So pretty decent. That's not too bad um using that perceptron. So pretty cool. Um that works out and that's just a that's just a simple neuron. It's not, you know, this is a very basic model. Um, it's just that single neuron. Of course, where we're going with all this is to build up our own neural network that's going to be a huge collection of these neurons, right? That are going to be um organized into layers that are connected into other layers and so on and so on and so on. So, we're going to build up to that. And one of the things we're going to have to learn is how to construct that in like pietorrch or tensorflow. Um so that's going to be more involved. And we're also going to learn how to set up this training algorithm this gradient descent. We need to learn more about that. So that'll be upcoming as well. And we covered this but I just wanted to start here with kind of a review of what we had talked about in terms of uh neural networks. Right? So just to refresh ourselves, um we're dealing with uh neural networks in general and um we know that those are comprised of kind of a network of neurons, right? And and um these are organized into layers where every layer has a different number of neurons potentially that we could we could specify how many there's going to be. Um but the the idea is that the data that we pass in uh will kind of flow forward through this network and every neuron is really doing something like this. Remember this is and this is the equation that we kind of had for a perceptron which is that uh we take a weighted sum times the input. Um, and so this W star X is really meant to be um kind of a dotproduct like a weighted sum uh times inputs, right? So we have weights times inputs and then we um might add a bias in there as well, which is what the B is here. So adding in a bias, which is just another weight really. Um and but the important thing is passing that weighted sum of inputs through an activation which is this f right. So we pass that through some type of activation which generates the output for that neuron and what happens in the network is these outputs just flow into the next layer. So whatever is output from from this neuron becomes the input into this next neuron. it also becomes the input into this neuron and and this one really all the neurons that are in the next layer. And of course those have different weights associated to them. So there'll be a weight here, a weight here, a weight here that that dictates how much of that input um from that neuron gets contributed to to this neuron's uh output. And same with this guy, like this guy's going to contribute a certain amount to this. this guy's going to contribute a certain amount to this and there's going to be some type of weighted sum through an activation and that goes from every layer to every layer. Right? So, um that's what we've kind of learned so far. The things that we need to study a little bit and we will coming up next are what are these activations um in particular and when do we use certain activations. So in certain situations um you know a certain activation will make more sense because they can if you look at the activation from a math perspective they really control what the output of this is allowed to be from a range perspective right because you're taking us an input passing it through this function and generating an output. So depending on what that activation function is kind of controls what you can possibly get out of there. If you're predicting things like in a in a probability sense, you probably want some type of activation that's going to limit your range to a probability like between 0 to one. That makes sense. Um if you're doing a regression, you don't want to do that. You usually want it to be uh just any real number value, right? To to mimic like a price or temperature or something like that you're doing a regression with. So this activation is really important. Um, and we're going to talk about some very common activation functions that are out there. But this is what we're talking about with neurons, right? Weighted sum through an activation. And we're doing this in this network of neurons uh that are kind of spread out amongst these layers passing data forward throughout. And you finally get to some output. Um and even the output remember the output when we get into larger and larger complex use cases the output is going to be different depending on what kind of problem we're solving. So if we're um just predicting like a binary output that's going to be just kind of a single output that's a probability right between 0 to one kind of like what we saw for logistic regression for like a you know spam not spam fraud not fraud kind of use case. But if we're predicting maybe we we're predicting between three different kind of image categories like this is an image of a cat, this is a dog, this is a giraffe. Um we would we would likely have three outputs, right? We'd have probability of dog, probability of cat, and probability of giraffe. Um or in you know, if we had 10, then we'd have 10 outputs here. Um so our output layer should kind of match how many outputs we're trying to produce. Um that makes sense. And so we're going to see that with different use cases want to set up those output layers to contain the right number of outputs and also you know actually be probabilities which is going to be hugely dictated by this uh activation right we want the activation to generate probabilities for those neurons. Okay, so this is a little bit of a recap. And if you guys remember um perceptron is a a special type of uh neuron. It that is just a single neuron that has an activation function that produces a zero or one, right? It's it's like a step function that produces a zero or one. Um depending on if this weighted sum is positive, this is going to produce a one. If it's negative, it'll produce uh a zero. Um so that's what we meant by the perceptron model. It's just a single neuron that does this activation. Hopefully that rings a bell from last week. Um and and what we're really interested in doing with bigger and bigger neural networks is basically having many perceptrons, which are basically all of these models of neurons here. So you have lots of perceptron style uh calculations and that's where we we graduate from just a single neuron to multiple layers of perceptron which is which is this MLP model multi-layer perceptron model here. Um so that's that's kind of where we're headed and then this is going to keep expanding. So we're going to have even more layers, even more neurons, and then eventually even more complex computations there that are not just weighted sum through an activation, but things like convolution, things like recurrent layers. Um, so we're going [clears throat] to work our way up there in the future, but uh that's kind of where that's where we're headed with all this. So, just wanted to recap that a little bit um and pick it up from there, especially the the activations. I think that'll uh we'll talk about that next. So, we we ended on this demo last time, which was um kind of going into scikitlearn. If you remember, we did this demo went into scikitlearn and scikylearn actually has the basic perceptron um model. Um but of course when we build a more complicated neural network it's going to be in tensorflow or u pietorch but um scikitlearn itself does have um a perceptron um because it's it's a very basic model of a neuron um but it's not really a network right it's not like a full network it's just kind of a single neuron. Let's talk about activations then. So uh we know what the definition is uh in in terms of it is a function that we're going to apply to that uh inputs that are um generated for the neuron where we have that weighted sum uh with weights times inputs and that generates this value that we're going to pass through this function calling that function the activation function which gets the name activation function really comes from biology and and kind of the actual ual um workings of a neuron which uh gets activated if there's enough kind of energy there. Um so same kind of thing here if if the input is enough we run that through a function and we kind of generate this uh output of it through this activation function. Um and so that's where it gets kind of its name. Um now in terms of the perceptron um we know what the activation function is. It's it's actually um a very simple function which looks at uh looks at this weighted sum of inputs. So it takes this um weighted sum. So you can see here's like an input one, input two all the way up to input in. We wait those. So there's a corresponding weight for every one of those inputs. We know that. we total that up and maybe even add in a bias. So you can think of an extra term here. there's a bias that get uh contributes to the sum typically and uh then this goes through this f right this f ofx um activation which is taking the f of that uh weighted sum uh that is all of these guys right so it this activation function is being applied to those and really in the perceptron it's a simple function The f really says that if so that the f um is such that um it has a definition such that like if um this weighted sum is positive uh then we produce um if this sum is positive then we produce uh one um else we produce uh zero else being you know it's negative negative or zero, then we produce zero. Right? So, the activation for the perceptron is really really simple. But that that's not the only activation that's out there. And in fact, when we start building our own neural networks, um there's going to be many different activations we can choose from. Some of them are going to be more um popular than others, and some of them have very particular use cases. As I said, one thing you want to pay attention to is this activation really helping us kind of narrow down the range of what the output of that neuron could possibly be. So like in the perceptron case, the output's only allowed to be a binary one or a zero, which is good for certain problems, right? It's good for like a spam not spam cancer not cancer, but it's not very good for regression at all, right? It wouldn't be useful at all for regression. wouldn't be useful at all for um generating a probability even because it's still structured to just zero or one. Uh and it certainly wouldn't be good for multiclass where we have maybe 10 different categories of things we're predicting. Um so so the perceptron activation is very limited. Um but we're going to explore some other ones that are found in a lot of lot more general neural networks than just perceptron. But but you see what its purpose is really to produce the output of a neuron um passing in those inputs through this activation function. So it has that capability of kind of um mapping those to a particular range which is important in terms of what are we trying to get out of the neural net especially as you get towards the the output layers right then you really care about what your final output is. So if you think about it, if like what would happen if we did not have an activation function. So just think about that for a second. If we didn't have that um then in theory like you know depending on what those inputs are, we really could get anything from this weighted sum with with the weights and inputs, right? They could be any real number. That's all this is saying is like you get kind of an infinite range of possibilities. It's not really limited to anything. um which is fine for some problems like a regression that's okay but that is very um very open-ended and not very good for other problems like like if we want to generate a probability this is really bad because we don't want things to be really large or really big actually want it to be confined to be between zero zero or one right coming out of the kind of final neuron so having an activation helps helps us kind of bound that output coming out of a neuron because if you think about it, this weighted sum could be anything. Um, and actually even in a regression, it's probably not good that it could be anything negative or positive. Like if you're predicting a price, you probably want that to be positive. So even minus infinity infinity is probably not even good enough. You probably want it to be restricted to between zero and infinity at the very least. So that's where the activation comes in. We apply a function to this and that helps us kind of map that to a potential range. Like maybe we map it to 0 to infinity. Um so we so for like a regression that'd probably be good uh for things like price, temperature, whatever. Um or maybe we map that to uh 0 to one for like a probability. um that that would be good um for that case. So, it just depends. It just depends on what problem we're solving. It also depends where we are in the network. Usually for things towards the end, we want to make sure we're using an activation that lines up the output to be in the appropriate range. Like if we're producing probabilities, want that to be between 01. um not you know all over the place like negatives positives uh which just the weighted sometimes inputs could be anything right could be anything um and again that's where the activation function kind of comes into play it's applied to this and allows us to kind of map this uh to to get a certain output from the output range at least for that neuron. Um so for that reason activations play a really critical role in in any neural network and as I said especially towards the end of the network you know the example I always go back to is if we're predicting like three different things like um dog cat and giraffe we don't want those numbers to be all over the place we really want them to be probabilities that add up to one actually so across this kind of out let's imagine this was our output layer we would want to make sure that these are all probabilities that are between like uh zero. So probabilities in 0 to one in this range. So we'd certainly want that to happen. We'd also want all of these to total up to one. We'd want them to be normalized in some sense, right? So um each one of these outputs should um total across the output should equal one, right? That should happen if they're if they're probabilities. Um so that's something we're going to be interested in doing. And what we would hope is there's an activation we can apply um across this entire layer really such that um all of these uh are probabilities certainly in 01 and total up to one for those kind of cases. So we're going to see activations that actually will do that for us be able to normalize that for us which is good. Okay. So, and the other thing about the um activation function as well is the the activation function actually also serves a really important part of the model um because it allow it basically allows us to introduce um nonlinearity because if you think about it like if we just had let me go back a couple slides to this. If we just had this this is basically a linear model, right? Just like this. This is a linear model which we've seen before. It's just a line. This is just a linear regression. Linear regression, right? That's actually all that that is. It's just weights times inputs plus a bias. We've seen that before, right? That's all that that is. So without activations, we essentially have just a huge collection of linear regressions that are um connected together. So activations actually play a really important role in introducing nonlinearity which is really really important for a neural network to be able to learn complex patterns. So activations are actually really critical. So it's having this f here this activation allows us to learn more general patterns than just linear ones cuz if we didn't have that that's all that we would have is just linear relationships all over the place right throughout the network. Um but having an activation function here that and the activation function will generally be nonlinear. Um it will gen generally be nonlinear. Um we're going to get into some examples but like some of them are going to be uh like the sigmoid like from logistic regression like a logistic function. Some of them will be hyperbolic tangent is a popular one. These are all you know highly nonlinear functions. There's also exponentials that are possible. Um, so, uh, different types of functions that are all nonlinear. That's actually a really big key, uh, because we want to introduce that in the learning process is the ability to kind of learn nonlinear relationships. That's another reason why they're important. They limit our range but they also introduce nonlinearity so that our network can learn those nonlinear patterns which is really important. All right so let's talk about some of the most popular activations. Now by no means is this going to be um an ex exhaustive list. There's definitely more than what's just on this list but I'm going to go through these because these are going to be the most popular that we deal with in this course um for sure. you know, there are other ones that are out there that are more advanced. Um, but I would say they rarely show up for the use cases that we care about. Um, the ones the ones that we care about are mostly going to be what's on this list. Um, and I'm going to explain what all of these are and kind of what they look like. The thing that these all have in common, by the way, is they are all nonlinear functions. All of them. So that's really that's a really important attribute of an activation is generally nonlinear. Um and again we're going to have these spread out you know across all the neurons and um so so in that way we're kind of like stitching together all these nonlinear relationships which is which is um how we learn complex patterns. Um but let me go through these. So these are some popular ones. We've actually already looked at this one a little bit, but let me I'm going to go we're going to start with that one and then we'll talk about these other three. The sigmoid relu and softmax. We'll talk about all those. So these are some popular activations. We've actually already looked at step that's the one that's involved in the perceptron. Um let's start there. So the the reason this is called a step function is because it essentially only it has one step up from a zero to a one. So if you look at it on a graph, which I'll do on the next slide, it's a function that um goes from zero up to one and it's just a single step up uh one unit from 0 to one. Now this activation is kind of special to the perceptron algorithm because um that's just the way it's designed, right? Is that uh whenever that weighted sum of inputs that is being activated, right? that that data that's that weighted sum going into this function whenever that is um positive um we get a one as our result um otherwise we get a zero whenever that weighted sum is negative or zero um so we we get zero so in that way it's it's uh let me show you it's kind of a step function that looks like this so we this is inspired by activating a neuron Right? So if you have a single neuron, it gets kind of activated when there's enough energy. In other words, it's bigger than zero. Um, and this is why it's called a step function is you can see on the graph, it literally looks like a step like like on the stairs, right? It goes up and then over. So what this graph represents is, you know, this axis is the input and the y-axis is the output of course. And what you're looking at is for anytime the input is negative. So you're on this side of the x-axis. Um you get zero as your output, right? You're you're down here. You always get zero. No matter what that is. The moment you have something positive, meaning you're on this side, um you get a one, right? You get a one for all those. No matter no matter what that is, as soon as it's positive, you get a one. So you get that step up to one. um and and that p that neuron is essentially activated, right? And we get a one as a as a result. Um so so this activation function kind of looks like this. Now that is nonlinear for sure. Um but what we also see is the range of this is very limited to 0 to one. So the this step function activation is really only useful for this perceptron when we are doing a binary classification that's going to be zero or a one. Right? Really only useful for that because we're not able to produce any other value in between, right? We only get a zero or one. We also like for regression, we'd be out of luck because we can't produce anything else. Um, so definitely wouldn't be a good fit for any of those kind of problems, right? So that's the step function. Again, this this one's only going to be used for the traditional perceptron. That's the only time it's really ever used. um is in that case. Okay, before I go on to the sigmoid, any questions about the perceptron step function? Like I said, it's kind of limited to that use case of the original neuron model when people were doing binary classification with that neuron. Um but again, um we will often use other kinds of activations. I'm going to show us some of those uh coming up next. So let's talk about the sigma a. Now this is one that we have seen before and it's actually the same function that we have seen with logistic regression. Um exactly the same function with logistic regression. So the sigmoid produces um a probability between 0 to one which also makes it useful for binary classifications. Just like the step function this is helpful for binary classification. And the difference is the sigmoid um allows us to produce a probability that could be anything actually between 0 to one. Right? So in that way we're actually generating a more direct probability rather than just a uh just either a zero or a one final result. um which we know from logistic regression we did the same thing right we produced a probability and what we did is we said okay if that probability is big enough meaning it's above 0.5 we categorize that as a one if it was down here we categorized it as a zero so we did that but the sigmoid activation allows us to produce a zero uh a probability between 0 and one as the output of a neuron. So remember, we're still applying this function to a weighted sum of inputs plus a bias maybe. And um now this function is the sigmoid is what we're saying is called the sigmoid um and this is going to map whatever this is um to between 0 to one. Some some probability we interpret that as kind of a probability between 0 to one. Um so where is this useful? This activation function is mostly used for binary classification as the output neuron. So if you're in the network and so let's say let's draw the network. So let's say we have this network and we have maybe we have a hidden layer here um and then we kind of produce one final output. Um what we would if we're doing uh binary classification like spam not spam or cancer not cancer what we would really like this activation to be is the sigmoid. This this guy should probably be a sigmoid because what what we want to do um what we want to do is basically produce a probability from this weighted sum of these guys coming into this. Right? So we're going to take this weighted sum and uh generate a probability out of that. Um so we want this final result to be a sigmoid. Um now these these others in the network don't have to be sigmoid activation um to produce their output. Um that's fine. We're going to look at other activations that are more typical for kind of the middle of the network that produces a general value. But um sigmoid is really really often seen at the very end when you want to produce a zero or one. Generally you don't want to produce a zero or one kind of in the middle cuz that limits your range, right? That limits cuz like then this this neuron would be very limited to a probability. This neuron would be kind of limited. This would be limited. We don't really want to do that in the hidden layers of the network. We don't want to limit ourselves because then we're not going to learn very effectively. Um but as a final output for a binary classification problem, yes, we want that to be a probability between zero and one. So sigmoid works great. Um remember the the function for a sigmoid is what we saw in logist logistic regression. It's this um fraction that looks like this. So it's it's uh definitely nonlinear, right? Definitely nonlinear. You can see it from the graph. Um but certainly you know this produces uh this produces um values between 0 and one. When x gets really big negative um this fraction in the bottom kind of grows exponentially big. When x goes this way it grow the fraction grows exponentially big in the bottom and makes it one over something really huge. Um which is basically zero. So that's why it decreases down this way as the x gets more and more negative. And then kind of the other way around is like if x gets really big positive and you go this way um you get closer to one because this term basically decreases down to zero um the larger x gets. So you get one over one. So you get something closer to one. So hopefully that graph makes sense like based on that function. Um but that's that's what the activation is. And and again this is the same that we this is some this is called the sigmoid. Sometimes you'll hear it called the logistic function logistic function. Um it's the again it's the same one that's kind of used in logistic regression. uh in logistic regression remember this it's basically this but we have um we have uh we have something like this beta we had our beta iix xi we were doing that um so we essentially have that same thing uh for this activating that that neuron so sigmoid we'll sometimes see now let me show you the most popular uh activation And the reason this is the most popular is because it's the one that's used almost exclusively in the hidden layers of a network. The reason is is because it does not really restrict the values. In fact, what this this uh activation will just basically return back to you the input assuming that it's positive. So as long as it's positive, it's just going to give you back what you put in. So see how this function gives you the maximum between zero and x. So what that really means is like as long as x is positive, what's the max going to be between zero and anything positive? It's going to be whatever that other what that positive number is, right? So this will always just return to you what you put in as long as it's positive. However, this is a nonlinear function because it has this piece right here, which is really critical. This piece is saying like whenever we have something negative, it's actually going to chop that off and just give us zero. So, this is actually discouraging negative outputs. It's always going to give us something that's either zero or positive. Right? So this at this activation function is known as a rectified linear unit um because this y= x is a perfect linear line um with a slope of one but we're rectifying that with um this kind of nonlinear piece which is this uh flattening of anything negative to zero. Um because a true line would continue this way for anything negative. It would still it would just that would be the true line, right? It would be um anything negative would just give us that value back in return. But we're actually chopping that off in favor of making it just zero. So we rectify it that way. Um the the way you pronounce this is called relu. Um, so relu, re Lu, usually how you'll hear that pronounced, relu. Um, th this is this activation is the one that's most often used in the interior of that network, those hidden layers, um, because it allows us to basically generate the outputs that we care about, um, as long as they're positive. So it it doesn't limit us down to a specific range like 0 or one. It does limit us to a bit wider of a range than that like 0 to infinity essentially. Um but in this case like these guys would likely be all relu activated. Um and then like this one would probably be a sigmoid for uh a binary classification. So this is a popular activation. We're going to use it quite a bit. And of course, like in our code, we'll be able to use this as one of our activations. And like I said, where you'll mostly when you would use this one is pretty much in any hidden layer. Um, your default is going to be ReLU. There there's very few reasons you should not use a ReLU in your hidden layers unless you have a good reason not to. Um most like 90% of the time you're going to see relu used as as a activation on the hidden layers. Um the output layer is where you'll see those differences. You'll you can see a sigmoid you could see that step function in the perceptron and then we'll talk about the next one which is softmax which will which will be used for probabilities as well. But most of the time we'll see relu show up in those hidden layers um between the input and the output. ReLU is very very popular. It's just an it's a very effective activation. Uh it's a very effective nonlinear function that kind of just keeps things moving forward um as long as they're positive. Okay. So what's the difference between the sigmoid and the relu that we've looked at so far? Well, the sigmoid has a uh it's a little bit more complex of a function. It's that exponential that logistic. Um, and one of the issues with the sigmoid is it can be susceptible to what's called a vanishing gradient, which we'll talk about later. Essentially, this is this is a consequence of the sigmoid always being between 0 to one. So, the issue with that is it's a fraction, right? a sigmoid is going to produce a fraction that's between 0 to one. The issue with that is [clears throat] um when we're doing the gradient descent learning process, if we're carrying around a lot of fractions, this is why you'll never see sigmoids in those interior layers is because if you're carrying a lot of a lot of fractions, they they multiply together with the because they become the inputs to the next layer. they multiply together with the weights of the next layer and really decrease that and and really limit your learning ability because you have such small fractional values. Um whereas relu can avoid that because it is generally just going to give you what you put in. So it's going to have a wider range, right? It's not going to limit you to 01. Um so it but it's still nonlinear. So so it can help you learn those kind of complex relationships still. um with the help of kind of the weights and the training process. Um so this is just to say you should generally be using a sigmoid in your output if you're producing a a binary classification output. You should avoid using sigmoids kind of in the hidden layers if you can because it's going to make learning more challenging. Uh it's going to make training the neural net more challenging. And we'll study a little bit more the details of why that's the case. But uh it's mostly due to its range. It's producing the sigmoid produces fractions which carry through the throughout the network and it it kind of compounds like if everything is a fraction things are going to compound really quickly and and make small values that are very difficult to learn from. Okay. So that's why you typically see relu in those uh hidden layers um because it makes sense. You don't want to have these small fractions kind of all over the place. Um, does that make sense? Like if I go I actually let me just illustrate that real quick is like you can imagine if we had uh going into a neuron if all of these guys were fractions coming out of a sigmoid from from other neurons in the layer. Um that's just going to compound, right? Like if this is um if this is 1/3 and then this is like uh 1/2 and then you have um maybe another neuron that's connecting to it that has like a 1/4, then you just have a bunch of fractions that are being weighted. Um and uh they they're always going to reduce these values and and limit your range and kind of compound things. Um, so, so you don't usually see those sigmoids in the hidden layers. You'll see them at the end if you want this to particularly be a fraction. You'll see that. Okay. So, relu nonlinear, it's just the maximum between zero and x, whatever is going into that activation. Uh, and you'll typically see these in the hidden layers uh in between the input and output. All right, I want to cover one more activation because it's also going to be really popular uh especially it it's really popular especially as the output um using it in the output layer for multiclass classification. So the softmax uh actually just extends the sigmoid. It's like a multi-dimensional sigmoid essentially. Um so the softmax produces values between 0ero and one just like the segoid does. It produces probabilities. The difference is that the softmax um is useful for generating multiple probabilities that add up to one. So um what you'll typically see the softmax used for is the activation across the uh final layer in the output if it has multiple neurons in that final layer. So let's say let's say we had um three. So we we have a neural network that kind of looks like this. And then we just have um maybe a hidden layer of four. And then our final output is three because we're we're predicting um dog, cat, and giraffe, right? Those three animals. Um what we would what we would use for this activation in this layer is the softmax. We would use the softmax for each one of these. And what that's going to do is produce probabilities that such that all of these are between 0ero to one and they actually are normalized such that they total up to one. Um so softmax is really really great as you'll see it in kind of your final layer. Um if you're if you're doing multiclass classification, right? So so you're predicting amongst a thousand things then you'll have a thousand probabilities that total up to one. In this case, we only have three. So, we have three probabilities all that are between bounded between 0 and one that total up to one. Um, so you'll typically see this in the output layer. Um, especially for like image classification. Um, you'll see that or it could be, you know, um, multiclass classification. Uh, you'll see you'll see this softmax. Another place you see softmax is in um like an LLM uh like a a large language model will use a softmax as its final layer because it needs to produce probabilities across every token that's that's possible right so and the reason is is because the model like a GBT is going to predict what is the most likely next token and so that'll be represented by one of these probabilities right what's the highest probability as kind of the most likely next token. So, softmax incredibly useful for anything that's multiclass classification and it should be your last it should be in the output layer, right? It's going to be the activation that's used on the output neurons and the final layer to normalize them between 01 and and get them to add up to one more importantly. So um you know it's a it's a really uh it's a really critical function. The softmax function if you're curious what it is mathematically is sort of like the sigmoid except it's um more of it's more of an exponential. So it's kind of like uh um across a summation of these. Um so you uh you actually apply the softmax to individual components and then you sum that up over all the components. Um and so this normalizes it to be between 01 and uh make sure that the total of it is one um effectively. So it's a function that looks kind of like this and you apply it to a component in the layer. So every neuron in this layer is going to get uh this function applied to it. And by the way, these are all all these activations are built into TensorFlow and PyTorch. So we'll be able to use them really easily inside of our code. Um they'll be built into those libraries. So uh it'll be really straightforward when we declare a layer of neurons. What activations do we want to use? um it'll be really straightforward to set that all up when we start building our own networks. So all of that will be readily available and there's really good documentation in both of those libraries on kind of what activations are available. Like I said there's more than this. Um you know there are other ones like some po some popular ones are are like hyperbolic tangent is a popular one. uh especially with some image problems, you'll see this uh uh tanch be used hyperbolic tangent. Um that's a popular one. Um there's other there's variations on uh on relu. There's there's one called gelu. Um there's there's one called leaky relu. There's all there's actually a lot of them, but um the ones that we've covered on this list are the ones that we'll primarily stick to. uh as we go through our examples. So that's a little bit about activations. So just saying that there are you know definitely multiple of them and uh which one we use depends on really the situation. Um all right so I wanted to do a little bit of a demo here to show you an example with some activations uh and kind of how the activation is applied. Um let me jump over to that. The purpose of doing this demo is uh mainly to see how we can build essentially a simple um neural network from scratch. So not using TensorFlow or PyTorch, but see how we can use like a weighted sum through an activation and see that actually work in code, which I think will be really interesting to see. um and and really start to put the pieces together of a what a neural network looks like. Um so we can kind of get a feel for the different components of it without yet going into our uh frameworks that are going to handle that more in depth like TensorFlow and PyTorch. Um so let's see. So first of all we're going to import some things from numpy. So some functions we might need like the dotproduct um random value initialization um because we're basically we're going to build our own neural net that has those weights. So we're going to start out with random weights. So we need some randomization here. Um and uh we'll have some various like exponential function and dotproduct things that we may need uh from numpy to do this. Okay. So what we're going to do is um essentially we're going to start by building a neural network uh a fake neural network. I say fake because you know it's not going to use our frameworks that we're going to learn about but it's going to show you the the basically from scratch some of the components that you would see in a typical neural network including the activation which is really important. Um, and what we're going to do is basically build a perceptron uh neural network. So it's going to it's going to take uh three inputs essentially um an input that's a size three vector basically 3x1 matrix um and uh it's going to map it's going to use this particular activation. we're actually going to use the hyperbolic tangent activation uh and and produce an output um which which would be kind of like a regression almost um but just to show you the different components of this network to so you can see what's really involved with it. Okay, so um to show you what's going on with this fake network. So we're going to build our kind of network from the ground up which is going to be kind of like a perceptron. Um what you're going to see is we need some weights and we have uh these weights which are generated randomly. Um and this ensures that their range is between minus1 and one because this uh numpy.random.random generates a random value between 0 to one. Okay. So if we multiply something between 0 to one and multiply the by two uh and subtract one kind of the range that we limit ourselves to is between minus1 to one in terms of those weights. Um so these are just going to be randomized weights that that live somewhere between minus1 to one. They're going to be random though. And the reason we're doing that is just to start out with randomized weights. This is actually how a network usually works is when you initialize the network, it has random weights to start with. What needs to happen is you have to train your network so it learns those weights during the course of the training process. But this is pretty standard. Um uh this is pretty standard that this is going to be uh um randomized to begin with and when we start building our own networks in in our frameworks like PyTorch, TensorFlow um those will also be random to kind of start with and then they get trained, right? How is this plural? Um because there's three of them. This random uh 31 generates kind of a a vector of size three. Um technically it's a matrix that's three rows and one column. So there's multiple weights. All right. So then we have our uh activation function here which is going to be hyperbolic tangent. Now all we're doing is when we give an input to this hyperbolic tangent. We're just going to um apply the numpy hyperbolic tangent to that input. Now what this is actually going to be our activation. Now we could use any activation we want. This is just an example. We could use relu. Um we could use uh sigmoid. This is just an example of declaring an activation and using it in our network. So I'm going to show you what that looks like. But you know if it was relu then that would be um you know it'd be like max between zero and x would be the relu. Um but in this case we're doing uh MP uh tanch which is hyperbolic tangent uh applied to x. Okay. That's just a function just an activation function. Okay. Now the other thing we have here which is um I'm not too concerned with right now because we haven't learned we're going to in in the later notes in this lesson. we haven't quite learned it is the gradient descent process for training a neural network. We're going to learn all about that coming up shortly. But one of the things you need in order to uh in order to do gradient descent is the gradient of the activation um which is this uh so the derivative or the gradient of uh the hyperbolic tangent um uh is this function here. That's just a mathematical uh function that involves that hyperbolic tangent. So the reason we need this is the reason we need this is for training uh gradient descent. Um and again I'm going to go into detail on what gradient descent is coming up shortly uh in the notes but um it's just here in this example for the an example of how you would train this thing. Okay. So we have the activation derivative which will be useful uh in the training for gradient descent. Um okay and then we have a uh function called train. So this function is called train and takes the training inputs um training outputs which are kind of like the labels and then um a number of training iterations. Um again this here is going to be our this is going to be gradient descent training uh process which we haven't learned about yet. So I'm not too concerned with this digesting all of this this moment. But just so we kind of preview what's to come. Essentially what we're doing here is an iterative process through our through through our data. So we basically iterate a certain number of times which is configurable this number of training iterations. We basically do this uh we iterate this many number of times through this descent process which is to say we basically produce the output given our inputs. So we do a forward propagation. Um, so that's what this is is we do a forward propagation to produce outputs. And why do we do that? It's because we want to see how far off our network currently is. So if we produce outputs, what we can do is actually compare that to our labels, which are these guys. and we can compare our output to that and see how far off we truly are. Um, which is what's called our error or sometimes called the loss. So, this part here is um computing the loss, which is how bad our network is currently. So, we're we're just gathering up how bad we are currently. Now, why do we do that? It's because we use that quantity to make an adjustment to our weights. That's the key thing is we actually want to make an adjustment to our weights to basically shift our network to perform better. Our goal is to minimize this error. I should I should write that down. The goal is to minimize this error. So essentially what we're doing is measuring our current error and then using that quantity along with the derivative of the uh activation which is here the derivative of the activation. We basically use that amount multiplied by that derivative um uh we basically use this amount to figure out how much we should adjust our weight. So this is a numerical calculation that's driven by this gradient. Um so that that's why it's called gradient descent is we we essentially want to uh make an adjustment that uh will minim that will lower our error over the course of the iteration. So as we iterate a certain number of times we should be getting better and better and better at making predictions using our network. Um and you can see here like we make adjustments. So you see how we take our weight, we take whatever our weights currently are and we add in an adjustment. So this makes a tweak. It's think of it like turning a knob. So we essentially turn the knob and the adjustment is telling us how much should we turn that knob. Should we turn it a lot? Should we turn it a little? Um we're figuring out how much we should make that adjustment to get better outputs of the network. So this is um turning the knob um so the network can predict better, right? So it so it can it can make the right adjustment and predict better in the future. So this in a nutshell is how any neural network's going to train. Now this process will generally be handled by the framework. So by PyTorch or TensorFlow will manage this for us because it turns out when you have a pretty big network that computing this adjustment is incredibly intensive on a big network. Computing what that adjustment should be is is actually very computationally intense and it's called back propagation. So com so basically uh doing the weight adjustment and uh figuring out uh figuring out how much you should adjust and then actually turning all those knobs. You can imagine like for a very large network that's going to be an expensive operation. Just to give you some context like modern OM uh have hundreds of billions of these guys, hundreds of billions of weights. So you need to compute an adjustment for every single one of them, hundreds of billions of weights, right? And you need and you actually need to do that every iteration. So you're imagine training for thousands of iterations and you have hundreds of billions of weights that you need to adjust. It's a that's why they take forever to train. [laughter] That's they're massive networks. They take forever. That's that's why uh because of this computation right here can be very expensive uh because this derivative can be very uh computationally intense to compute. Um it depends on your activations. It depends on how big your network is. But generally those large networks like an LLM uh it's going to take a while. That's where you benefit from a GPU by the way is to try to make this computation faster. Um so we'll actually see that like when we build our computer vision networks uh we'll do a similar training process um and we'll utilize a GPU there to make this faster. Okay, by the way I forgot to mention uh this forward propagation when we produced output uh we call it think um mainly because we are uh it's like sending data through our network. So it's like all of our neurons are thinking together in this network. Um so thinking what is thinking? It's really um th this is uh doing our weighted sum through an activation. That that's all it is. So it's like our our weighted sum through an activation. That's all it's doing, right? Weighted sum through an activation. So you can [clears throat] even see that here. Notice how we apply the activation the tanch to this weighted sum. Now why is it a weighted sum? It's because you're taking the dotproduct between your inputs and your weights. So that literally now what does dotproduct do? Remember dotproduct adds together the product of these guys as vectors, right? So that's that's taking our weighted sum and then does that make sense? This is taking our weighted sum and passing it through this activation. So that's this think is just computing that neuron essentially, right? Computing that neuron. All right. So just to finish out the demo, um that's the network. So that's our that's kind of our fake network. It's got three weights. Um it it does uh it it thinks by doing a weighted sum of those inputs through. So it has three inputs, three weights. Take that weighted sum, pass it through a tanch activation. Um, and again, the activation is kind of arbitrary. We could really have it be whatever we wanted. We it could be a sigmoid if we were producing a probability here. It's just producing a tanch, which is going to be tant has its um outputs are actually between minus one and one. So we're always going to get something between there. Um so then we can run this code here which is going to take a random set of inputs um and then the labels for each one of those. So these are all like different labels like so we have four different input sequences. Notice how every input is size three which is what it should be. So here's here's one input. Here's one input. Here's one input. Here's one input. These are more or less random just just to show you how it works. And then here's a label for each one of these. So this guy is labeled zero. This this guy's labeled one. This guy's labeled one. And this guy's labeled zero. So we have four labels here corresponding to each one of these uh sets of three inputs. Um so here's our random weights to start with is this vector. So we're printing that out just so we see what our random weights are. Now when we go through the training process, we're just going to have 10,000 steps and use those random inputs and outputs. Look at what the weights are after we do this. The weights are now completely adjusted. And we can actually put in a new uh input and see what the output would be. The output which is just running think on this input. Um it produces a label of one. Okay. So the network functions as we would expect. Um and the big thing is to to take away here is to see how those weights actually get adjusted. you see how far off they are from what they started as. So here they started as these really random numbers between minus1 and one and uh here they get updated through the training process. So those weights actually get you know changed which is the objective of the training. Uh okay. So let's talk about more let's talk more about those uh uh terms we just used mainly forward and backward propagation uh in our neural nets particularly um you know just thinking about it from a neuron perspective data going in um and then making those weight adjustments. We'll talk more about that. So training any neural net really involves two particular phases and we just saw those in the code really uh involved in that training function, right? And um those two phases have a name which is known as forward propagation and backward propagation. We're going to break down what each of those really means. Um but it's kind of what the name suggests uh generally is that forward propagation is going to be all about sending data through to produce outputs. That's mainly what you want to do. And in fact once you're done training and you're just using the neural net to make predictions. That's actually all you're doing is just putting data forward through it to produce an output which is sometimes why that is called inference uh because you're just making a prediction at that point. Now the backward propagation is what makes those adjustments to all the weights across the whole network. And so when you have a very um deep network, you need to be able to adjust all the weights across the whole network. And back propagation is going to be the algorithm that kind of does that and manages that weight update. Uh, and it's all going to be driven by this gradient descent that's that's dictating how much of an adjustment should we make to these weights. And then we basically need to propagate those adjustments throughout the network, which is going to be what back backwards propagation is actually going to do. And so you really only do backward propagation in the course of training. But it is an iteration of doing forward propagation to generate predictions to see ultimately to see how far off you are in those predictions from an error perspective. Uh and then you basically do backward propagation to go back and uh adjust all your weights and then you just keep iterating doing these over and over and over until you make enough adjustments to kind of uh have a solid network that makes predictions really well. Um, so training is really just an iteration of these two things. So we're going to we're going to dive deeper into what these two things are, but that's that's basically the summary of it. So I think forward propagation is the easiest one to understand of the two and I think it's because it's really straightforward. is essentially um it it's called forward propagation because when we have this network of neurons um data flows kind of forward through all of the layers, right? It goes uh we have inputs that go into our first layer that generates some output which becomes the input to the next layer and on and on and on and eventually we reach that output. So in a perceptron in a which is a very basic obviously a very basic network basically just one neuron. It's not even really a network but uh just a single neuron. Um we think of that as as forward propagation really as just producing the output. Um and we know what does that what produces the output is the weighted sum through an activation. Right? That's really what produces the output is this typical uh what we've been calling weighted sum uh through an activation. Um and so this produces the output data flows forward in this process. So forward propagation is just generating the output through weighted sum through an activation. Now obviously when we have a more sophisticated network so if we were to draw multiple neurons in the input and then maybe some hidden layers um and maybe multiple hidden layers and then like a final output let's say and let's say that's a probability um you know the the forward propagation is going to be a little bit more involved because we're going to have I can even draw this in a different color you know we're going to have contributions from all of these that are going to be weighted And then contributions here, right? Contributions here, contributions here. So it's this much more involved network. And then data is going to flow like this. And then data is going to flow like this. So much more sophisticated um flow. Um but forward propagation is generally moving this way, right? We're sending data in generating an output. That's the that's the forward propagation really producing that output. Okay. So for the perceptron, we know that the output is either zero or one, right? Because the activation for the for the perceptron is this is basically this, right? So if that weighted sum is positive, we get an output of one. That's the activation that that gives us the final output. Um if that weighted sum is negative or zero then we get zero. Right? That is the that's just the activation function. Um now what needs to be learned are these weights. And that's true of any neural net. All of those weights are really what need to be actually learned during the training process. So that's kind of what we saw in that in that demo is we need to make adjustments to those weights to basically have the best network structure that we can have. We want to make adjustments to those weights so that we accurately predict one or a zero or accurately predict the probability um for our labeled data. So we can't forget that training a neural net is still like a supervised learning problem, right? We still have labels like on our like think about images. We have all these images of cats, dogs, giraffes. Every one of those uh images has a label associated with it. Um so that we know if we're getting an error or not and that can inform us making an adjustment to all of our weights. Um so ultimately we need to find the best weights which is what the training process will do. it will make adjustments to those based on the error that we're producing. We can only really calculate that error if we're doing forward propagation, right? Because we need to know what our network thinks is the right answer. That's again kind of in that demo we we saw it was called think is really the forward propagation, the weighted sum through an activation. That's what forward propagation does. It produces those outputs. Okay, so we have forward propagation just the simple act of sending data through our network producing an output which we know is really just this series of weighted sum through activations. Even if we have a big network, it's still going to be this collection of weighted sum through activations over and over and over again to produce some type of output. Okay, so that's forward propagation. Now let's talk about the error function that we saw like in that demo, right? So, we need some way of evaluating where our network is currently. If we're producing results in order to make the right weight adjustments, we need to know how far off our network is from the true labels. And that's where something like a loss function comes into play. We've actually seen loss functions in machine learning quite a bit. Remember we had mean squared error, we had uh cross entropy um for classification like with logistic regression. Um so loss functions are not new to us but in the context of neural networks they are a little bit new and we're going to break down kind of how they're used. But this idea of a loss function should not entirely be new to us. Right? We've seen it before in machine learning especially with things like linear regression. Um you guys remember we had the mean squared error loss function that told us how far off our regression was from from the the true value. It was kind of that distance, right? So in terms of neural networks, it's actually the same kind of concept. We have um we need some way to measure how far off our output that we get from forward propagation is from the true value. And this is really really important because this is going to inform how much of a weight adjustment we should make. Should we make a really big one? Should we make a tiny one? That's that uh that's that notion of turning the knob. Should we turn the knob a lot? Maybe just a little bit. That's all informed by how much error we have when we do forward propagation compared to the true label. So um that's just the general idea of the law. So uh in in neural network terminology um we usually think of this error we usually call it a loss function which is some type of function that computes what that error is between the label and what our network is producing. We usually call that a loss function. Um uh and the goal of the training is to actually minimize that error minimize that loss function over the course of the training. But how much loss we have influences how much of a weight adjustment we make to the network. That's a really key idea in the training that that we should uh kind of understand intuitively that you know if we're really far off we need to make more of adjustment to our weights. If we're doing pretty good and we're not that far off, then it's a smaller adjustment we have to make to our weights. So it's it is proportional to that loss or that error. and we'll see exactly how proportional in in the gradient descent formula. Okay, so just to give you an example, you know, um it you know we've seen the mean squared error loss function before from something like linear regression, right? So we know like this value here would come from the prediction of the model uh in a linear regression that was our typical like beta times our times our features right beta 1 times feature one beta 2 feature 2 and we added those up and we could we generate a regression value and we see how far off we are from this in terms of the distance right that's the mean squared error we've seen this loss function before this is just an example of one uh that we see in In machine learning generally, when it comes to neural networks, we're basically goin