New Paradigm: Single Layer AI

Key Takeaways

Hello community. So great that you are back. Is it possible to have a one layer artificial intelligence system? Can we compress this? We have some brand new research. So welcome to my channel Discovery. Let's have a look at the latest paper. Two papers both published here on December the 8th, 2025. One is from Apple and one is from you're not going to believe it here. So let's have a look. Now you might say hey one paper is here about visual encoder for image generation and the second paper is about a deep neural collapse here in in an asmtotic analysis of shallow and deep forgetting with a neural collapse here in an AI. How are they connected? You are going to be amazed. So first paper beautiful they have a simple idea. Now you know we want our AI to have a continual learning process. Now aims to train neural network on a sequence of task without the catastrophic forgetting. So important for adaptive AI system such as autonomous agent they must integrate continuous new information without a full retraining here classical supervised fine tuning or reinforcement learning or centralized data access. So how we do continual learning in AI solution is easy. We do have a replay. So this means the practice of storing a small subset of the past samples of the past data for joint training with the new data. This is among the most effective and widely adopted strategies for a continual learning process. Now those replay buffers here affect the two forms of EI forgetting. We do have a shallow forgetting corresponding to an output level degradation that is recoverable by a linear probe and a deep forgetting of our EI corresponding to an irreversible loss of feature space separability. And yes, we are talking here about geometric features in EI. So let's have a closer look. Yeah, you see here simple you have some data points. Great. If you try to separate these data points then somehow you can separate here the classes but then some shallow forgetting happening. Now the interesting part is what is this forgetting process in deep? What is really happening if an eye forget something? Can we have a deep dive on this? I'm so glad you asked. Now you have to have a knowledge here basic knowledge about here the neural collapse here during the terminal phase of the deep learning training. This was done by Stanford University and Cornell University. You see here August 2020. So this is an old paper but we need the concept here of the neural collapse. To my knowledge this was the first paper and they explain beautifully. Of course you already spotted here the terminal phase of the deep learning training. So in the last layer of our transformer architecture there is something happening that is now becoming more and more interesting. If you want to have a closer information on the neural collapse here, this is for you. Otherwise, we go now back to the main study from the Hatsur and they tell us they examine four main or they have four main findings. Let's formulate in this way. There's a replay efficiency gap. The researcher from Turik tell us hey we identified an intrinsic asymmetry in the replaybased continual learning. some minimal buffers are already sufficient to anchor here a feature geometry thereby preventing here the deep forgetting whereas for the shallow forgetting it is something completely different happening. So it seems that these two kind of forgettings are different features different methodologies. seconds that tell us that they extend here the mathematical framework of the neural collapse theory to a continual learning theory characterizing here the asmtotic geometry of both single head and multi-head architecture and with their mathematics they have here an deeper insight and they say you know what's happening we have some rank reduction in task incremental learning and I will give you a simple explanation what this means in practice Effect number three is the effects on replay on the feature geometry itself. And the authors tell us we demonstrate that a shallow forgetting arises because the classifier optimization on buffers is underdetermined. This will be an interesting part. And finally for the out of distribution detection, the authors tell us we reconceptualize here the deep forgetting as and now hold on to your socks because this is interesting. Deep forgetting as a geometric drift toward and out of distribution subspaces and those subspaces are orthogonal to other spaces. So this perspective bridges here the gap between the continuous learning process for AI and whatever we encounter with out of distribution literature offering here a real beautiful and rigorous geometric definition of what does it mean if an AI starts to forget something that goes far beyond a simple accuracy loss. So we have now a geometric definition what it means hey the eye is forgetting something. Why is it happening? Now the evidence comes now from continuent learning. So I mean what we thought I thought hey the neural network forgets something simply by learning new task and simply somehow in the brain here in the tensor structure. No some data is simply overwritten. We have here a recalibration here of the weight tensors and this is what is deep forgetting. Now the discovery of this paper here is no this is not what's happening. They prove mathematically and therefore I just give you here the main idea that the brain here the deeper layer of our transformer architecture actually remember almost perfectly all the data they tell us you know what in the weight structure there is no forgetting the deep features remain separated and distinct we have a linear separability everything is preserved there is all the data that you want and then you might say so what is app. What is forgetting for an EI system? No. So this is now my part. I want to give you a simplification that everybody understands it not only if you have a PhD in mathematics. So how can I explain this? Let's try this one. Let's say the network has learned the task A and now the network learns a new task, a task B. And it holds the data up like a sheet of paper facing you. No. So we in a three-dimensional room. And you see here clearly here perpendicular to your line of sight you have a sheet of paper and there are all the instruction for the task B and all the data of task B now and you can see this picture perfectly now and now the classifier of course this is one of the last layer in our transform architecture is simply trained to look at this frontfacing view that's it now when the network moves to the next task it simply wants to clear the view without deleting the old picture so what it does and this is really happening It rotates now the old sheet of paper, the old information 90°. So this means the old data is still there in this space but it is now orthogonal perpendicular to the current view of the last layer of the transform architecture. So this data kind of vanish out of sight. Although they're still there in the weight structure, but it is really happening that they the if you want the internal procedure of a transformer stores this new data in a subspace that is orthogonal to the other space because the classifier never learned to turn its head. No, it's just looking always in the same direction. It's in my simple example. Now, it's still starring straight forward. No. So what it sees it looks like the rotated old tasks and sees only the thin edge of the paper. This is now become an invisible line classifier in our reports as the last layer of the transformer. Let's it's a classifier layer. Hey I have forgotten let's say this image I forgotten the data and as I told you the image is not erased. It is still still there. It is just turned in an orthogonal subspace and is not visible for the classifier in its current position. Why does the artificial the AI network does this? It is the most efficient way to stop old memories from interfering with the new memories generated. Imagine you would project everything on the same single or two-dimensional flat plane. You would just get a soup of data. Yeah. But rotating all new data in a new orgonal subspace. This is an elegant solution. So by rotating the old data into the blind spot into the orgonal subspace of the transformer the EI minimizes the conflict the interference and the energy and the weight decay while keeping the memory intact deep inside. It is just that the last layer is not anymore recognizing that there is an image, there are the data and there's everything still in the not the last layer but in the depth here if you want of the transformer uh weight structure. So what are the replays doing now? Even a tiny amount of replay now kind of puts a finger on the sheet that wants to rotate away preventing the sheet from rotating a full 90° out of sight. So it keeps the paper if you want grounded tilted just enough so that the classifier still can see it. I know this is a simplification but it is really the exact mathematical thing that is happening here in the mathematical um explanation. Have a look here at the original paper. So this means and this is now amazing the failure that the eye in total the classifier says hey I have lost all this information is just in the last layer it is in the classifier head itself because the statistical anomaly is like the co-variance deficiency and yes if you look at Matt this is it this is the main reason the co-variance deficiency and I will show you this in a moment the final layer gets confused and draws you the wrong decision about the boundaries here for its decision decision making. So what we have we have a one layer conclusion. You do not need to retrain the whole network the whole llama 3 or the whole whatever you have to fix here the forgetting process if you want to have a supervised finetuning for some new data. You can do this. Yeah. But the main point is the data are still then in the network. You just need to fix the geometry of the final layer. And this of course means you need to fix the geometry of the orgonal subspaces. How the data on the transformer architecture layer are stored. And then how if you have a classifier head at the very last layer, how you can optimize this. So it can if you want rotate here across the complete space and has sees a 360° view of all the data in this limited space. Yeah. On the mathematical explanation is of course a rank deficiency co-variance. So one minute mathematic is this okay for you? Great. So the paper formalizes this using here coariant matrix here and the coariance describes if you want the shape and the spread of the data cluster in a particular representation in a new mathematical space and the rank as you know describes how many dimension that this spread of this data cluster covers. Now if your feature space let's say is classical 512 dimensional mathematical space the real data has a coariant matrix of rank 512 because hopefully it's really spreads out in all direction and you do not have here that it uh is limited to a sub manifold but if your replay buffer only has 50 samples guess what the empirical coance matrix can only have a maximum rank of 49. So these are degrees degree of freedoms that are now left here for the replay buffer. Now this creates of course for anything that's watching here from the last layer a blind spot. Now so it simply means that there are 512 - 49 dimensions where the co-variance is exactly zero. So this means that your classifier your last layer in your AI architecture is mathematically blind to those other 463 directions. It perceives no width, no spread, no obstacles, nothing. The data are really gone. The AI seems to have forgotten the data which with this insight now we know it is not true. It is just not able to see the data but the data are still there. So consequently the mathematical optimization is simply what we called under determined. This is a rank deficient covariance explained in one minute. Thank you. Back to the main paper. And now and now you know what we have in the first paper this one layer. And now here in the second paper by Apple. Hello Apple. We have already yeah this is yeah one layer is enough. So the headline is giving it away. But you see here this parallel thing that is happening but on complete different level. So I'm fascinating by this and Apple is uh is talking here about the visual encoder for image generation. So let's have a little bit deep dive into this. Now in computer vision you know we have a mismatch no because we have our models that try to understand what they have what they see in computer vision and then we have our AI models that try to generate images. This is our diffusion model, our flow models and understanding you have a dyno or cichlip or whatever you have. No. So understanding model is easy. No produce highdimensional semantically rich and geometrically complex embeddings hypospherical embeddings they optimize for the discrimination segmentation identification everything. Now if you want to create no image or generate something you should require lowdimensional compact and smooth latin space the gorian distribution they are sensitive to noise and high dimensionality and it is not as easy and nano banana pro is something special absolutely. So if you want to have a look at the classical architectures here, you have here yeah the standard variational autoenccoder and then we went here in my channel here to a quantized version of the variational autoenccoder and everything. But they tell us you know what you don't have to do this I mean just look here at those numbers here the number shows you the channel dimension of the generative modeling space and we minimum 1,500. Imagine we could do this with a new architecture and look at this that dimension here would simply reduce to 32. Is this possible? Guess what? So what are the new insights by Apple? I mean unbelievable. Apple is telling us they discovered here a counterintuitive phenomena that the deep encoders hurt the adaptation. So they say you know what when we're training here an adapter to map a highdimensional feature to a lowdimensional latent a deep network like a six layer transform architecture tends to overfit this compression task of course no it learns now it has enough intelligence if you want enough layer it learns to reconstruct the signal by memorizing patterns essentially scrambling the original semantic geometry it is trying to optimize the representation of the data the information and the knowledge into a dense of space and this is more or less what we want from a deep network low an optimized representation but in this case and only in this case we don't want it we want something else we want a single attention layer and you know why it is simply too simple this s single attention layer to scramble here with the data so this is it is forced to perform a linear-like projection that preserves the spatial and the semantic relationship of the original backbone data like a Dino version two effectively acting here only as a semantic pass through rather than a complex re-enccoder. So we want the shape and everything to stay in place and you know what a single attention layer is just perfect for this. So they build now with this inside here a new architecture they call it a feature auto encoder. You have already guessed it, an extreme simple design compared to our original water encoder and everything else. You just have, and you're not going to believe this, a single attention layer that is paired, of course, with two relative lightweight relative lightweight decoder structure. So, we have an encoder and we have a decoder and this is it. And you say this is not it's not possible that a single attention layer can do this. Well, have a look what Apple find out. As I told you, this feature autoenccoder architecture is deliberately now unbalanced. No, we have a tiny encoder really one layer. I mean, this is really this is it. And then the decoder is now split into a feature decoder and a pixel decoder for some reason. So we encode the data and then we decode the data and we hope that there is some interesting representation that helps us further to build the next images. Now what is the goal of the encoder? Simply no to compress here the massive representation from a frozen backbone like a dino. I have a dozen video on dynino just giving dyno and the search into a tiny latent space set without losing here the important semantic geometry of the information. Why one layer? If you use a deeper more complex encoder the network becomes too smart. It learns to encode the information in a convoluted nonlinear way that destroys it relationship. So therefore I love or they love here a single layer that acts as a simple projector. This is it because it forces here the tiny latent space to maintain the same shape and the same topology as the original richer embedding representation. So this simply in simple words it ensures that if a dog and a wolf as a term or as an image are close in the original dino space they remain close in this tiny little latent space also. So we just project without any distortion. Now the decoders on the other hand have a much harder job. Now they have to take this tiny compressed summary and turn it back into a high fidelity image. Now they do this in a clever way. They do it here with a two distinct difficult steps. So they split the job job up. So the first is a feature decoder. It's the unpacking. Now you know the latent Z is lossy because it's so small. So to get back to the rich dino embedding here X head or whatever you like to call it, the network needs to hallucinate or come up with a solution or unpack the missing details based on the context. And if it has the data it is doing it's building the bridge and it has a not enough data it starts to hallucinate and invent this bridging function. So this requires you a deep network to really understand the correlation that we are looking for. So we can build a bridge firm over this correlations. So for the job to be done is simply we start with a little bit of math in a little tiny space and we have to do much more mathematic in a much more complex space. Beautiful. Now the second step is now building on the first decoder. So the second decoder, our pixel decoder is now doing more or less the translation. Now that we have the reconstructed embedding x hat, this beautiful huge embeddings, it's still just a list of number not colors. So no problem. This is why we have not the pixel decoder just needs need just needs to translate the semantic language into the visual textures, our RGB pixels. Now it's a heavy translation task but yeah we have the architecture so what is the job to be done we have heavy mathematics and translated now into pixels quite an elegant thing and you might say is this really working yeah if you want to have another framing a little bit more on the mathematical side here I have this here for you one layer encoder input operation output input is simple frozen patch embeddings here from the dino version 2 in 1 1536 dimensions The operation is a single attention block followed by a linear projection. You notice the output is a tiny compact latent set in 32 dimension. In this extreme compression forces you the latent space to retain only the most critical semantic topology of the original space. Then we have the double encoder strategy where we have the feature encoder that reconstructs original embedding from the latent zed. This ensures that Z retains here the language of Dino version two and the pixel decoder generates here the actual pixel from the reconstructed embedding X head not the latent one and therefore great this thing works. Now you might ask how do we train this little something something here that is here one layer encoder. How is this possible to train? They have here particular training process developed. This is not so interesting. Yes, it's working. Beautiful. What is the inside? We do not need or you do not need massive variational autoenccoders to bridge here this perception and generation of images. A minimal linear attention projection preserves here if you want the brain of the visual encoder best allowing here a new generator to focus purely on the dnoising dynamics. We don't need this old heavy clumsy architecture of visual transform of visual transformers. How is this beautiful? Look at this. This is just some images where they tried here to identify the most similar patches here from two photos of a bird or two photos of an elephant. Look at this correlation that could achieve here with this. I think this is just beautiful. So you see we thought till yesterday that to turn a reading AI like a Dino version two into a drawing a creative AI that generates here the image we needed those massive complex alignment models or giant vational autoenccoders to translate all the features from the different mathematical spaces. It turns out no we don't. So the discovery if you want of this paper by Apple and it's amazing that it's by Apple. So this pera proves empirically that you don't need those complex architectures. A single attention layer sufficient to compress you the deep understanding of a frozen backbone into a format that then the generator can use. So our one layer conclusion is is beautiful and simple. The semantic gap between understanding an image and then creating an image or creating a modified image can be bridged by one single layer. What an amazing insight. So let's take a step back. What is the summary? Both papers show us interestingly at the same time December 8 one layer seems to be enough. We have been on the wrong track. We thought we have to be build bigger and more complex version of autoenccoder and everything has to grow and scale up. No, it's completely opposite. We don't need any of those. So why we should stop training now our backbones and think more about intelligent one layer or last layer architectures? Yeah, old way. Yeah, we updated all the weights. We had a complete run. It was really expensive. If it really took I don't know hundreds of GPUs all of those great the future of an efficient seems it's not building bigger mouths not at all if we have a frozen backbone that really has been trained on our domain knowledge on our theoretical physics on our finance on our medical knowledge then what we need is not a model but just this frozen backbone and a special head structure. in the last layer of the architecture. Now in learning, keep the backbone frozen to preserve the geometry. So you prevent the deep forgetting and only statistically calibrate the final layer. And if you want to generate an image, keep the backbone also frozen to preserve the semantics and use a single adapter layer to drive the generator that a new image is generated. Wow. So the moment we have let's say a unique backbone a frozen for I don't know including physics mathematics chemistry pharmacology whatever whatever the job is we take the same backbone and we just exchange here specialized head just how we look at this amount of data seems to be a much more efficient artificial intelligence system. So let's have a look at the future. This is my idea. I think we're effectively arguing here for a model AI. No, we kind of want just have a frozen universal cortex and then just some swappable single layer interfaces for the different tasks, memory versus creativity. And I think we have a very strong scientifically grounded narrative with the two papers I introduced here at the very beginning of this video. Have a look at this paper. Read those paper. Try to understand them. They have a deeper meaning. Of course, the paper for itself has a message. But combine both papers. Have a look at both papers and I think there is something now that we clearly see at the horizon happening here. Not bigger malls. No. If you have a frozen body, a frozen cortex, a frozen data, if we really would need only single layer interfaces for different task, this would make our computation so much simpler, so much easier, so much less expensive. Maybe we would not even need those data centers anymore. I hope you had a little bit of fun. I hope you discovered some new knowledge here about AI. Anyway, it would be great maybe you leave a like, subscribe, become a member of the channel. Anyway, I hope to see you in my next video.