Point Cloud Classification - Keras Code Examples

Key Takeaways

welcome to the henry ai labs walkthrough of keras code examples keras has provided 56 code examples implementing popular ideas in deep learning this ranges from the basics such as simple mnist and imdb text classification all the way to cutting edge research ideas such as knowledge distillation supervised contrastive learning and transformers we'll also explore fun generative examples like variational autoencoders and cyclegan my contribution to these code examples is to explain every single line of code in each of them walking through each of the individual keras examples i'm not the author of these code examples please consider starting the github repositories to show support to the original authors the next keras code example is point cloud classification with pointnet point clouds are these three-dimensional point sets that are basically like voxels except for the way that they're manifested as just points so differently from our previous chaos example with 3d image classification from three ct scans we in this example we have this three-dimensional volume where we have height width and depth and then in each slot we have a voxel which is a like a different kind of pixel that represents the each individual point in this three dimensional cube a point cloud is different from that in the sense that it's just going to have an x y z point and that's all we use to identify it in this three dimensional space so here are some more resources if you want to learn more about point clouds more generally it's a really exciting area of you know computer vision and computer graphics and things like the lidar scans that help with autonomous vehicles so i found this useful this wikipedia page also i found this on 3d printing stack exchange a quick description of the difference between roster graphics and then vector graphics and you see the pixels voxels compared to this idea of point clouds and then these things like triangle meshes which we'll get into also then this is uh geeks for geeks talking more about vectorverse roster graphics and a little more details about this but for our sake what's most important to understand is that we're going to be representing each point in our three-dimensional space just as an x y z coordinate space so finally before getting into it if this tutorial inspires your interest into these three dimensional meshes point clouds and this kind of geometric deep learning and computer graphics i highly recommend checking out this pytorch geometric library it has documentation with these basic introductions all the way up to more advanced things different data sets you can use different neural network layers that have been implemented and data augmentation functions this is a really exciting library the pytorch geometric and so maybe this tutorial will help you develop an interest for this as well the first step in our notebook is to install the try mesh library basically what's happening with this tri-mesh library is it's drawing triangular meshes using our xyz point cloud coordinates so we can't we can just sample this and just plot them on the three dimensional grid with our sample of x y z coordinates but what's common to use in computer graphics is to draw these triangles and i don't know too much about exactly how this happens but it looks like what you do is you draw these polygons that connect the point clouds and that's one strategy of rendering and visualizing these three-dimensional objects that are stored as these xyz coordinate data structures so then we have our pretty standard imports our operating system glob for aggregating these file paths the try mesh library that we just discussed numpy tensorflow keras chaos layers and matplotlib and then we're gonna set our random c so the next thing to do is to download the model net 10 data set the model net data set maintained by princeton university is a very interesting effort for maintaining these 3d deep learning data sets so as shown in the top we have over 125 000 cad models labeled into 662 object categories and then it also describes that we have 10 categories with annotated orientation so i'm not exactly sure what that means it must have something to do with the xyz three-dimensional space so in the beginning of this model net project page they describe how they constructed this data set how they use mechanical turk to label the cad models how they derive which objects to construct cad models for with the sun database and so on so what we have is we have this data set we have these different benchmarks of classifying these objects and in a second in our keras notebook you'll see exactly what these look like so each of our point cloud objects are stored in these dot off mesh files as downloaded from this link with our model net 10 data set so we use the try mesh library to load this and it's going to draw these polygons that connect the point clouds and render this mesh so we see how we can rotate this around with our mouse and this is really interesting this is really different from you know the two-dimensional data that we classically use with computer vision like imageness c4 and so on so this is really cool we're seeing this three-dimensional object and how to access it by downloading the modelnet data set and then how to use the tri-mesh library to load it and visualize it the next step is to convert our mesh file to a point cloud file by using a sampling operation so we're going to randomly sample 2048 points from this chair object and then we're going to plot it in two dimensions by just indexing the x and the y coordinate in our scatter plot so we have this visualization of our chair image in two dimensions this is the result of taking 2048 xyz coordinate samplings from this original mesh object of this chair image and we have all sorts of other objects as shown later in the presentation we have things like tables and addresser and i think there's even more object categories as stated in the model 10 project page this next function is going to loop through all of the paths for these different objects that we've downloaded with our modelnet link to getting this data set and it's going to loop through them load them with the tri mesh library and then sample the 2048 points that are the xyz point cloud points they're going to represent each object so then we convert these points into a numpy array we have our train points our test points our trained labels we're going to be classifying these into object categories similar to like imagenet but with these three-dimensional objects then we return this data set that's going to be pipelined into our tf.data.dataset object so this is actually calling that function parse dataset and here we see the different uh three-dimensional classes that we have we have a bathtub sofa dresser desk table nightstand monitor bed toilet and chair so we have these different kind of like objects that you see in a room or something like that and we're classifying them into these object categories based on their three-dimensional coordinates so definitely a really cool idea of deep learning so next we're going to actually apply data augmentation as well to point cloud data and the particular data augmentation that's used here is jittering the points so the points are randomly sampled on this mesh surface so an interesting and natural data augmentation would be to jitter them around and add this random noise to their position in the xyz space so that's what we're doing we're using this jittering and then we have also tf.random.shufflepoints where we're randomly uh shuffling how they're presented sequentially with respect to this data type of 2048 xyz pair so the um we're shuffling them so that the model isn't biased or it's any particular ordering there could be some the way that it's doing this sample function the way that it's sampling these 2048 points it could be that there's some high frequency pattern with the way that this function is actually implemented and we don't want that to bias the solution of our model so we implement this shuffling as well with the point so we have this random noise style jittering and then we have the shuffling to make it so we can augment our data set and then prevent overfitting in the data space so then we convert these uh data types to our tf.data object by using the from tensor sizes call passing in our trading points these are these 2048 xyz pairs for each instance and then our labels of the object categories such as bathtub sofa addresser and so on so finally we have our data set we chain in our augmentation function by chaining on dot map augment and then batch batch size is the syntax for using these tf.data.dataset objects in keras so now that we've prepared our data set with our input outputs for deep learning let's look at the model architecture itself so i think it helps to jump ahead and look at the model.summary so our input data is 2048 points each with an xyz coordinate so therefore our input shape is 2048x3 because we have 2048 points each one has xyz values so that's our input layer and we're going to originally convolve over that with a one-dimensional convolution so one-dimensional convolution different from say the classic two-dimensional uh convolutional kernel is instead of using like an n-by-n kernel we just use an n by one a single dimensional kernel so we're going to pass this kernel along through the image convolving it and blowing up the feature space so we go from 2048 by 3 to 2048 by 32. this keras code example replicates the architecture described in pointnet deep learning on point sets for 3d classification and segmentation so what i took away from this is the high level idea is that it uses these tnet blocks to kind of move these 2048 by 3 point cloud points into kind of a more regular space for sequential processing of deep learning architectures similar to like you know pixel grids or any kind of data structure so this is done by having these nested t-net blocks and so the t-net it has this way of taking these input space of the 2048 point clouds and then projecting it into like a matrix that we can use for a more standard deep neural network pipeline so the key to this is this orthogonal regularization shown here so the high level mode i mean the equation is just that the regularization is that the identity matrix minus this feature space matrix by itself transpose should be minimized and by doing that they you know they reason about how they say that that you know helps with the alignment of the feature projection and so on and so on but you know i think it's a little hand wave it's kind of you know it's maybe a little it's just kind of like in my opinion just these feature transformation ideas and i don't think there's too much intuitive reasoning behind it in my opinion but um you know i don't know too much about this so but the idea is that you have these t-net that are you know projecting these point cloud representations into these matrices where we then apply standard things like you know once we uh once we project this then we're just going to kind of stack these layers of convolution and batch normalization regular activation so the tnet is the way of projecting the projecting these point cloud representations into a more normal feature space for deep learning so i know that wasn't the best explanation if you're curious about the exact intuition behind the the pointnet architecture i highly recommend just reading the paper but we're going to kind of try to understand this by seeing what's been implemented in keras so the next step is they construct these convolution batch norm activation layers is a pretty standard block in deep learning and now we're getting into the meat of it this is the t-net block so as stated in this documentation the t-net is the way that you do an affine transformation matrix by its own mini network transforming these point cloud set of 2048 xyz inputs into a canonical representation so affine transformation is going to preserve the collinearity of these points and project it into kind of a more standard matrix feature space that we're used to seeing in deep learning and so here are the actual details of how they construct this t-net they have this orthogonal regularizer which is going to try to push the distance between the features and then the features transpose subtracted by an identity matrix which is just a matrix that has ones along the diagonals and zero everywhere else this is just the regularization technique that's going to be you know i don't know the exact motivation behind this kind of operation but you know i imagine it's trying to keep the diagonal dimensions uh orthogonal distance from each other with the way that you transpose matrix and multiply them together and then subtract the identity matrix along the diagonal so that you have these different dimensions of the matrix so obviously there's more linear algebra behind that and you know that's a horrible summary of it but so here's the actual implementation in chaos that gives us a more tangible sense of what's being done here so we have our orthogonal regularizer where we have the number of features the l2 regularization which is where you just square the distance and then take the square root of the sum of that and then we have our identity matrix so we're going to reshape our intermediate features so they're compatible with this you know matrix multiplication where you multiply the intermediate features by itself transposed and then you're going to do the reduce sum where you have this operation of multiplying the the l2 regularization which is where you're doing that squared then the square root of all that and then by the distance between this transpose and itself multiplication minus the identity matrix so this is how we're going to be regularizing these intermediate features and then we see how this is all compiled into the t-net block so we start off by having our bias parameter this constant another identity matrix with a number of features and it's fine into a vector and then we have our orthogonal regularizer which we see is into our dense layer as we're projecting this into the final number of features so we stack together these blocks we have a max pooling where we're reducing the dimensionality it's kind of like kind of like just stuff just template stuff as we add these sequential processing layers and then we reshape it into say 2048 by 2048 this matrix and then we do this dot product between the inputs and then the features that were constructed from our t-net so we do kind of have this uh like skip connection operation with these transformations so you know it's a bit complicated and i'm not sure exactly what's going on here but you know here's the high level overview and you know if you're interested in this further i recommend checking it out and going deeper than this quick tutorial okay so that was a bit of a fail with trying to understand these t-net blocks maybe we'll see how it all comes together and it'll make sense then so we construct our architecture by just repeating these blocks together x equals t net the com batch norm activation blocks and then another t-net block and then finally we have max pooling dense layers and then mapped into our classification layer so now we see the model in keras model.summary so the input layer we understand that that's the 2048 point cloud each with an xyz coordinate and now we have our first tnet block so this first stack of operations that ends with this dot product it returns 2048 by 3 as a result of our tnet block so this is the output as we saw the last uh dot product it's layers that dot this is the last thing we're going to do in our tnet block so we see that in our t-net block we have this blow-up of the features or we just use a one-dimensional convolution that goes from 2048 by 3 to 2048 by 32 batch normal activation another stack of that as we saw as we're repeating that in our architecture and then we have so this dense layer here and here is where we applied this uh is where we're applying this orthogonal regularization or not actually the first two the last one where we're projecting it into the largest space so let's see where that was so that is the one so that is this no that's this dense layer nine so the number of features is actually the three by three referring to the xyz coordinates the number of features this parameter of our regularization in our tnet blocks is about the xyz points not the 2048 sample points so we end up with this 9 then we reshape it to 3x3 dot product with the original 2048 as we skipped ahead this original representation so overall it's you know in my in my understanding it's kind of like a black box set of operations but you know you can check out this paper if you want to really understand you know why they're aggregating these transformations in this way and you know i'm sorry i didn't quite have the interest to really dive into this but you know here it's somewhere in this paper if you really want to mine through it and figure out the motivation behind why they you know why exactly they're stacking these operations like this and their motivation behind projecting this from 2048 back into 2048 by three and you know the motivation behind these intermediate operations so that was our model and it's a bit complicated and i don't completely understand it but finally we're putting it together and we're back to the familiar territory of our keras modeling because at the end of the day we have our input output space we're just fitting these labels with these points point cloud coordinates and we have some models some way of transforming from this original 2048 some a sequence of these layers and then finally we have an output 10 classes for predicting 10 class classification so we use our sparse categorical cross entropy loss function we have an atom optimizer and now we're just fitting the data with model.fit 20 epochs taking about 10 seconds per epoch we see the loss monotonically decreasing and our accuracy is pretty good about 80 at modeling these object categories from these three-dimensional point clouds which is a really exciting result really high accuracy for modeling this with you know about 700 000 parameters and only 20 epochs through the data and it does look like if you train this for further it would do even better at modeling the training data and then we do see that the validation set accuracy is decreasing as well although it's more uh jumping around we see how it jumps from 76 percent here back to 70 and so on but you know it does seem like it hasn't overfitted quite yet so finally we have a trained model and we want to visualize some of its predictions on the test set so we do this with our tf.data.dataset.object.take is how we sample the next batch of data then we're going to index it to take the first eight points which we're going to visualize and then we're going to run them through the forward pass of our model and then we're going to take the arg max which is the you know we have this softmax distribution of the predictions and the arg max is the most probability density that the prediction had put on any given uh class label like table sofa desk and so on so we have the points equals points at numpy we convert it into a into a numpy array from our i think we originally have a try mesh object that's or let's go back to this data loader or okay so i'm not sure exactly why they need to convert it again to a numpy object i think it already is that but now what we're doing is we're just plotting the x y points of each of these individual point cloud objects and then also titling them with the ground truth label and what they have been predicted so we see how our model is pretty good at predicting these things table table bed bed chair chair and so on so overall that's our presentation on point cloud classification and keras code examples so to summarize all this i hope this was a decent introduction to point cloud classification and i hope i didn't mess up walking through this chaos code example too much mostly i was pretty confused by these this pointnet architecture and trying to understand exactly what's going on with this t-net transformation and how they're using this orthogonal regularization thing showing the identity matrix minus the feature matrix transpose you know exactly why that's used and the motivation behind it you know i wasn't so sure of that but i think from this keras code example we're able to at least have a sense of how to load these point cloud objects these triangular meshes this library and how we have this file path these model net 10 data set you can see the um the project page which will be linked in description of this video as well uh and then i mean we can really just treat this as a you know regardless of this pointnet architecture which is a bit confusing we can just treat this as any mapping of inputs from 2048 by 3 into these 10 object categories so how you do these intermediate processing you know is up to you and i think like maybe a transformer layer for this unordered set of point clouds you know something like that could be interesting i don't think this is the only way of doing it and then we also saw this uh data augmentation which i think is interesting this jittering and then the shuffling of the points to make it have this permutation and variance with the in the data space data augmentation then overall we saw just you know with this architecture fitting it and how it performs really well in only 20 epochs we get 80 training set modeling accuracy 77 on the validation set and then when we visualize our predictions uh you know they look correct and then so also this is just the syntax if you want to take this apart for how you sample this point cloud representation then just plot the xy points in this two-dimensional plane in matplotlib so overall i hope this is a decent introduction to point cloud classification and inspires your interest further and again if you're interested in this i highly recommend the next step would be checking this out and you know me personally i have never done this in my life before this is my first time actually looking at a you know one of these point cloud or even you know a vox with this well i've seen this um we've previously looked through this three-dimensional image classification where you do have this volumetric object and you use the three-dimensional convolution but i think that's really different from the way that data is represented in point clouds and again if you're curious this blog post from geeksforgeeks uh describes vector versus roster graphics and you know i think that further communicates this difference between point clouds and then uh like voxel representations and 3d volumes so thanks for watching and please check out the rest of the keras code example series on henry ai labs