Persistent Homology | Introduction & Python Example Code

Key Takeaways

hey folks welcome back this is the final video in a three-part series on topological data analysis or TDA for short in this video I'll be talking about another specific technique Under the Umbrella of TDA called persistent homology the big idea behind persistent homology is finding the core topological features of your data that are hopefully robust to noise I'll start with a brief discussion of key points surrounding persistent homology and then dive into a concrete example with code of how to use it and with that let's get into the video there are many layers to persistent homology so I'll try to start super simple and build things up in a way that hopefully makes some sense like I've mentioned throughout this series TDA is all about looking at the shape of data so let's go back to preschool and talk about shapes or more precisely polygons like the ones shown here but not all polygons are equal there is one that is special and the reason it is special is because it is the simplest polygon we can construct the triangle and one neat thing about triangles is that we can use them to make any other polygon for example a square is really just two triangles stuck together a pentagon can be made from four triangles like this and a star is just the same Pentagon but with five triangles coming out of it so one thought is if we want to analyze the shape of our data maybe we can break it down into a bunch of triangles well as it turns out this is essentially what what we do in persistent homology but with one technical detail since most data sets live in more than just two dimensions that's to say we have more than just two variables flat two-dimensional triangles may not capture the full richness of our data's shape don't worry like most things mathematicians have generalized the notion of a triangle to any number of dimensions and they call these generalized triangles simplexes so the triangle that we know and love is called a two Simplex since it lives in two dimensions a line segment is the simplest shape we can construct in one dimension and it's called a one Simplex similarly a tetrahedron is called a three simplex and a point is a zero simplex and so on for all the other dimensions so just like a collection of triangles can make any two-dimensional polygon a collection of simplexes can approximate just about any complicated high-dimensional shape that may underly our data and so since you'll probably see it elsewhere the technical name for a collection of simplexes is called a simplicial complex and this is a key Concept in persistent homology okay so this gives us a clue as to how we can take unstructured Point clouds in other words data sets and translate them into shapes so now let's talk about how we might compare shapes together no matter how different or complicated they may seem so one way to do this is by looking at holes for example these three objects shown here we have a Taurus a loop and a coffee mug so while these may appear to be very different shapes they have something fundamental in common they all have a hole and this is like the joke that a topologist looks at a coffee mug and a donut and sees the same thing the reason being that one can continuously transform one into the other for the fadas out there this is called a homeomorphism but the fundamental thing here is the number of holes so one way we can characterize and group shapes together is by counting holes and just like before when we generalize triangles into simplexes we can generalize holes as well we can think of cavities as holes in 3D and we can think of singly connected components as holes in 1D and so these generalized holes form the basis of what are called homology groups and these give us a formal way to characterize different shapes so when we talk about homology we are essentially just talking about holes okay so now that we've talked about constructing shapes with generalized triangles and characterizing those shapes via generalized holes we can finally talk about persistent homology and the first step in persistent homology is to convert data into a simpal complex to see this consider a data set I.E a point cloud like this and one way we can construct a simplicial complex out of this is by drawing n dimensional balls around each point and since our data here is two-dimensional we just draw circles around each point which might look something like this so at the center of each of these gray circles we have a point we can form one simplexes I.E line segments by connecting the data points whose corresponding circles overlap which might look something like this and so now we have two shapes we have our original Point Cloud which is indeed a simpal complex where each point is a zero simp Lex and the shape we just constructed made up of both zero and one simplexes and then we can compare these two shapes by looking at their homology more specifically by counting the number of connected components which corresponds to the h0 homology group that we talked about in the previous slide and there we go so we can see that in our first shape on the left we have 20 separate connected components while on the right here we have 13 singly Connected components but there's nothing special about this radius Epsilon sub one so let's do this again but with bigger circles now we can start to see two simplexes appear in other words triangles and the number of connected components decreases but still there's nothing special about Epsilon 2 so let's go even bigger and now we see three simplexes appear I.E tetrahedrons and so on and so forth however there is a special radius value here which is when every circle overlaps with with every other Circle and we are just left with one big connected component and this is a natural limit to this process as we can see with each of these simplicial complexes the shape of our data is evolving and its evolution is captured and Quantified by the number of connected components in other words by the change in its homology so although only four different choices of radi are shown here corresponding to the four different shapes on the screen we can do this for every choice of radius between zero and the limit I mentioned earlier so this gives us a way to sus out which topological features of our data are significant based on how long they persist during this circle growing process in other words the holes that persist over a large increase in radi are more significant than the ones that persist over just a short period okay so how can we track the Persistence of these holes so so one good way to do this is by using a persistence diagram these look something like the plot on the left here which is showing the persistence diagram of a hollow sphere and looking at the plot each of these blue orange and green points corresponds to a topological feature or in other words a hole in blue we have the H Subzero homology Group which are the singly connected components in Orange we have the H1 homology Group which are closed loops and in green we have the H2 homology group in other words W cavities the x axis of this plot indicates the radius at which a hole appeared in the evolution of the data's shape in other words in this circle growing process that we showed in this previous slide and on the y- AIS we have the radius at which that hole disappeared so therefore a point that sits near this black dashed Line This yal X line corresponds to a hole that disappeared soon after it appeared conversely points that sit far away from this line represent holes that disappeared long after they appeared therefore two key points of a persistence diagram are the points close to this yal x line are noise while the points relatively far from this line are significant so in this example we have two points that are far from this line the blue one in the top left here and the green one right here so we can ignore this blue one here because this corresponds to when every and dimensional ball overlaps with every other ball so the significant topological feature of this data is captured by this Green Point here which is telling us that the data is characterized by one cavity and this makes sense since the data for this example are organized on the surface of a sphere okay so up until this point I've discussed only toy examples and meant to give you an idea of what's going on with persistent homology so now we'll switch gears to an example with with real world data so in this example we'll walk through how one could use persistent homology to analyze Market data and I suppose it's worth mentioning that this example is not meant as Financial advice I'm a physicist not a Trader never taken a finance class in my life however I hope this example gives you an idea of what an analysis using persistent homology might look like and Inspire ideas for analyses using data that you might be working with okay so similar to the last video we start by importing python libraries the notable libraries here are y Finance which gives us an API to grab Market data and the riper and pum modules which are part of the same pyit TDA ecosystem from the last video next we load in Market data over a 4-year period using Y Finance here we are grabbing four major Market indexes namely the s&p500 Dow Jones NASDAQ and Russell 2000 we have daily prices for these index organized in a pandas data frame so you can imagine four columns for each market index and many rows corresponding to each day that the markets were open over this 4-year period then we convert this pandas data frame into a numpy array and compute the log daily returns of each index and this choice of data prep follows the procedure used in the paper by gidia and cats which was the inspiration for this example and you can find it at the archive reference here okay so now we get into the TDA SEF so in this analysis we want to track changes in the shape of the markets by looking at how the homology of the market changes over time so to do this we start by initializing this object that constructs simpal complexes from data next we Define a Time window size which will allow us to grab a chunk of data to analyze the homology of so here we're sending this window size to 20 days next we Define the total number of these chunks we will have and finally we create an Umpire rate to keep track of a number that quantifies changes in homology okay next we go down to this for Loop and we do some persistent homology so first we take the first 20 rows of data to do persistent homology and create a persistence diagram that is we grow four dimensional balls around each point where each choice of radius creates a simplicial complex and we track the holes that appear and disappear using a persistence diagram so we do all that with just one line of code and we do the same thing but now for another set of 20 rows specifically the second row all the way down to the 21st row so now we have two persistance diagrams corresponding to two overlapping 20-day windows in which the market was open so next we can quantify the change in the overall homology between these two persistence diagrams using something called the washer Stein distance which is essentially a distance measure between two persistence diagrams so at the end of this whole process we get a single number and store it in the numpy array we created earlier then we repeat this whole process for all the rows in our data set okay so after this whole process we have a set of values which quantify the changes in homology between consecutive days that the market was open and so we can just plot this as a Time series which is what's happening in this block of code here and the plot will look like this blue line here which we can see there's this clear peak near the middle of the time series and then for some context we also have scaled S&P 500 close prices plotted in Orange just above and this vertical red line here is indicating when the crash of 2020 occurred and then as it turns out the peak in this waserstein distance time series seems to correspond very closely with when this crash occurred so did homology changes predict the crash of 2020 well I wouldn't go that far but this is indeed interesting one idea to investigate this further is one could try to use these waserstein distances to predict future market index prices so if past distance values predict future index prices then maybe there's something here so as you may be able to see from this example there is a lot of room for creativity when using persistent homology in practice and in some sense this is more art than science so that bring brings us to the end of our three-part series on topological data analysis A TDA is a young field with a lot of untapped potential so I hope this series was helpful in getting a better idea of what it's all about if you'd like to learn more check out the other videos in this series Linked In the description below there's also a corresponding medium article to this video and the others in this series which you can find in the description if you enjoyed this content please consider liking subscribing or sharing this video like many of you I am indeed still learning so if you have thoughts questions or concerns please feel free to share those in the comment section below and as always thanks for watching