Comparing Time Series with HCTSA

Key Takeaways

this is data skeptic time series the podcast about how to predict the future based on historical sequential data episode number [Music] if you've been listening this season you're aware there are no shortages of different ideas and algorithms for doing time series analysis there's also absolutely no shortage of time series problems which seem to occur in just about every industry and context that makes it even more important to compare different methods on different data sets figure out what approaches are robust or at least understand the nuances of when they work and when they don't or maybe i should say when they work better and slightly worse the highly comparative time series analysis or hctsa is a code framework that helps to do that it enables the extraction of thousands of time series features and gives a range of tools for analyzing them in this episode i speak with ben fulcher about hctsa [Music] ben welcome back to data skeptic no thanks kyle thanks for having me and remind listeners about your background and area of research yeah so i'm a physicist with background in physics and i also study time series analysis and neural systems one of the areas you've worked on that i wanted to zoom in a little bit on today is hctsa the highly comparative time series analysis and the software package that accompanies it could you give listeners the high level on what it is so this it's a bit of a tongue twister hctsa the idea went all the way back to my phd when we saw that many people in different parts of science were all using and developing different types of methods but with minimal comparison so someone could be developing a really exciting new method in astrophysics and they'll just use it on their astrophysics data and someone working in biomedicine might have a really cool new method for analyzing the dynamical structure of heartbeats and they'll start sharing their method and applying their method to heartbeats but there were kind of big barriers to sharing methods across different fields even though this is what as scientists we should be doing is trying to find methods that can help each other solve problems and so our aim was to do kind of what the genetic micro array did for genetic analysis or in the past you know people might have their favorite gene and really go hard studying what that single gene does and spend their whole careers on a small number of genes and then with the molecular genetic microarray you could suddenly measure the expression of thousands of genes at once and kind of have this more global perspective of what all the different genes are doing and so we imagined having the same approach applied to time series analysis whereas instead of subjectively picking a handful of methods to work with you could actually zoom out and measure simultaneously the results of methods from all across science so we developed implementations and took implementations of methods from as many different parts of science as we could and put them in one unified framework and we call this approach of using many different scientific methods to solve a problem time series analysis as highly comparative time series analysis and we made this available in the software package hctsa and i don't know if you have an official count or just a broad generalization but how many different methods are baked into that library in total over 7000 features are constructed so each time series analysis method is coded in a way that takes in a time series and outputs a single real number summary statistic and there are over 7 000 of those summary statistics and they could be things from you know what's the variance of the data would be a simple one or what's the order correlation at lag one would be another simple statistic is the mean different in the first half to the second half would be some measure of stationarity how well can i predict the time series using a simple model of its past and which models do well all these can be coded in the form of summary statistics that take in a time series and output a single real number that summarizes some aspect of the structure or the patterns in that time series so there are over 7000 of those but of course the number of conceptually distinct method is less because you know take the autocorrelation example you can extract it at lag one at lag two lag three you can see on which different time scales the time series is correlated to itself and so you know the number of conceptually distinct methods is lower maybe about a thousand but the number of unique features that you can pull out of a time series is sitting at about 7 700 at the moment one of the use cases that jumps out to me personally is to take that as the auto feature engineering process for something in machine learning if i want to do some sort of time series classification let's say the raw data might not be enough but if i convert every observation into a feature vector of 7000 and i've got a really big data set maybe i can skip the whole feature engineering phase do you see that as a common use case that people are trying yeah this is something that many people use this for in classification and one of the first things we did was try it on classification problems and yeah the parallel is interesting i mean in this case the features that we're using are kind of selected from methods that scientists have developed that has some benefits in that you can take one of these features and there's a kind of deep theoretical literature and community of people that have used these types of methods you know an entropy method comes out as really useful for solving your classification problem you can then go and learn about information theory and entropy or if some garch model comes up as as helping distinguishing your data you can go and read into the arch modeling community so the features become interpretable which is something that the deep learning approaches don't have and and it kind of draws on the existing best methods that we as scientists have developed for structure and time series and therefore it's a kind of a guided constrained search of that feature engineering process some of those features we've talked about lag entropy auto correlation they're familiar to me with statistical training that i've had but as you'd mentioned you're pulling from all different areas of science are there any domains that have very custom things like i don't know do people who study heartbeats have their own set of methods that i wouldn't know so i tried to be as even-handed as possible in covering as many different literatures as possible and in heart rate this is a kind of a community of people took some methods from physics and so they'll they'll try and understand kind of scaling power law scaling properties so thinking of the time series as a fractal but do the fluctuation patterns scale with rescalings in time and they have their own kind of toolbox of methods about identifying pathological heartbeats like they'll see they'll extract different heartbeat intervals and they'll see if successive intervals are larger or smaller than a given threshold is a common method called pnnx that they use there and so when you go through different types of literatures you'll often find these literature-specific tools and methods that they've developed to solve their problem and then you never really know if that type of approach will be useful for a different problem unless you try it and so this was our philosophy of trying to be as comprehensive as possible in including different types of methods does having 7000 features open me up to overfitting or spurious correlations or things like that yeah this is always uh one of the worries you know we've applied this to a small number of time series data sets in which there are a small number of time series in each class and you know even controlling statistically for comparisons across 7000 we can still be confident in isolating small numbers of time series features that are significantly different between those two classes but when you've got a small number of time series in each class and they're noisy then you're right that when multiple hypothesis correct across thousands of different things you may end up with no signal because of of that scale of comparison and so in one hand you can see this is just being honest about the number of comparisons someone could do and scientists in the past have probably behind the scenes compared things and then only reported on the ones that worked and so you could imagine this just being upfront about all the comparisons that you're making and making sure that you're statistically controlled in any positives that you do report in the literature and the other approach is to just reduce the number of time series features that you actually compare and so we showed in our original work that of those 7000 features or more there are only about 200 features could approximate the behavior of all 7000. so this points to a large amount of redundancy across the scientific literature methods that exist in one kind of domain might have a similar match in another domain that might have slightly different names but overall have very similar performance and so we had some follow-up work that's tried to create reduced feature sets for example there's a catch-22 feature set which just has 22 features that give the best performance from those 7 000 initial candidates you can reduce it down to 22 that still give high performance on classification tasks and reduce that potential for statistical comparison and overfitting well let's put ourselves in the shoes of a scientist so obviously they've got some amount of math and stats background but maybe that's not their primary training if they're a biologist or something like that but they've come up with a time series data set probably they have some expectations for what features are important or work but they're curious to see if there's insights that they could get out of applying this solution to their data what is that experience like where do they start and what are the kinds of things they're going to learn i always recommend using a smaller feature set first i guess it depends on the size of the data set but they might say well these are the two classes of data in a classification problem say that i want to compare and they can take our features if they've got their own they can compare theirs but they could kind of run their feature extraction process through our pipeline so each time series in their data set gets turned into a set of features either the 7000 features in hctsa or the 22 features in catch-22 and then they've got a very familiar setting for people who've done machine learning or multivariate statistics where you've just got all of your time series represented as rows in a matrix and all of your features as columns in that matrix and then you can fit whatever statistical learning model you want see how well leveraging diverse features simultaneously can do it classifying your different types of data you can also compare columns one by one and say well which individual features are the best at classifying my different types of data and you can sort all the different features according to how well they distinguish your types of data and you may discover that or the best feature for distinguishing my data is something very unexpected to me right some method from nonlinear physics or you know the second best feature is something from stationarity or these types of insights you can get just automatically it kind of saves you the trouble of manually trying lots of different methods it just automatically tries large number of methods and then pulls out the ones that it says you should look into further because they're performing well on your data set in that exploration process am i using hctsa as a library and kind of executing it in my programming interface or is there a ui to it what's the experience of guiding myself through those learnings hctsa is a matlab based software so there are tutorials for how to use it load your data in do the feature extraction and it also has functions for doing some basic statistical learning and visualization one of the reasons that that can be a barrier is not everyone has access to matlab so the catch-22 feature set has native implementations in python r julia all of these kind of c coded features so it runs very very fast and it's available in all different coding libraries but yeah you need to pick a coding interface to run these feature extraction methods and have you seen any adoption in particular areas of science are there fields that have been ready to pick this up and gain insight from it yeah so it's always a bit surprising when you get the kind of alerts on people that have used this approach and it's always an unexpected field you know from people trying to find signatures of different energy usage in fringes and distinguishing that kind of thing people looking at accelerometer data and seeing what distinguishes different types of motion there asphalt engineering people looking at the dynamics of neurons and different types of neurons and which different types of time series properties distinguish the neurons people have used it in speech so can i predict whether someone speaking has parkinson's disease or is a healthy control so yeah very many different applications in engineering biomeds lots of neural dynamics i think you've had the opportunity to see this run on a wide variety of different scientific data sets are there any that stand out or any particular insights you've learned through that process i mean our work has worked a lot with brain imaging data so a lot of my own interest is is understanding the structure and organization of the brain how the brain structure allows different types of emergent dynamics of neural dynamics and whole brain interaction that give rise to these interesting patterns that we can measure in the brain and so some of my favorite applications have been to brain imaging data where we've used it to find new ways to quantify the structure of brain imaging data in particular how it varies as a function of brain region and as a function of how those brain regions are interconnected rather than coming at it with a specific time series analysis method it's very hard to motivate in a context like the brain where you don't have a good model of what's happening you just have to come at the data and try to find interesting patterns it provides a good way to kind of in an unbiased way say what types of methods are useful for picking up the pattern that i'm interested in for example which brain regions are highly connected and to those dynamics of brain regions that are highly connected differ from those that have very few connections in the brain we can just come at it with these tools and pull out interesting ways to quantify that structure that we're interested in visualizing a vector of 7 000 plus features is a challenge but you guys have risen to the occasion and produced this kind of heat map type plot that i think is effective especially when you look across data sets could you describe a little bit about that and its creation since this is an audio podcast i know i've challenged you there but how can people read some of this data set at a high level yeah this is the first thing that we kind of looked at and you could imagine as this was growing from you know the very first incantation where i had eight features two you know every week you can imagine me sitting in a dark room as a phd student slowly grinding this number up and up until it got to over seven thousand and as it grew kind of the columns became harder to visualize because with eight columns you can kind of look at each one individually and so i guess the listener should be thinking of a heat map with some number of rows where every row is a time series every column is a feature and then the output of each feature on a time series can be represented as high or low as a dark or a light color for example as entries in that matrix and so it becomes very hard to see the structure when you've got thousands of columns because you've got all these tiny little pixels each represent the output of a feature on a time series and so what we did was we kind of reordered this is a common technique in visualization now reordered all the columns and rows to put similar columns close to each other and similar rows close to each other and so now we can actually visualize groups of features that all have similar behavior as creating quite striking patterns when you look down this heat map and similarly across rows you can see time series with very similar properties as measured by thousands of different time series analysis methods and so the structure of that matrix often gives you some interesting hints as to the structure of your time series data set and also the behavior of lots of different types of time series analysis methods on that data set and that can then motivate other types of visualizations like looking in low dimensional feature spaces and seeing how your data is embedded in low dimensional feature spaces yeah i noticed those patterns is being quite striking in a lot of the things published on this work it almost seems to me that someone who is a regular user of this as maybe you're the best example could develop kind of train their eyes to see things like oh the wavelet features are over there i bet that's an audio data set or something along these lines do you envision that visualization being kind of a fingerprinting tool in that regard yeah i mean there's a lot of structure in there and a lot of students in my group are still working at ways to pull out relevant pieces of structure in these data sets the visualizations you know can be beautiful and give you a sense of just how broad the number of scientists working on time series has been through history and all the different types of methods that they've developed i mean you could think of every column in this matrix as you know one scientist's little contribution to the field right they've come up with a method they've encoded it and now it's a column in this matrix and there are 7 000 such columns and so you want to treat every column with respect if you think of it like that but yeah in terms of of thinking about you know how different types of data come up with different signatures across all these different methods is a really interesting way to view the field rather than zooming in on just the wavelets as you mentioned or just the entropy measures it allows you to kind of find complementary groups of scientific methods that might work well on your data i'm wondering if we can zoom in on the parkinsonian speech example if we took a different example something like i don't know sunspots i have a sense that those have been very well studied and it's almost like a softball to your project if you couldn't measure sunspots what are you working on but parkinson's is something that is somewhat rare maybe not as well studied it's not obvious to me that you would have the features out of box that would be useful for it what do you see when you put data from speech of potential parkinson's patients through this process yeah so this was a challenging one and there was a bit of literature out on it before and we just kind of wanted to see whether we could get close to the state of the art in that field and so the existing features had kind of taken speech analysis features like you know measures of jitter and they have ways to pull out the audio series and run their algorithms and get a statistic called jetta say they've got other types of features that they'd used and the field was starting to embrace non-linear techniques pulling out things like fluctuation scaling patterns and other types of attractor reconstructions and looking at dimension estimates and things like that so we just wanted to use it as a way to see what how well could we do on this quite difficult task in which as you say there's kind of no obvious approaches in fact you might think that it really requires a specialty hands-on expert to kind of tackle this task who's used to working with speech data but yeah we found that our approach could get close to more consistent with the state of the art on this problem and it was pulling out some quite interesting and unexpected methods including some that that i developed and just threw into the library not knowing if they'd ever be useful which as a side note this is a another useful benefit of this approach is that if you come up with a new method you don't necessarily need to have found an application on which it works well you can just throw it into the mix and it becomes 7001 and you've added your extra one to it and i did this a few times when i just come up with a method and go i'll just throw it in and then years down the track it might come up as being the best method on some new problem that i've tried and this was an example where some methods that i had implemented on simulating physical systems and driving those physical systems with the time series actually came up as the best features for this parkinson's disease classification of speech because it was picking up very slightly different patterns in the oscillatory patterns when these patients are saying like ah or kind of these types of vowel sounds you can pick up these slightly different patterns in those signals so it was a very interesting application that pulled out both measures that would be expected to someone in the speech analysis literature and a bunch of kind of unexpected but useful methods as well the benefit of applying many features at once is that you can distinguish a new method that may be unexpected is it actually doing something new because often we find a method that you know no one's ever applied to this data and it's performing really well but it's highly correlated to a method that already exists in that literature so it may just be that a method from economics that's doing really well in parkinson's speech you could write a paper on it but if you actually compared it to the things that people already do in that field it's highly correlated to something that already exists and so it's not really new it's just got a new name and so when you compare across thousands of methods you've got to be careful not to just pull out the ones that sound interesting but actually reproduce the behavior of something that exists and so our analysis also pulled out unexpected methods that are redundant given what's already known versus unexpected methods that are also contributing something really new to the state of the art thanks to this week's sponsor master class with masterclass you can learn from the world's best minds anywhere anytime and at your own pace i signed up for masterclass and when i got to the category section you know what i clicked on science and tech the first master class i signed up for is chris hadfield's space exploration course you guys know chris hadfield he's an astronaut that got a lot of popular press very eloquent speaker and his master class on space exploration not only did i learn a lot i learned it from the guy who went there and did that i don't know space is the final frontier but clearly we're moving into a new space era i want to keep up with that and this master class was a great foundation for me with over 100 classes from a range of the world-class instructors they use there's always something you'll want to learn more about this holiday give one annual membership to get one for free that's a great deal if it was me i would keep the free one for myself give the gift masterclass.com data skeptic that's where you get that deal that's masterclass.com slash data skeptic terms apply [Music] [Music] if you've ever wanted to make your home feel safer there's no better time than now this week our friends at simply safe are giving data skeptic listeners 40 off their award-winning home security linda and i love simplisafe because it has everything you need to make your home safe indoor and outdoor cameras comprehensive sensors all monitored around the clock by trained professionals who send help the instant you need it simply safe was even named the best home security system of 2021 by u.s news and world report you can easily customize a system for your home online in minutes and even get free custom recommendations from simply safe so yeah forty percent off these are simply safe's biggest discounts of the year you can get a complete home security system starting at just a hundred dollars there are no long-term contracts or commitments it's a really easy way to start feeling a bit more peace of mind take advantage of simply safe's holiday sale and get 40 off your new home security system by visiting simplysafe.com skeptic again that's simply safe dot com slash data skeptic for forty percent off your entire system hurry this offer ends soon [Music] so when looking at that detection for parkinson's you were describing and you were able to say it was detecting this aspect of the phonemes of the speech can't remember the precise terms you used but you sussed that out somehow of the model itself can you walk me through that process of getting that interpretable insight from the you know long vector of features yeah so once you've got a handful of features that are the best performers and that are unique from each other so this is generally the first step in trying to get the interpretable insight you might pick the top 40 features say that did the best on your data you organize them into groups that perform similarly and then try and analyze what each group is doing so sometimes you'll get a group that's all looking at linear correlation structure in slightly different ways but more or less they're all doing the same type of thing they're capturing linear correlation structure others might be looking at predictability in a nonlinear way with say entropies or things like that others might look at the variance of the data right so you can say oh these are the different groups of types of ways that i could analyze my data that are performing really well and so in this one we had this method that was really unique and performing the best and that was looking at the kind of mean of this particle that was being driven by the time series under some sort of physical laws and so you know speech patterns that are asymmetric would be pushing the particle in one direction more than the other direction and you could measure that as a mean offset and so that's the second stage is once you've found these groups of features that are capturing kind of conceptually different properties of your time series that are useful for classifying it you can then zoom in on one and then look at the algorithm and visualize what it's doing and then try and understand oh okay look it's finding these asymmetries in the data or you can start to get mechanistic and really zoom in on what's happening and i'll mention that this is kind of a reversal of the typical time series analysis textbook will have on its first page the step one is you know look at your data step two is you know think about the assumptions of the methods before you apply those methods to the data and we're kind of reversing that process in that you don't look at your data you apply all the methods to the data before thinking about whether those assumptions apply to your data or not and then at the end you pull out the methods that actually worked well and so the kind of burden is on the scientist at the end to go every feature that worked well then you have to go and use your domain expertise read up on the literature does it make sense how is it working why is it working so yeah you don't get out of as a scientist you don't get out of doing all of that hard work you just do it at the end of this process where hopefully by doing this highly comparative analysis it's pointed you in the directions that you ought to look rather than you having to guide that process manually do you find that that's at all controversial in you know philosophy of science departments is this looked at as a backwards way of doing science at all or are there trends in this direction oh well i was shouted at some number of times during my phd for this not being how people train the scientists to do their work where there really is that tradition of following through the steps doing things one at a time being very careful and that makes sense right because there's a potential to misuse this type of work if you don't know what you're doing and you throw something through what is and ought to be a very subjective careful thoughtful process just through an automatic method and just simply pull out the top ones and over interpret them or don't do that kind of proper care there is potential to misuse approaches that try to automate something that ought to be careful and thoughtful and so i guess my job is really just to encourage people to at the end of the process put in that can thought i mean it's it's not to say that if something has a potential to be misused it shouldn't be used it's just to encourage people to use it well and in this case that means if some a method comes up and you want to make a claim about it you should really think carefully about what it's doing how it's working and the assumptions of that method rather than just treating it as a blind kind of feature extraction method you really should treat it as a way that guides a thoughtful data analyst in the right direction and then it's up to that data analyst to use that information in the best way they can along those lines uh if i came in with the wrong intentions i was you know motivated by like some political point of view or something like that what's the expression if you torture the data long enough it'll tell you whatever you want to hear i think there's a certain truth to that here but if i were just an average scientist who wanted to try this tool in sort of a broad brushstroke exploration step what are the chances i would accidentally make some mistake well if you follow my uh instructions carefully you should and i guess i've you know even recently i keep updating my kind of how to be careful how to interpret i mean there are hundreds of pages of documentation that we've made that try to guide people through every step and kind of list kind of pitfalls and little words of caution as to how to interpret things but also the tool itself when you try and say you know through hctsa if you put your data in does the feature extraction and you run this function it says tell me the top features on my data set it will do under the hood all the statistical learning that will kind of say you know oh you don't have the statistical power to actually say there are any top features for this data set so we've tried to be careful algorithmically in doing the statistical multiple comparison correction and making sure through the software that these errors aren't made and we've also put a lot of care into these hundreds of pages of documentation that try and guide people through the pitfalls and how to use this properly so there's seven thousand some features that can be generated today do you have a vision for what that looks like in the future are you basically finding them all or will it be 8 000 in six months and so on and so forth yeah it has grown and it also has shrunk i mean the original paper that we wrote on this had over 9000 and sometimes i'll go through and say well which features are like actually redundant such that you know one feature exactly copies or almost exactly copies another feature that exists and that's different you know in our first goals of this process we wanted to incorporate everything because one of our aims was to see how redundant is the literature other methods in economics kind of you know reinventing the wheel of what already exists in astrophysics and they are reinventing the wheel of what people are doing in ecology and so we wanted kind of redundancy we wanted to copy up methods but when you're actually using this as a feature extraction tool you kind of don't want this high level of redundancy and so if it's a trivial level of redundancy i've removed and then we got down to kind of 7000 so 7 700 now and then sometimes a new method will come up and i have a process that i follow called the candidate feature lab where if a new feature comes up in the literature and they want to add to our library we'll run it across a thousand different time series see how similar that new method behaves to what already exists and if it's doing something sufficiently unique i'll add it to the library if it's reproducing something that already exists i'll say well done on the paper but you know this method already existed you know someone developed something like this 20 years ago in this literature and that's another kind of key part of what the philosophy of this comparison is that in the literature people are often developing new methods and publishing them without a lot of comparison to what exists and it's indeed difficult to compare a new method to thousands of different methods from literatures you've never heard of and so this makes it easy that you can compare the behavior of your new method on a wide variety of scientific data and then see which methods across science perform similarly and that can point to redundancy or in the better case scenario can point to oh my method really is doing something that no other method in science that's ever been developed is doing and actually can highlight useful forms of novelty have you been able to identify any interesting affinities in that regard do you i don't know astronomical data somehow correlate with data about phish movement and things like that yeah so the data comparison side is the other end of things where we can represent each piece of data with the behavior across not just one or two hand-picked features but across thousands of features and so we could say two time series are similar if they have similar properties as measured by there are thousands of very different types of methods and so we've also done work on organizing different types of time series i've talked about this with you before about the comp engine and there yeah we do find some interesting connections between different types of data often it's sensible the same classes of data are clustering like rainfalls clustering with itself or rainfall is clustering with time series models that generate rainfall-like data you know financial data or clustering with other share prices and with stochastic models that have been used to model financial data and so a lot of those connections are similar but every now and then you will get a weird one like some brain signal clustering with a you know burke song or you know some sort of weird thing that you have to kind of look at the structure of what's happening but normally it's it's fairly sensible fairly sensible matches but i mean the other side of that on the method side is that we can also organize different types of methods and so we can find new types of analysis methods for time series analysis and see well across all the different other methods that could have been used what are the most similar you can use that to kind of contextualize any method that someone might be using in a hand-picked way let's imagine i have a very very large data set how is hctsa going to scale as i want to push more and more data through it yeah so how large oh i don't know a petabyte let's go with a petabyte yeah yeah i mean so there are two aspects to a large time series data set right one is how long is each time series in that data set and the second is how many time series do you have for hctsa if you've got say a thousand sample long time series it might take something like 20 30 seconds to compute all 7000 so time series features if you go with the reduced feature set that i mentioned before that we hard coded in c and made available in python julia etc i mean that runs in a fraction of that it would run in milliseconds and so you know this is where you can select the right method for your approach if you're doing something on a very large scale where speed and storage space becomes an issue then working with a fast c code or in a streaming application say where you actually want to compute these features quickly in real time and maybe just sacrificing the breadth of comparison going from 7000 down to 22 where we're telling you we think they're a pretty good set of 22 if you could only have 22 would be the way to go but another approach would be to take some subset of that data and go i really want to the comprehensiveness of comparing thousands of things i don't want to miss out on something that's not in that 22. you could take a reduced set to kind of train the types of features that are useful so you might go okay i've got petabytes of data but i'm just going to look at you know a thousand time series in the two classes that i care about classifying i'm gonna run hctsa on those you know 2000 time series that might take some hours and then i'm going to pull out the top features that best classify those data set and then maybe you only find two or three or four features that are really helping you do what you want to do with your data and then you can take those three or four time series features and you know recode them in c so that they run really fast or only compute those three or four and scale that across the rest of your data set so there is a cost to training and working out what features are going to work well on your data but once you've done it and you know that there's only two or three or four then you can very easily scale that across the rest of your data set and then you're having a huge dimensionality reduction right because you're turning a time series that might be a thousand long or ten thousand samples long and you're turning it into just two or three numbers so you can then represent a large data set in a far more compressed and useful and interpretable way with just this handful of features that are tailored to the structure in your data that you want to quantify hctsa is up on github tell me a little bit about what it takes to get started using it the big barrier for hctsa is just having a matlab license it was designed for use in universities and for research projects where that tends to be more common but yeah there's lots of documentation there if you have a matlab license you can just clone this repository or press download if you're not a git lover then there's uh as i mentioned hundreds of pages of documentation they're linked through there there's a wiki that can help guide you through some youtube videos where i can take you through how to use it there's lots of documentation there are some tutorials you can download some data and start using it and it'll guide you through both the process of extracting thousands of features and all the tools that we've developed to visualize and pull out useful structure from your data set and i see there's many more contributors starting to show up on the repo besides just your contributions can you talk a little bit about the management of the project and what kind of contributions people are making to it yeah these have tended to be some of my students that have corrected a typo or added another feature here and there if you have a useful thing to add then we're very open to feedback or pull requests and i'm curious about your vision for where this goes certainly it's reached some milestones as it is do you have a vision for where you'd like things to be down the road we really want to develop new and useful ways to reduce this number of features down to smaller numbers in more efficient ways i mean the goal is to really come up with this canonical set of time series features that really draws from the best in the time series analysis literature distills it down to the most useful pieces and then makes that freely available to the scientific and non-scientific people out in industry solving real problems we want them to have as fast access to new and powerful methods as they become available as possible and for that to happen when a new method gets created we want to be able to assess its performance across lots of different types of time series analysis tasks and we want to pull out the methods that perform best and in ways that don't already exist in the literature and make them available through open source packages in many coding languages so catch 22 was the first example of that where we said well of these 7 000 if you could only have a smaller number what would be the best and we assessed the behavior of all these features on a large number of classification problems and distilled that down to this 22 that you can now run in all these different open coding languages and so we've developed also some web tools where if you come up with a new time series analysis method at the moment we're supporting think r and python you can drag it onto a new website that we've developed that can say well how similar is that method to all the other methods that exist in science so really we want this to be a tool that kind of guides progress and brings together all these disciplines towards tackling the real problems that need to be solved rather than the tendency of the literature to just develop a new method post it with minimal comparison and often that method isn't really contributing much to the literature so we're all about developing tools that can help both understand which methods are useful and unique and making those best methods that are the most useful and most unique available to as many people as possible i've been observing some trends in the scientific literature about papers that should have code or should have a data set open sourced alongside them do you have a vision for a world in which maybe certain journals associate that new methods that are published should as part of that publication process be submitted to the library yeah exactly i mean this is a very good question i mean this is this was part of the vision that i guess we're still trying to achieve and that was part of what developing this website was motivated towards you know it's hard for an editor of a journal to say you should compare your method to you know 7000 different things that's a pretty big burden to put on every author to your journal but if we can actually say no you just need to drag your method onto this website and it will just pull up you know make sure the results of that drag and drop you know is attached to your journal article or if enough reviewers of scientific papers in the peer review process go okay but you haven't really compared it to anything else is it really new and once this kind of idea takes off in the literature that really you need to have compared your method to alternatives in as comprehensive a way as possible and that most comprehensive way as possible has now become much broader than it would have been a decade ago because it's gone from comparing to the three or four most popular things in your literature that a reviewer could kind of fairly ask someone to do and now all you have to do is drag and drop your python code onto our website and it'll say you get a uniqueness score of 20 you're out performing these other methods on this problem and these are the most similar other scientific time series methods and that gives you a lot of information that you don't actually have to work that hard for so yeah we really do hope that this takes off and people start embracing this comparative approach maybe you'll get less papers because sometimes your new idea is redundant but when you do get a good idea it would be celebrated because you can really argue just how unique and useful your method is because you really have compared it to a really broad representation of the existing literature ben remind listeners where they can learn a little bit more online oh yeah so you can follow me on twitter at ben d fulcher b-e-n-d-f-u-l-c-h-e-r and you can find more about me from there well thanks again for taking the time to come on the show and share your work thanks for having me that concludes another installment of data skeptic time series our guest today ben fulcher thanks to our sponsors master class and simply safe myself claudia armbruster as associate producer vanessa bly guest coordinator and our host kyle pollach [Music] you