Toward Generalizable Representations and Scalable Citizen Science for Brain-Computer Interfaces

Key Takeaways

Good morning everyone. Uh thank you for joining this talk titled uh neural representation learning in the wild towards generalizable representations in scalable citizen science for brain computer interfaces. And here we have Chris Simmonia who is the chief innovation officer and co-founder of interaction. Actually this company is way more known as manufacturer of muse quite popular BCI device which many of us use in our practice uh trying to create a novel BCI interfaces. I'm not going to read the entire uh speaker bio. it is in the meeting announcement. And without further ado, Chris, you have the floor. Thank you so much, Ivan. And and thank you very much for the invitation. Uh it's exciting to have everyone uh here so can listen to some of the cool things you've been up to. Um great, thanks Maurice. We have a couple others uh on the call from Interaxon who will be presenting with me. Um and we will meet them shortly. So, uh, as Ivan said, people know us as Muse. Um, and this is because it's the the namesake of our, uh, neuroch devices. Um, and I think, uh, in many ways the the this kind of kind of tension of how we're known is also, you know, observed in just the fact that we really are a very deep data and AI company. uh and people they they get to know what they see and that is the the Muse product which is really the facilitator in uh the neurochnology uh space. Uh so the Muse device and and I'm sure some of you have seen these before uh is very simple uh consumer oriented neurochnology device that has like a very good quality EEG and bio signals that can be used for many applications. Um for us we really started in the space with the device that you see on the far left uh which is a headband that goes across the forehead and behind the ears and it was really targeted at helping people uh with their focus uh through focused attention training exercises. Uh but we've made many other form factors of the device and they would all share the same sensing montage and that really keeps a consistency across our data and helps us now as we look to big data methods for machine learning. Uh but we've also done some other things as well. We've made some very cool uh glasses, EG glasses. Uh and also we've done some cool work in VR as well. Um and really what we are doing is is to try to build a foundation for many different uh BCI applications uh through our experience in the consumer space, you know, where we're able to uh work directly with uh end users who are trying to improve their their health. uh and we also end up with a lot of uh data to help us facilitate the training of better models for for BCI applications. Um the the device that is the most commonly used is is this device. Uh it's called the Muse 2. Uh Muse 1 looked very similar. Muse 2 had the addition of a PPG device so we could co-apture um heart data. So blood flow data uh from the blood flow in the forehead. So there's a PPG in the part that goes across the forehead. And it's a 4 channel uh EEG. You wear it like this. And uh people have adopted it because it helps them train focused attention. So really helping someone develop their skill of attention to stabilize it to concentrate it because it has enormous uh health benefits. And this really is uh you know has a relationship with the way that we tend to use our bra brains these days in a very distracted way where we tend to jump around a lot from task to task and it erodess our fundamental abilities of directing our attention especially to the present moment. And so uh the techniques that we have leveraged really have their basis in very old old traditions of meditation but we have uh worked with the same techniques but with a real time twist. We give people real-time feedback on the stability of their attention so that if you're able to achieve that quiet mind that's not chaotically jumping jumping around from task to task that you get an auditory feedback to know that you're there. And so the basic way that we would do this is a weather metaphor. You'll hear the sound of rain. If your mind is scattered, you'll hear stormy weather. And if you're able to focus and settle the mind, you're going to calm the rains and hear birds. uh and so there's a reward there to get people uh to learn uh in a classic neuro feedback way when their brain is in the right state. So it's a very interesting application that combines neuro feedback techniques as well as meditation techniques to help people train the skill of focus. And so you do your meditation and then at the end uh we give measurements um and these measurements also help to track your brain health and performance over time. Uh, and there something that's becoming more and more central to us as we're more strongly going into the brain health space. Um, you might have if you've come across this before in years past, our brand was very meditation focused and now we're much more going into generalized brain health. And the the technique is is helpful. We've done a bunch of research. There's been a lot of third party research done to look at the effect of these techniques. uh and no surprise it really uh follows the same uh arc of benefit that you get from meditation training techniques of increased resilience to stress, higher attentional performance, reduce perceived stress, these kinds of things. Uh but the learning is more engaging and it's more accelerated when you have something that can kind of hold your hand. Um something you might experience if you were to, let's say, meditate and you'd have a group that you'd go to and have a teacher. Uh but in a way that you can actually use technology to to fill that role for you. So um the the next really popular device that we created is this one. Uh we call it Muse S. Um S can be for sleep. Uh since we were really focusing on how we take our technology into the sleep space and this required generating a new form factor that was super comfortable. And in many ways, we wanted to do this right from the beginning, but it was really the uh the availability of the right material technologies that could create very very good, flexible, stretchable sensors uh that could be worn comfortably night after night. And so we were very happy when all of these material innovations kind of uh came together so we could actually take this this new direction. Uh you know, our our interest in helping to support health. Obviously, sleep is a pillar of health. uh you know you can't you can't really help yourself much if you're not sleeping well and we really believe that there was a lot we could do to facilitate better sleep the uh the the utility of it uh for for the end user I mean the first part sleep tracking that's quite obvious uh and as you might expect you know we can track the stages of someone's sleep their sleep position their heart rate these sorts of things one differentiator is when you have access to the brain you can actually do very very accurate uh sleep measurements And so something that we've been able to achieve is to have a sleep measurement accuracy that's the the same as you would get is if you would go into a clinic do an overnight PSG in a lab and then have an expert uh stage your data. And that's very helpful if you're actually trying to use this as a tool to help give you feedback and improve your sleep and how and your behavioral patterns around sleep because you can actually, you know, be assured that you're actually seeing the changes that are uh, you know, truly happening as opposed to might be happening as you might find if you were trying to use, let's say, uh, an acttography device or a ppgbased device to to to analyze your sleep. But the other thing uh and this is really kind of in our core in the muse DNA is is the sort of real-time applications. And so we really had a strong belief that we could create something to help people uh take themselves to sleep more effectively. Uh we knew especially with our work already uh with the Muse device uh for meditation that the that many of our users would would bracket their sleep either you know at night or in the morning. And those who were using it at night really they were doing it because it helped them go to sleep and it was really about calming the brain. And you know going deeper into the the challenges people have when they sleep. Insomnia is is really a a huge affliction for many people uh adults uh in the population. And then we all we have so many others that are subject to mild insomnia which is like having trouble falling asleep or waking up in the middle of the night and having trouble sleeping again. And the biggest culprit for that is having a busy brain. And the busy mind is really something which is stimulated by an emotional process. So often the feeling that we have generates thoughts and the thoughts generate more feelings and you go on and on. Uh so how do you break that cycle? And so we created this thing called the digital sleeping pill that was really based on our ability to track that transition from waking to sleeping. So we could intervene in creative ways to help people detach from that cognitive process that's keeping them awake. And the simple story of how that works, Maurice will talk more about this uh later in the presentation, but the simple idea is if you can get someone to step out of their present moment experience and into something else which is less provoking. Uh so maybe not um feeding back an emotional process and thoughts that intensify an experience but rather allow you to ride along in another experience. A great example is listening to a story. That can actually give someone an opportunity for sleep because once you detach from that energizing, discursive thought process, this is when you feel tired and you start to doze. And dozing is something that, you know, we go from consciousness into semic-consciousness, but we're not really falling asleep. And we found that if we could detect those subtle moments that we could actually go in there and change the audio experience, you know, and you can imagine if you were a parent reading to a child, if you're, you know, reading a bedtime story, what you'll do is you'll actually watch to see, you know, is there a potential you might fall asleep right now? And if there's a possibility, I'm going to stop reading for a moment or I'll slow down or I'll change the tone of my voice. And these cues create space where the mind without something to hang on to sort of falls asleep by accident. And having an AI that can predict this to happen uh and to have a more nuanced measure of these kind of micro states between waking and sleeping uh really facilitates the creation of a technology that does the same thing very effectively. So, um, that's kind of our our last device and its innovation with the digital sleeping pill. We just launched a new device. Um, and it looks almost the same, but it has some really cool differences. Uh, we've incorporated um FER's array, so we can measure EEG, which is measuring the electrical activity of the brain, but we also measure the um blood flow and oxygen saturation in the front of the brain. And this is called functional near infrared spectroscopy. uh we've expanded some other things you know it measures SPO2 now which is very important for sleep apnea uh and uh we can also do ERP measurements so the um the the array that measures the fneers uh it if you look at the image on the right here we have um EEG electrodes these are the silver parts on the front of the headband and then the you'd have these five windows in the middle is the emitter optode and you have three different wavelengths of white light emitted From there you have red and two infrared um wavelengths. And then you have on both sides you have four different detectors uh that measure both the short channel and the long channel light that's flowing through the skin and then through the deeper tissues including the brain. And then looking at those is what really what uh allows you to kind of decode that blood flow um through the brain uh measurements. So I'll just give you an idea of what this looks like. Um so this is a simple example. So um brain activity. So if you're engaging in a cognitive task, that's going to affect uh your frontal blood flow. But also different breathing techniques also affect it. Um which is also no surprise given how prevalent different uh breath work uh techniques are now uh you know coming into vogue. Uh but on the left you have just very simple deep breathing. On the right you have a sequence of breath holds. And what you see when you measure the uh blood profusion uh in the front of the brain is you see you know these disturbances that are caused by these respiratory uh effects. So we're seeing here the red lines are oxygenated hemoglobin and the blue lines are deoxxygenated hemoglobin uh and you're measuring uh blood uh concentrations here. The other thing I I mentioned uh and I think those of you are in the EG space you'll know what ERPs are but basically uh when you uh present a stimulus uh to your senses and you have a task to do um if you actually measure uh what happens in the EEG uh time synchronized with those um tasks you actually see repeatable patterns of uh you know polarization and depolarization uh in the electrical brain activity and you can measure those and they're very useful measurements. that you can make from that that have been done for years and years. One of the challenges is you need to have very good time synchronization between the measurements and the stimulus you know. So to make this work we need to you know create protocols to do clock synchronization between the Muse device and the mobile device. Um so so getting the the clock synchronized between the the uh the OS clock uh you know and the Muse clock. And then you need to also be able to predict, you know, when you say, "Okay, draw the green circle on the screen." Uh, knowing exactly when that'll be presented. Uh, and there's also this rendering pipeline. So there's there's a bunch of unknowns you have to solve for. But once you do that, you can have very good time synchronization between stimulus and the measurements and that allows you to do these uh very cool uh way of looking at brain activity where you can look at this sort of millisecond response. it's actually more hundreds of milliseconds respon response to the brain and make the measurements that have become so common in in the EEG science uh and how that relates to things like attention uh and other uh processes in the brain. So oopsie click next. So this kind of just is like a highle map of the evolution of our technology from a device perspective. And you know we've tried over the years to expand on the sensing capabilities but the whole time we're keeping the montage EEG montage or the locations of the sensors across the devices the same. And this allows us to have harmony across our whole data set which is quite important as we're looking at building models. you bu you can build single channel models, you can build channel agnostic models, but also if you're able to use the spatial dimension of the data, you know, it's it's more powerful. So, we've really tried hard to keep the data consistent across the different generations of our of our tech. So, okay. So, I'm just going to take a step back. I mean the I just this was the sort of hardware perspective of Muse and one thing I just wanted to kind of um you know stage is that you know overall we have a much bigger mission around health. Uh we have a deep belief in the power of the data with these new AI technologies to really help us uh in many dimensions of our brain health. Uh so our consumer apps they already touch upon these pillars like sleep and recovery uh mental health and cognitive function which are really supported by these techniques of meditation but there's a lot more that's possible and the challenge has been the creation of uh generalizable algorithms and more sensitive algorithms so we can actually do uh measurements that are going to help to unlock some of these categories you know whether it be for a therapeutic intervention or for a diagnostic. And so the you know from you know inside of Muse the the company uh you know specifically working with the data developing AI python uh pipelines and and working you know with the latest in the machine learning methods is really where we have our focus and we will talk about that uh and so there's there's a lot actually that go that's in our platform besides the kind of the device that you see you know and this is like a huge database we have over a billion minutes of EEG which is very exciting. exciting of th for those who are into self-supervised learning techniques uh and big data we have a great AI infrastructure you know we do biomarker development we have pipelines to do real-time diagnostics and measurements in the app we also have cloud pipelines for that and that allows us to have this whole kind of like clinical and research arm to our business which is super interesting and we also have an SDK that helps people then build upon the tools and models that we build and so a big part of this uh and we'll talk more about This is the creation of a citizen science platform. And the the genesis of this really came from the understanding that if you're going to build therapeutic interventions that helps somebody with their brain health, how do you know you're building something that works and how do you optimize it? And it's not something you can do with like 20 people in a lab, which is traditionally how neuroscience is practiced, you know, very small sample sizes. And you know, when you consider the complexity of the brain, the complexity of our behavior, that's nuts. And the only way to really do this properly is to really be able to take a scaled approach at it where we can really involve many people to try different things and to see what happens. And so we're very lucky. We have an extremely engaged user base that are very interested in pushing the science and also pushing brain health. And so we put advertisements out saying, "Hey, do you want to participate in the study?" and we get thousands of people sign up where we can actually do brain science together. And so we built uh an infrastructure in our app which I could talk a lot about. Um and it allows us to deploy experiments just like you deploy content. So it doesn't involve app changes. is you literally just download protocols from the cloud and you can have you know learning questionnaires, cognitive tests and real time interventions where you can have like this whole uh real-time adaptive system you know like what I was describing with the digital sleeping pill uh you know and and be able to try these things and and change the parameters uh of these interventions across different user groups and to evolve this in this more human in the in the loop kind of machine learning uh approach. So, uh, this is something that we're quite proud of and we're actually leveraging a lot now. We work a lot with academics, uh, in in research groups as well as other companies that, you know, are trying to get in there and learn more about what's happening at scale. And it could be everything from like a digital intervention to a psychedelic therapy to a drug therapy or a CBTI approach. So, there's a lot that you can do once you can kind of have this sort of at-ome deployable, easy to use um, you know, uh, decentralized study uh, approach. Okay. So, what are we doing with that? Uh, I will hand it over to Arvin to kick us off uh as we get into what we're doing on the AI side. Thank you, Chris. Um, so as Chris mentioned in this uh next section, we will explore how we are moving towards generalizable neural representation learning with EEG. uh and specifically we look at the need for models that can adapt across individuals tasks and uh real world conditions. So historically, EEG analysis relies on the manual feature extraction that requires domain expertise and typical features extractor are like band powers and other features that are then fed into classical machine learning models such as a linear discriminant analysis LDA or a support vector machine SVM. But what we've observed is this fails to generalize across individuals and sessions and it's also not very robust to noise or hardware variability. And on the right side we see the needs uh for a consumer scale neuro tech. And we show some of the challenges where there are different settings, there are different uh applications, there is sleep, there is meditation, there's daytime use, there's also different types of hardware. Um and so there's lot of inter subject variability and interdevice variability that we need to take into account. So EG signals they vary in all these conditions and so there is a need uh for methods that can extract invariant and generalizable representation regardless of when or where the data is recorded. So at news we've collected an enormous amount of EG data um through the participation of our community both during meditation and other wakeful states as well as during sleep and this large scale brain data it allows us to go in these two different pipelines or complimentary systems. Uh one we call it the normative EEG database. um where in this database it helps us to benchmark an individual user by comparing their characteristics and typical patterns with rest of the population um across different age groups, gender, sleep profiles, etc. And then on the other side, we also have foundation brain models which are built using self-supervised learning on large volumes of unlabeled uh EEG segments. So, so these two pipelines help us to learn some of the representations that can generalize across different uh states and and tasks. On the bottom here, u we have a visualization of the brain embeddings in a two-dimensional space. And this was a model um that's projecting the data from our sleep uh EEG. And we can see very nicely these different stages of sleep the wake N1 N2 N3 REM all of them being nicely clustered on this two-dimensional representation. So currently foundation models are needed for brain health and why now? Um there are few converging factors. So firstly it is the data. We now have unprecedented amount of data access to large scale EG recordings from real world users and something that was not available even just a few years. There have been advances in GPUs, parallel processing that make it feasible for training such large scale models and also methods such as self-supervised learning that has really matured and now we are able to adapt these techniques that were typically used in natural language processing or computer vision applications bringing them into the domain of bio signals and EEG and and lastly the clinical and consumer demand. There is a growing demand for scalable personalized brain health tools um not just in the labs but in the homes in clinics and wellness platforms. So putting all this together we we call this new class of generalizable models as foundation brain models. And what makes all of this possible is self-supervised learning. And as most of us know, self-supervised learning um leverages the inherent structure in the data to create toy problems u which are also known as pretext tasks and that will let the model to learn these generalizable representations um by solving these pretext tasks and a classic example in the LLM case is to predict the next word in a sentence. Similarly, we apply those principles in EG and by solving these tasks for example um EG is processed in segments or windows of data. So if two segments are from the same user or if they are correctly ordered in time, this information can be used to train a pre-text task and the model learns this rich uh representation. And once we have this representation, we can then use them in uh downstream uh applications such as like sleepstaging, apnea detection, event classification, state classification and so on. And soon we will see an example of a study that uh we have conducted uh internally. Uh Arvin can I ask a question now or you prefer this to happen after the talk? Um either ways is okay. Uh on the previous slide yes how how big how big is the large brain data corpus and how you obtain it it? Yeah this is a very good question. Um we have a slide actually showing you the size of the data set. But what we have done is we used a subset of about 80,000 sessions approximately from 18,000 users um as as a preliminary analysis. Yeah. Okay. Sufficient for unsupervised training. Thank you. Yeah. Thank you. So typically as pre-text tasks um these two methods are typically uh done in the literature. Um we have one called the relative positioning another is the temporal shuffling. The relative positioning is quite simple. Uh we have several windows of EG. We have an anchor window and then define a time. If a particular window falls within your positive time window, we call that as a positive example. So if windows are close together it's a positive example. If windows are further apart it's a negative example. And this way when we train the model the model learns to capture temporal continuity in the EEG. In the case of temporal shuffling instead of two windows now we go to three windows and we order them in a certain way more like a solving a jigsaw puzzle. um where there are three windows in sequence and then the order is shuffled. And so positive example is windows in sequence and negative example is if the order is shuffled and so now the model learns to find the temporal transition and structure um between these two examples. Another interesting approach that has been uh used is the contrastive learning. And contrastive learning is trying to bring similar samples close together and dissimilar samples further apart. And so with this we have explored two different approaches. There is one at the segment level and then there's one at the participant level. At the segment level we take two segments. You take one segment of EG, corrupt this with noise and then we give it to the model and we want to say okay these two are similar segments and then you take a completely unrelated segment and say this is a dissimilar segment and so this way the goal is to learn features that are robust to noise. But on the other hand there's also inter subject variability with bios signals. So the participant level pretext task um is actually quite beneficial in this case where two windows coming from the same user are positive pairs and a window compared to another user is a negative pair. Um so with this uh we have run a study and uh this has been currently accepted for publication at the IWE EMBC conference and based on that pre-text task um we use two different backbone architectures for the foundation models. One of them is the shallow net the other is the EG conformer. The shallowet is what is typically used in BCI literature. it's well understood to emulate the filter bank analysis. So there's temporal convolution followed by spatial filtering and then in the case of EEG conformer it's actually the same shallow net with an additional uh transformer encoder. So now once we train this we can use the representations and embeddings um to train linear classifiers to probe the performance and in this case we used couple of example tasks. Um this is uh primarily the downstream task for predicting the age and sex uh from a given window of uh EEG. So what we see here um on the top left is we are seeing that with a larger pre-training data set the performance improves but then on the right side what we wanted to mainly compare was the difference between the methods. So we proposed this new method of participant level pre-training and we see that with both the shallownet and the EG conformer the participant level pre-training improves the performance compared uh to just using the segment level pre-training and most importantly the linear models um that we are using although they are simple we're able to see that on the bottom two graphs that especially in the low label percentage regime that the pre-train model does much better than a model trained from scratch. So they show us encouraging trends especially in the lowle settings. But it is also worth noting that like the downstream task of age and sex classification like this this is not the end goal but definitely they offer a lens into like what is the strength of learning this kind of representation and we're really only beginning to scratch the surface and this kind of answers part of your question uh Icon we only trained this on 80,000 sessions but with more and more data um we can actually improve the representations, the generalizability and also um across different tasks. Then next what we'll do is we'll look at how we are leveraging not only the raw data but also our normative uh database for uh example brainage estimation longitudinal tracking and also some emerging directions such as emotion state classification which is uh a future work something that we are excited about. So one promising example is uh to estimate someone's brain age and this is a biomarker of brain health and what we are using in this study is to extract features from sleep eg um such as biomarkers such as sleep spindles, slow waves and spectral features that have been known to show this trend with age and they shift with age. So there is a potential in extending this towards uh AI based early prediction of cognitive decline. And another exciting frontier is we have um as news we have the simultaneous bio signal along with natural language data. So bio signal with EG ppg now fneers and we have uh users going through the guided meditation and sleep stories. So this offers a good connection between language, physiology and self-reported emotion. So in the future we could use the embeddings from the foundation model uh to model how these emotional states evolve and they would open a door towards like adaptive emotion aware experiences and this remains a promising direction for future work. And another uh very good example is how we leverage the normative database to have a longitudinal view of data. And what you see on the left side here is um we have run our automated sleepstaging algorithm on a large population. And we see the percentage of sleep stage of different sleep stage across age. And as expected with age with aging um we see an increase in stage two or light sleep and a decrease in the deep sleep or N3 sleep. So we can now perform these kind of comparisons. And then on the right we have two more examples um which is looking at micro events in the sleep such as the slow wave density and spindle density. And here for example, we're looking at the distribution with two different age groups and then a particular users's age group and being able to compare them with respect to the population. And since we are now on the topic of sleep, um let's explore how we apply these large scale data sets to build automated sleep staging models and investigate their roles in diagnostics. And so now I will hand it over to Maurice who will walk us through the next section. Thank you Arvin. Yeah, that detailed sleep tracking that you showed is actually very important. Um as Chris mentioned, um we're very interested in brain health and sleep is not something that just beneficial to you. It's actually a fundamental pillar of our brain and mental health. But actually many people struggle significantly with sleep. In fact, for example, clinical insomnia impacts an estimated 10% of adults. I actually was shocked when I learned that. And if you suspect of having a sleep issue, um, what you typically have to do is you have to go to a sleep lab and get what you would call a polyomnography or a PSG. So basically what happens is that you get hooked up with a bunch of different sensors that measure your brain and also your um breathing activity. Um you have to sleep there overnight and then there's a sleep expert that's going to come and painstakingly look and sleep score your data. And probably as you can imagine this is time consuming. It's expensive and labor intensive and really like sleeping in this way is not really representative of your true sleep. So what we aim to do um with the muses that Chris u presented is changing the way this is done by simplifying and bring this technology at home. So the muses can actually collect brain and heart data at the comfort of your home and can send that data to a mobile app that analyzes it in real time. And in order to do that um we would need to build these machine learning models that run in the app oops that run in the app and that um automate the tasks that a human sleep expert can do. And when you're building such a model, it's not just about performance. There's a lot of several considerations that you need to take into account. Um because you're developing something that will be used at home by consumers with minimal supervision unlike if you're at a sleep clinic where you be actually supervised. So you need to um your machine learning models need to be robust to transient nodes and bad channels that are caused by motion artifacts or a bad headband fits or sweating etc. Um the model needs to be able to be running in real time to enable these these like these adaptive neuromodulation application that Chris was talking about. And the model should be able to be packaged in a form that is able to be um run in the app. And um that means it doesn't have to be confusionally too intensive and at the same time also memory friendly. And finally since we want this to be used by thousands and thousands of people the model has to be um calibration free and generalizable. So with all these constraints in mind, we designed a deep neural network that has been trained to effectively get detailed measures of sleep states. So our models consist of hybrid convolutional neural networks or CNN's and recurrent neural networks. in this case a unilateral long short-term um sorry unidirectional uh long short-term memory network or LSDN that enables these real-time application I'm talking about. So the CNN's they look at the raw EG data and they're going to extract features uh from this data. These features are then passed on to the LSTM which model then the temporal dynamics of these sleep states and also narrows down your current sleep state to um based on the feature that I extracted from the CNN's in the CNN's we have different kind of filters convolutional filters we have small and large time windows um and that's designed to be able to see different kind of EG features um also the algorithm has we have multi- channelannel and also single channel models and we have an ensemble aggregator that takes to account the signal quality coming from the channels and also the outputs coming from these models and ways the contribution of these models to the final output and because of that we are able to get a measure of sleep states even if only one of the channels is working well and the others are not. We have also something called a dynamic spatial filtering layer which is essentially a multi head attention module that is plugged um before the first layer of the multi- channelannel model. And this layer here it really learns to focus on the good channels and ignore the bad ones to reduce the effect of noise on the subsequent layer of the models and to make our model even more robust to noise because noise is really an issue especially when when people are using it at home. So we injected during training these models we injected noisy data into our model. So we injected white noise but also we injected noise that we took from our real world data and injected that randomly in a random fashion during our model training. And since our models are have several layers, the deep embeddings um that are in these layers can serve as representation that can generate more nuanced and continuous insight of someone's sleep state. So and also to be able to look into how what the model is doing, we have methods like gratam for example where we can kind of have an idea of what the model is looking at to measure sleep states. For example, when you're in light sleep or stage and two sleep, we are seeing that the model is looking at these signatures of um of light sleep which is called K complex. Great. So we package this u algorithm into the mobile. It's running inside our muse app and also in the cloud. And now one now someone can go to sleep night after night and get a detailed analysis about their sleep at home. But how do you validate that this kind of sleep analysis um is working well? How do we know how performant our model is? in particular, how does the model, how does the output of our sleep analysis compare to the outputs of a sleep expert um in a clinical PhD where the sleep expert is assigning a sleep score wake and one and two and three or doing something that we call a hypnoggram. Well, our model is also um able to work the same way and on the same sleep segments that the sleep experts are looking at, producing its own hypnogoggram for your entire night. So now what you do actually is you have to go to a sleep clinic, you wear both the Muse EG and also um the PG system at the same time. Then you would have an expert sleep technologist that will score the PhD and give you your golden uh labels, your standard sleep labels. And the sleep analysis model would output the automated sleep stages which could then be compared to um the gold standard. And this is exactly what an independent group at the University of Ottawa carried out with a cohort of um 47 participants and included good sleepers as well as people with sleep apnea. And what they found is that the overall coins scapa of the automated sleep algorithm against the PhD is 0.76. So what's coins scapa? Konoscopa is just a measure of accuracy that is adjusted for chance. And what these results mean is that the Muse achieves expert level sleep staging. Because if you get two human sleep experts and you have them analyze the same recording, they will not agree 100%. They will agree on average with a coins sca of around 0.75 or roughly 82% of the sleep sleep stages. What the study also showed is that the core recordings um validated that the sleep landmarks that we typically see in a PhD like sleep spindles, slow waves, eye movements can also be seen on the Muse um on the Muse EG. Right? So great, we have this this great sleep model that can be used to sleep score Muse EG as accuracy experts. Two question before before you go. So two questions first uh when you train the classifier what are the labels? So the labels in this particular case are the sleep stages. So um how yes so great question. So since the landmarks of um the sleep um stages for example you see the sleep the spindles here the slow waves the eye movements alpha and wake they're visible on both the PSG and the muse. So we actually so when we developed the model we had the sleep experts um handcore the the muse EG so they can look at this okay this is N2 this is N3 and we use the mod the the the labels from the sleep experts staging the muse to train our model. Okay but there are not that many. you lose the advantage of having a large data set when you count only on the human label uh uh segments. Anyway, my second question is uh is this model a person dependent or person independence? Well, this model is totally person independent. It would be it would work on any any person. Perfect. Thank you. Thank you. And that's very good question. I mean also that's an important thing to consider during testing. So when we developed our model internally our training uh the subject we use for training are not included in the testing. So when we test our model internally um we made sure that the people the recordings in the testing sets are from people that were not seen in the training set. And also when these people did this independent study, the people that they had were totally um unseen by our training. We didn't know about it and there were there's no calibration, not nothing to do. It just works um right away. Very nice. Um thank you. Thank you for the questions. Um so we have again we have this this algorithm that can do the sleep staging but we want to to go beyond that. um how do we go actually beyond that and use the representations that this model learned to support really support adaptive modulation application but also unlock deeper understandings of our sleep. So to do that you can look at the inner representations of the sleep model and what we actually found is that the model is learning nuance and continuous measures of sleep. For example, if we look at the embeddings of the new sleep EEG using a technique called UMAP, we notice that our model internally reveals kind of a sleep continuum with a large N2 cluster which flows into N3, your deep sleep, flows into REM, also known as this period in in and sleep where you're actually dreaming, and also flows through N1 through this stage here in light gray that you see um that is called classically N1 and more informally the hypnogogic state. Now that region here, this transition from wake to sleep through N1 is actually pretty important for the process of you falling asleep. Um and in fact in classical sleepstaging this stage here, so here you have um you're awake, you have your alpha, then your alpha start disappearing, and then you have your spindles that appear. This transition here between the wake and your N2 sleep is called N1. And even within that state, people are trying to divide that period into micro states based on different EG patterns. And while this is still a poorly understood area like one proposition was to divide that here into nine different states called the holy stages. And when we look at the output of our sleep model versus the hory stages, we see that our outputs are effectively tracking these hor stages. Therefore, kind of confirming that there is a sound and scientific correspondence between what people have been noticing before with these hy stages, but also um what our model is learning and the deep representation that our model is learning. This also means that we can use our model to track a person's transition from wake to sleep and offer these neuromodulation based on our sleep models output which actually led to the genesis of our digital sleeping pills that Chris talked about. So these digital sleeping pills or DSPs effectively are using our sleep models outputs and auditory neurom modulation to guide a person to sleep and we deploy that in our app. People are using it. Um and we ask ourselves how well does it really work and how well can we improve it? And this really spurred us to build that platform that also Chris talked about in the beginning where we can leverage our community to test our algorithms and get feedback. So with with this platform we are able to do decentralized studies, upload different protocols, algorithms, content to the cloud, deploy them in the app and really collect data from our users at scale. Um and we use that platform to test our little sleeping ps. It was actually the first study that we did using that platform and we advertised um for a couple of weeks and we enrolled uh 160 participants and we distributed them in different groups and they collectively went through a week of sleep recordings and questionnaires where they reported details about their experience. And really in in a few weeks um from advertising to data collection to analysis we uh were able to do this study and we found that people loved it. So kind of compared to a control group um the groups who used uh DSP they subjectively and objectively had um reported easier times of falling asleep. And not only that, but we also got reports about their conscious experience. And that give us more insights and more confidence in what we have building, what we have built before and we are building now. And I see that Chris removed his camera. I love I love talking about this stuff. Okay. Um yeah, I'll I'll jump in here um just because I love it. Um so uh so this is sort of like the the the first fora that you know we made into sleep improvement and I really love this kind of technology generally for uh sleep applications because we are asleep and there's very little action we can take on our own uh to try to intervene and improve our experience. Uh but there's a a a way for technology to get in there and to really assist in many different ways. Uh that's that's very cool. And so um you know this is a like what we have a picture here of on on the slide is some of the ways that we are exploring. And we're not the only ones doing this. And we have like like an amazing array of researchers that we work with that are working on some different strategies. U but there's very cool things you can do. Everything from uh entrained stimulation during deep sleep to improve slow waves uh to having ways of waking someone up so you feel more refreshed. So getting woken up at exactly the right time or staging someone out of sleep. Um sort of the opposite of what we do because we try to put people back uh generally uh with our uh digital sleeping pill. Uh and then there's all kinds of cool things you can do with dreaming as well. Uh so an example of that is um you know what's sort of generally termed targeted reactivation um where oopsie where you can actually um have somebody prime themselves during their waking hours. And this can be a visualization, it can be even a video uh or a narrative. And then you would combine an audio stimulus like a certain kind of bell sound uh with that experience. And then by presenting that at different points in your sleep, you can achieve different effects. So for example, presenting this stimulus during the right times in deep sleep can actually improve memory consolidation. And presenting this in a dream state can actually bring this kind of content into your dreams. You can think of it as like a dream engineering. Um, and this is particularly helpful if someone is experiencing nightmares. If you're suffering from depression, as an example, you'll often have nightmares that are recurring. And this is a way to help to reggu the brain or potentially you're just interested in trying to use your dream time in some beneficial way. So, I feel like there are many possibilities there and we're just kind of getting started, but we we definitely know uh through the research that's been done that there are many powerful things you can do, including helping someone to reach a lucid state where you actually become conscious that you're dreaming within your dream. Uh and and that in itself can be very very interesting for someone to peer into that subconscious process. So, lots of fun to be had here. But generally speaking, we have like a intelligent model. In this case, we're using a model that was trained for sleepstaging. But we can use that sort of underlying representation to kind of get a finer tuned um uh idea of what's happening uh in the brain processes. And we can link that to another kind of adaptive intelligent model that can have an end application uh you know be it sleep induction, dreaming, you know, or diagnosis. And so another example of this which is quite cool and this is something that we've been started working on uh recently uh and it's um wasn't created by us uh but it's a really interesting idea where you can train another model where the input is not EG but in this case it is sleep state so those sleep stages and looking at that um you know sequence that happens throughout kind of like a word and uh in this case the GPT2 model was used and it trained to predict the the next uh sleep uh stage and the model that was learned during this um showed some very very useful characteristics. I mean one of them is you were able to make corrections to a hypnogog. Uh so if you had noise in your data or or other problems you could uh improve uh its performance in the case where you're not able to have uh appropriate uh classification because of some issue and be able to do quite a good job at that. Uh and then you can also use this for other things like diagnosing uh some kind of sleep condition. Uh and uh and the beautiful thing is the interface between these two models is super simple. In this case, you just take the sleep stage output uh of a sleepstaging model and you can pass it into a subsequent model uh which can now make an assessment based upon you know that learning the the dynamics uh of sleep stages uh that you that you can find in many open data sets. And this this particular group used a whole bunch of open data where sleep um architecture so the sequence of sleep stages was associated with different uh medical uh sleep conditions and that provided the foundation to train a model like this and so the really the availability of data you know enabled uh this kind of model creation u but I think generally speaking this is you know the thing that we're kind of most excited about right now is starting to connect different AI models together like you could think of it as like a model stacking or model augmentation and you know specifically you know something which is very obvious and very easy to do uh is to start to connect these kinds of uh brain model uh measurements to language models. So in our application, we have users that, you know, are um using our our application. They make measurements and they want to know what it means and and they want to know what it means in a way which is related to, you know, what they're actually trying to do and what they're experiencing from day-to-day. And just by creating good combined prompts where you take information about someone's brain state and yo