Developing a Brain-Computer Interface Based on Visual Imagery

Key Takeaways

[Music] just recording pending okay uh without further ado justin you have the floor okay thank you for the introduction and thank you all for coming to my talk today so i'll be talking about my project on developing a brain computer interface based on visual imagery that i worked on this summer with my mentor yvonne and the rest of the bci and audio and acoustic research teams here at microsoft so first off starting a little bit of background into this research area and i'd like to ask the question first of what exactly is a brain computer interface in its simplest terms a bci is a technology to facilitate communication between the brain and an external device and importantly this communication does not depend on any inputs from peripheral nerves or muscles it's just a direct communication between the brain and the device so a basic framework of this experiment i might have a person engaged in some sort of task where they're you know maybe imagining interacting with some object on a computer screen first off you're acquiring their neural data that data is being processed and then transmitted over for classification and translating that code from like a neural stimulus to a control code to control something on the device so the first step in any bci is determining how exactly you're going to measure the neural signal and there's a few different methods each with their own pros and cons but one major consideration is the spatiotemporal resolutions of each of these devices so i'll go through a few of them starting off with eeg or electroencephalography so eeg is like electrodes placed along the surface of the scalp that can pick up the small electrical potentials that occurs when neurons fire so eg is perhaps the most common most well used device are for bci applications because of its you know relatively inexpensive starting cost it's pretty portable like you can have just a cap that you slide on easily or electrodes embedded into some other device like headphones something like that it's got a pretty high temporal resolution but it has the low spatial resolution so because you're trying to you know record the signals through the skull through the scalp it could be a little bit difficult to localize where exactly those brain signals are coming from so next up in the non-invasive side is fmri so this would be functional magnetic resonance imaging this is actually what i'm using in my phd research right now so fmri has a much higher spatial resolution than eeg you can get like small like almost millimeter size called voxels or brain area which might have you know a few thousand neurons in there so it's better spatial resolution but it has a very low temporal resolution so you're only able to capture an image like every second or two another issue with fmri is it has a pretty um pretty big delay because it's an indirect measure of brain activity so like with eeg you're measuring the like electrical potentials of the neurons directly but with the fmri you're actually measuring changes in oxygenated blood flow into an area of the brain so the idea here is that once a region of the brain becomes activated the body has to feed that area with additional oxygenated blood so fmri is actually measuring that increased blood flow in an area sort of as a proxy for brain activity so because of this you have to deal with an issue known as the hemodynamic delay which it takes about like four to six seconds for you to be able to determine that you know blood flow is increased in that area so it can be a little bit difficult timing like when an action corresponds to an activity in the brain so moving on to the more invasive techniques uh one common one is electrocorticography or ecog so this is pretty similar to eeg and that it uses electrodes except now they're placed along the surface of the brain so this has a much higher spatial resolution and still taking advantage of the high temporal resolution of eeg however of course it does require an invasive surgery which has its own inherent risk with like infection things like that and also these electrodes can't be left in for a very long period of time because you know over time as the body's reacting to this foreign object in the brain you might have like scar tissue build up things like that that can cause the signals to degrade over time even farther in the invasive side there's like local field potentials or spiked electrodes these would be electrodes placed into the cortex that could read like a single neuron or like clusters of neurons so it has a very high spatial and temporal resolution however once again it requires invasive surgery the signals can degrade over time it's got a high cost and a higher power consumption and latency issues so next i'd like to talk about some of the common bci paradigms used in vci applications so these paradigms can fall really into two different camps of evoked potentials and non-evoked potentials so an evoked potential would be you know providing a participant with some sort of external stimulus and you're measuring the brain's response to that stimulus where a non-evoked potential would be you know participants engaged in some sort of imagery so like an endogenous stimulus and you're just measuring their brain activity during that imagination so any evoked potential possibly the most well-studied and well-known paradigm is the visual p300 so this is a common event-related potential that appears as a reaction to a presented stimulus so one application of this that you might have seen before is the p300 speller so how these devices work is you would have a subject be able to spell out like a word or a phrase just by looking at one of the letters at a time on this this keyboard so the different rows and the different columns are going to be highlighted at different times and based on what the participant is looking at and the changing intensity levels of that that letter flickering on and off you can judge you know when the letter highlights and when the erp response happens in the brain which letter that they're looking at another kind of similar technique is the steady state visual evoked potential so how this would kind of look like is a person would be presented with this uh array of flashing targets or like flashing leds each of these targets will be flashing at a different frequency and based on what which target that they're looking at you can match the frequency components in their eeg to the flickering frequency of the target and identify which one they're attending to some less studied evoked potentials it could be like you know measuring a person's response to an auditory stimuli or to like a somatosensory or a tactile stimulation and then moving on to the more like non-evoked side of things um so this would be you know mostly imagery based paradigms and by far the most studied one is motor imagery paradigms so for example of a motor imagery paradigm like sensory motor rhythms this could be the imagined movements of large body parts so for example you might be imagining opening and closing your right versus your left hand there's also imagined body kinematics which is like the imagined movement of a single body part in a multi-dimensional space so this is actually a technique that i use quite a bit in some of my past research so an application of this might be like if you're trying to have a bci to control a computer cursor like on a screen you can have a participant imagine they're moving their right hand as if they're using a computer mouse to control the movement directions some less studied non-evoked potentials could be like visual imagery so this would be like a conceptual reconstruction of a perceptual experience so like directly imagining an object that you're trying to interact with there's like speech imagery which is like the imagery of speech production and there's different categories of mental effort so this could be things like you know imagining rotating an object in a three-dimensional space or mentally performing math calculations something like that so to give a little bit of background into this project that i was working on this summer my main goal was to create a bci using some kind of nano vote potential and we wanted to kind of steer clear of motor imagery so for that reason i selected visual imagery as the paradigm that i was going to use throughout my experiment and one reason for this is you know we wanted to do something a little bit different that hasn't been studied as much and also visual imagery provides like a more intuitive control for bci so say for example you want a vci to control different applications in your house like maybe turning a lamp on or off with visual imagery you can imagine just that lamp directly and interact with it whereas with something like motor imagery you'd have to remember a map between your right hand might interact with that lamp so next i'd like to just give a little bit of background into some research in the visual imagery bci field so one study by natalia cosmena from 2018 in this experiment she was trying to perform classification of visual observation and visual imagery periods between two object classes of either a flower or a hammer so you can see her task stimulus down here the participants would first be shown like a image queue of say like in this case the hammer it'll would appear on screen for four seconds and they'd just be asked to you know attend to that stimulus and then it would disappear for four seconds and that would be like an imagery period where they tried to you know imagine the stimulus that they just saw and then there would be a rest period so for her results particularly for like the observation periods when the picture was actually on screen you know she had decent classification accuracy however when the image was taken away and the participants were asked to just imagine it and she's only able to find around 52 decoding accuracy so right around chance level another study by leodoll in 2019 and repeated in 2020 in this study they were trying to perform classification of visual and speech imagery and they had 13 classes of like common words for patient communication so these could be things like um bathroom water ambulance or more abstract concepts like help or yes so for the visual imagery classifier when they first did this in 2019 they were able to get around like 27 accuracy which doesn't sound great but with 13 classes and chance level being around 8 like there's still something there you're able to classify between the categories and then when they repeated this a year later with a larger cohort of participants and some different processing techniques they will do increase this classification pretty remarkably up to around 40 percent so the next thing i'd like to talk a little bit about is some of the pre-processing steps that you would go through with working in a eeg bci so starting out you know you have this steady stream of raw eeg data come in one of the first things that you might do is perform a channel selection so perhaps at the beginning of a study you might work with like all the channels in the cap but sometimes depending on what the task that you're doing it might be beneficial to narrow down those channels to a particular region of interest so for example if you're doing a murder task where you have your participants moving their right versus their left hand you might be interested more in like those c electrodes directly over the left and right primary motor cortex or if they're performing like a visual task you might be more interested in the o electrodes on the back of the head directly over the visual cortex next step you would probably do is apply a filter to the data usually at this stage it would mostly be like eliminating any low or high frequency noise from the signal that you're not really interested in so eeg is usually grouped together in these bands of brain activity each of these bands kind of correspond to a particular state of the brain so for example like delta waves are usually seen in like a deep sleep like mu and beta waves are seen during like motor movements and executions and imagery alpha waves usually carry information about like subjects attention so if you're trying to see like if they're attending to a task like alpha ways usually carry that as an example some other filters that you might apply is just to remove noise like power line noise another step that you might want to take is perform some sort of artifact removal so eeg can be pretty prone to artifacts from like motion like eye movement from like eye blinking or muscle movement from things like chewing so the picture up above kind of demonstrates what an eye blink would look like particularly on those frontal channels if you're doing any sort of analysis that is taking advantage of those frontal channels it might be beneficial to remove those eye blinks also be a good idea to epoc the data so this would be like dividing the data into smaller windows so this would kind of depend on you know like the task period if you want to divide the data into like when the task starts and when it ends or maybe even break that into like smaller categories that you can process your data and classify it based off of and then finally would be like feature extraction so where you would extract the time or frequency components from the signal that you'd like to use as features during your classification so like for example maybe before you started with a broad filter of eeg like filtering from like 1 to 50 hertz but now you want to narrow it down to just use like the alpha band of activity for feature selection so you might use like a fft to pull out just that like 8 to 15 hertz range so next i'll talk a little bit about my experimental methods and the protocol that i used here so the objective of my project was to create a brain computer interface using a visual imagery control paradigm and the recording device that i used for this experiment was the brain products acticap express twist so this cap featured 32 channels of dry electrodes with a 500 hertz sample rate so usually with eeg applications you'd use like a wet electrode system so this would have a small amount of conductive gel placed underneath each electrode to improve the signal quality of it so unfortunately with the dry electrodes you know due to me having to work at home alone it's a lot easier to put on a dry electric cap however they are a little bit more prone to noise and you're a little bit limited by how much data you can collect because if you can imagine like 32 of these little spikes pressed against your head it's kind of difficult to wear that for a prolonged period of time so i can only maybe wear it for around like 30 minutes before i start getting a pretty bad headache so for like the overall framework of my my project here the programs that i use are open vive lab streaming layer cycle pi and python so for this i have like a participant you know engaged in visual imagery where they're you know imagining an object that they're trying to interact with the visual imagery eeg is recorded through openvive which transmits the signal by a lab streaming layer into a main bci processing script at the same time i have cycle pi running the stimulus script it's controlling you know when the stimulus is presented for the subject and you know when the feedback is presented and it's also sending the stimulus markers to that main processing script just to tell it you know when the image appears on screen when it goes off and everything so this main processing script is just taking in all that data and labeling it appropriately and sending it over for online analysis which carries through all the data pre-processing the feature extraction and the classification and that classification result is sent back to the psychopay script to display the feedback now most of the experiments uh that you'll see coming up are going to be like offline analysis so the only difference here is that we're not actually giving a feedback to the psychopay script it's just this main processing script is saving all the data for offline analysis so for the first experiment that i carried out really what we wanted to do was just recreate the protocol from natalya's study that we looked at initially mainly just to make sure that our processing pipeline was working as intended we were getting the results that we were expecting so just as a reminder this paradigm had first a fixation cross appear for a period of one second a blank screen for two seconds an image would pop up for four seconds and the participant would be asked to just attend to the stimulus and focus on the features of it we had two different classes of images of a flower and a hammer so it was always the same flower always the same hammer throughout the experiment just in a random order so once they focused on that for four seconds the image would disappear and they'd be asked to imagine that stimulus that they just saw there'd be a rest period for four seconds a blank screen and that would repeat so i recorded a single session that was divided into four runs and i had 40 trials of hammer images 40 trials of flower images and this was about like a little under 30 minutes worth of data collection total so further moving on to the results from this experiment the first thing that we wanted to look at was a classifier built off of the observation periods so the objective here was to perform a two-class classification of when they were looking at a flower versus when they were looking at a tool so some of the pre-processing steps that i took first i extracted the oc01 and o2 channels so these were the ones directly over the primary visual cortex in the back of the head i re-referenced those channels to the average of the tp9 and tp10 electrodes so those are the two directly behind the ears on the mastoids the data was filtered with the 1 to 40 hertz bandpass filter and a notch at 60 hertz to remove the power line noise and i e-pocked the windows so if you remember i had around four seconds during the observation period i kind of narrowed that down to a total of 3.5 seconds of data and split that up into window sizes of 1.75 seconds with the 50 overlap so they gave us like three windows for each trial and then i have for my features i used the full 1 to 40 hertz power spectrum i used a linear support vector machine classifier and cross validated it using a leave one run out cross validation so for the results of this visual observation classifier it was pretty good we were getting around 63 accuracy so it was relatively good at classifying whether they were looking at a flower versus a hammer and our accuracy of 63 was pretty close to what natalya was getting at 61 so it looked like everything was working as intended there so moving on to the visual imagery classifier so the objective here was to perform a two-class classification of when they were imagining that flower stimulus and imagining the tool stimulus so similar processing steps of extracting the oz012 channels in the back of the head re-referencing them to the ones behind the ears except now we're filtering the data with the larger frequency band so with the observation periods we use 1 to 40 hertz but the visual imagery it seems like the higher gamma range contains a little bit more information about imagery periods so we bump that filter up to 1 to 125 hertz with a notch at 60 and 120 hertz to remove power line noise and it's harmonic and use the same e-box size and our features here were the full power spectrum in the one to 100 hertz range and use the same svm classifier to leave one run out and that's for accuracy here unfortunately we were still getting around chance level accuracy we weren't able to fully identify if they were looking at or imagining a flower or a hammer however this was in line with what natanie was getting in her experiment as well so the next thing that we looked at was an observation period versus risk classifier so the objective here was two class classification of rest versus observation periods so most of it was the same except for we combined all the observation trials into a single label so this seemed to work pretty well we were able to pretty accurately predict uh whether you know they're resting versus looking at an image so around like 77 and a half percent where natalie was getting 75 so everything was going as we were expecting there and then we looked at an imagery versus rest classifier so the same kind of thing combining all the imagery trials into a single label and just trying to classify if they're resting versus imagining a picture and this actually worked really well like we were able to get 81 accuracy here where she was getting 77 so that was functioning as we intended as well so this was a pretty interesting result how you know we weren't able to differentiate you know within like the imagery period we couldn't really tell if they were looking or thinking about a flower or a hammer but we could tell that they were trying to perform visual imagery and when they were resting so the next thing that i wanted to look at was to try to evaluate this performance with different image stimuli so it was kind of interesting that we were able to differentiate between those categories during the observation period but not during the imagery period and one reason for this that i might have that i started to kind of notice that might help explain this was that you know the flower that we were using was a pretty large white flower and the hammer was like a smaller darker hammer so maybe during our observation periods we were really classifying like a bright stimulus versus a dim stimulus and that might be why we weren't getting any uh information during the imagery periods so to test out this i tried you know adding a few more pictures of flowers throughout the experiment keeping the same general size but having slightly different colors or different shapes and then for the hammers you know use the same like size hammers but maybe at different orientations and slightly different colors so interestingly enough here first with the visual observation classifier doing all the same pre-processing and classification stages steps as before our classification accuracy dropped down to chance level so before we were you know around 60 percent now we're down to chance so we weren't able to identify if they were looking at flower versus a hammer and same thing kind of with the visual imagery classifier again we were still around chance level it seemed to kind of bump up a little bit like maybe we had a little bit better luck with the flower category but still our overall accuracy was mostly around chance then for the classifier for the observation period versus rest periods we saw still a buck chance accuracy had a slight drop here but we're still getting around 69 percent we're before we were like around 75 and then for the imagery versus risk classifier we saw a pretty big drop here i think before our accuracy was like 81 now we're down to like 61.5 so the next experiment that i tried was um including different image stimuli of a face versus scene and this was just to see if we could try something different to help boost their classification during the imagery periods so the idea of using a face in the scene this is a pretty common image stimulus to use in a lot of visual neuroscience applications so there's multiple sites throughout the occipital and temporal regions in the brain that seem to respond pretty selectively to faces and scene images so perhaps the most well-known and well-studied one is the areas in the ventral temporal cortex the fusiform face area and the para hippocampal place area but there's also areas throughout like different areas of the occipital regions that seem to respond more maximally to either faces or scenes i want to be careful with saying that that you know it's not like the brain is really compartmentalized as like you know this areas faces the serious scenes this area's hammers like it's not exactly like that you know these activity patterns are pretty dispersed throughout the visual network but it does seem particularly for faces and scenes like there are some regions that are more maximally responsive to each one so perhaps this might give us a little bit better results and so i also study showing that you know face and seeing images have also shown to be effective in visual imagery eeg bcis so it's a pretty early study from 2011 where they tried to do a three-class classification of resting versus face versus scene imagery and they're able to achieve around 56 classification accuracy so pretty good above chance accuracy there so using these space and scene stimuli first off with the visual observation classifier performing similar pre-processing steps and everything our accuracy was still right around like 60 like we were seeing before so everything stayed pretty good there and then for our visual imagery classifier here's where we saw the biggest benefit so we're before we were right around chance level now using like all the electro channels re-referencing them to the ones behind the ears doing the same sort of filtering we're now getting like 64 accuracy so we're able to differentiate when they were thinking about the person's face versus thinking about the scene pretty well so one other thing that i tested here was getting a few extra sessions of this experimental paradigm so i had three sessions total of this visual imagery classifier i used the same pre-processing sets for all of them you know using all the channels except the ones behind the ears that i were using to re-reference doing the same 100 to 125 hertz band pass filter uh same epoch sizes 100 hertz power spectrum for features and so we're seeing for first off the face versus scene imagery uh from sessions one to three it looks like we're pretty consistently around sixty percent what was really cool about this data is when i tried to do like a between session classification so like test are training my classifier on say sessions one and session two and testing it on like a new session data that it hasn't seen we're still able to get like 60 classification accuracy which is pretty cool because most of the time with eeg applications it's kind of hard to transfer data from one session to the other then even for the rest versus imagery periods uh from sessions one to three it looks like our classification accuracy you know increased a little bit starting around like 60 up to a little above seventy percent and the between subject classification accuracy here i was still pretty high around sixty five percent so training on two sessions and testing on a left out one so the last thing that i tried here was you know splitting apart that rest and imagery periods into its individual categories so doing a three-way classification of rest versus face versus scene so from sessions one to three it was all performing pretty consistently above chance around fifty percent and even from our between subject or between session classifier of you know training on two testing on a third was still giving us pretty high around fifty 50 classification accuracy so next thing that we tried here was an online real-time vci application so you know having the participants you know imagining one of these stimuli and then giving them feedback directly from the classification output to see how it was working so in this experiment the first two runs were like exactly from the training ones you know we had the fixation cross blank screen an observation period for four seconds where it was looking at either a face or a scene image they performed imagery for four seconds a rest period for four seconds and then a blank screen for two seconds then in the last two runs it was a little bit different so during the observation periods instead of just having the two image categories they could also be queued to rest during the imagery period and then instead of a rest period at the end they were actually provided feedback directly from the classifier's output of you know what it predicted they were thinking about during the imagery period so as for the results from this experiment this the objective here was the three glass classification of face and scene imagery versus resting state i the pre-processing steps were all the same as before you know using all the channels except the ones behind the ears re-referencing them to those mastoid electrodes the same higher filter band the same e-box size then for the training data i used the three previous training sessions as before plus the additional two runs from the current session to kind of initialize the classifier and it seems like our accuracy was still pretty high like right above our right around where we were getting before with 50 you know we just saw a slight drop and looked like we were getting pretty good accuracy at identifying the rest periods and identifying when they were thinking of a face there was a slight problem with the scene images it seemed like that was more around chance levels getting confused with the faces quite a bit but it still seemed pretty consistent of you know identifying visual imagery was occurring and resting was occurring so the last thing that i want to show you here is just a demo of this real-time pci experiment so first two runs of this is like a demo of the initial training sessions so this would be like what happened in the first three training runs in the first two runs of the experimental session so in each run the person would first see the fixation cross and observation image here so in this case a face image then it would have the imagery period for four seconds where they're just trying to imagine that image and then a resting period and that would just repeat observation imagery and then rest so next moving on to the like real time feedback stages so those final two runs of the experiment so in this um they're seeing the observation queue of like a face in this trial performing imagery like trying to remember the features that face and then you see the classifier's output like in this case they got the prediction right so the next trial starts we see the fixation cross this time we're queued to rest during the imagery period so we're just trying to keep our mind blank during this stage and the classifier predicts risk so i got it right that time again new trial starts this time is rest again resting and then the feedback and a new trial starts this time it's a scene image pops up so we're focusing on the scene image and then trying to imagine that picture that we just saw and then for the feedback in this case it got it wrong it predicted as a face instead of a scene so moving on to just like a conclusions a summary of everything that we've seen so the objective of this experiment was to create a brain computer interface using a visual imagery control paradigm for the offline results that we saw we were able to achieve above chance classification accuracy when using imagery of faces and scenes so around like 62 percent class of classification accuracy between these two for the we also saw that the visual imagery was pretty stable across sessions so we had 50 balanced accuracy of rest versus face versus scene between those three sessions so one thing i forgot to mention before was that during these three class classification on the offline stages here i was using balanced accuracy instead of just the standard classification accuracy is because our um our data sets were a little bit unbalanced in this case because we had you know 40 trials of faces 40 trials of scenes but there was a rest period each trial so there's twice as many rest as each of those categories so this balance measure just kind of helps to create a little bit less biased classification score where instead of you know just judging how many totally got right and how many totally got wrong it finds the classification accuracy per category and then averages them together so it helped out for here because you know we're our classifier was actually pretty good at identifying the rest periods uh so if we use just standard classification accuracy it would have looked like a lot higher score than it actually was and then on to our online results we saw that we were able to achieve similar performance in our real-time vci around 47 classification in this three-way of rest versus faced and scene imagery some of the limitations of the study and potential areas for future work is that you know due to limitations of you know me having to work home alone i was the only participant in the study so definitely need more data for more people to fully validate the conclusions in this experiment i think it would also be beneficial to try to collect some more data perhaps from like a wet electro cap to improve the data quality so like i mentioned before with these dry electrodes it can be a little bit more susceptible to noise and we're also kind of limited on the amount of data that we can collect so like we were only collecting a little less than 30 minutes of data so perhaps getting a little bit more each session would help our accuracy a bit more and then finally i just like to thank my mentor this summer yvonne thank you so much for all the feedback and advice throughout the summer i couldn't have asked for a better mentor thank you to the entire bci team your feedback and all the support and help during our meetings was very valuable for getting the project to where it's at right now thanks david for helping me get all the technical side of stuff worked out at the beginning and thank you demetra for organizing a great morale event for us thank you to all of the all my fellow research interns in the audio and acoustics groups it was great working with you all this summer and getting to know you all a little bit and i wish you all the best and special thanks to natalia for you know meeting with me and giving me some feedback and advice on this project and thanks to all the research intern program coordinators who organized some fun events for us this summer to kind of give us a break from some stressful work times and uh thank you all for coming to my talk and listening to me and i'd be happy to answer any questions you might have thank you justin dear colleagues the floor is open for questions oh please go ahead hey they just did great stuff um just a i guess an experimental question in terms of how you uh like do the train test split uh so with the 50 overlap you kind of have to be a little careful in terms of making sure your training data isn't actually leaking into the test data and so were you kind of like breaking down each like three like three or four second epic into train tests and then doing the overlap or how exactly did you handle that that issue yeah so my experiment was like the full experimental session was divided into four runs so it allowed you to get like a little break between each rung so when i was actually you know testing the data like pulling out the data for training and testing all the training data came from three runs and then the testing was from a left out one so there was no leaking of you know those windows maybe overlapping in the trial like it was completely different trials that was being tested on great right thanks any more questions i have a quick question uh justin in the real time demo that you showed us one of the you had two paradigms and the second paradigm actually had a real-time feedback is that um something you're planning to use in the future or have you actually used it already um for learning during the training session yeah so during the real time um the paradigm was set up where like you know the first two runs were like the same as the training this was just to get some additional training data to help initialize our classifier i initially tried like you know fully training the classifier from previous runs that it hadn't seen before and just doing fully online feedback but without like giving it a little bit of data from the current session it didn't perform as well like it's i think around like 38 39 accurate but increase but adding a little bit of training data from the current session boosted that up to around like 48 so i did actually have fully online um vci and accuracy you can see the predictions here it was around 47 got it is there so for the feedback that you're showing to the participant what is the benefit either to the participant or to you are you recording the reaction to that feedback and you may use that as an additional um signal yeah at the moment right now i wasn't doing anything with the data during the actual feedback periods but one thing that you know we were kind of initially planning to do at the beginning of this was creating an adaptive pci to where you know you can use the data from that feedback period and see like you know if you see like maybe a an error related potential in that in their brain waves during that feedback period we could see that you know the classifier got it wrong and we could try to you know more finely tune our classification between that but as for right now i wasn't using any of the feedback data got it thank you thank you more questions well some people think about a potential question justin one of the most interesting findings here is the independence of the classifier from session to session we usually see a drastic difference and reduction in the accuracy when we have a classifier training on one session and try to use it in the next session how would you explain the fact that you can train a classifier on two sessions and use it on the third without substantial reduction in the accuracy yeah that's a great question i mean yeah a lot of times with these you know eeg applications you really do see a pretty drastic drop between the sessions i mean this could be from many reasons of like you know fatigue during the session or even just the electrodes being at slightly different positions so it was pretty cool that you know in my data set that i had right here i wasn't seeing any drop between that and honestly i'm not really sure exactly why uh we weren't seeing a huge drop here um i mean definitely to make sure that there isn't a drop with visual imagery we definitely need to try to test this out on some different participants instead of just myself get some more data to actually validate this but yeah thank you uh more questions dear colleagues oh come on go ahead um hello uh can you hear me yes he came for you hi um a great talk i really enjoyed it um i was wondering if um there's any um thoughts about sort of analyzing the features that might be allowing for the separability in the different classes maybe like looking at the different power bands and what actual features will be uh contributing to the separability yeah so what i was doing right now like what i saw with a lot of the um other kind of similar visual imagery bcis it seemed like using uh features from the frequency spectrum particularly the high gamma spectrum of brain activity seem to carry the most relevant uh information from visual imagery however that one study that i talked about that had like you know the 13 different classes that had 40 accuracy they had a little bit more advanced uh feature selection they were using like common spatial patterns so definitely there could be some more work on this side to see if you know using some different types of features could help improve her accuracy some more right now you know just kind of starting with a little bit simpler uh just frequency spectrums and seeing how that worked good great question thank you more questions okay i just i'm gonna wait for my turn if someone else has a question um so justin you in this project you have been both the principal investigator of the study but also the participant and typically what the one wants or is expects is kind of conflicting with what the other side of of this study um the other side of the of the coin i suppose one one of the issues you mentioned is that for example as a private investigator you you wish you could have more data recorded but as a participant at times it may get a little bit painful so you couldn't leave the device on on your head for long periods at a time is there anything else that you may have observed from these two sides of of this project regarding the stimuli perhaps improvements that could come because you've been taking that experiment yourself that study yourself as a participant yeah so that's a great question some of the things that i noticed about this experiment um i mean first off you know it did seem that we saw some improvements like when we were you know adding that additional data from other sessions so it does seem like you know we are a little bit limited just by the amount of data that i was able to collect from myself each session uh one other thing that i noticed that i didn't really talk about in these slides one of the things that i tried to you know the very start of the summer was having three different classes of you know like face images scene images and then having like an object category like showing like hammers and tools and things like that someone uh interesting aspect that i if i had more time to kind of dive into that more is you know trying to add some additional categories in here because even with those three different imagery categories you know some of our results were actually looking pretty positive there uh in this i was actually able to get two different participants in that myself and my wife participated in that and we were both around like 40 to 45 percent between all the three of those categories so it did look like we were still able to get a pretty good above chance classifier between three different image categories instead of just two is here and i wish i could have gotten some more data but yeah after my wife tried it the first time and realized how painful it was she refused to do any more of it so all right that's interesting yeah thank you um yeah it's a big struggle uh trying to get um participants especially when you're doing a virtual internship um or working virtually um yeah uh justin we have a question in the chat see if i can pull that up or harvey if you prefer to just read it okay i can see it do you have any ideas for doing this at home at scale having people help out being shipped equipment giving you experiments worked from home it's both the investigating subject um so yeah carrying out this experiment you know at home like if we were trying to you know ship people equipment i think that would be kind of difficult uh sharing equipment around um you know the g caps they are relatively inexpensive but they're still not super cheap to send around to people and you know it'd be hard to rely on people having equipment like that at home so yeah working from home is definitely a challenge with experiments like these that do require specialized equipment it kind of requires a bit of uh lab space to do the same it is actually not that easy simply put not everybody has the skills to put properly the eg cap and to make it to work and to get the data we have didn't done in the bci team in microsoft research the opposite thing when we have the equipment installed in a lab and members of the bci team go and one per day take turns to put the equipment and to record the data this is how we did the data collection for one of our studies but those are people members of the bci team with certain skills and ability to work with the equipment unless the equipment is enormously simple to use that could be a problem more questions dear colleagues yeah there's quite a bit to check in you know making sure each of the electrodes have proper placements and getting a proper reading like having the full contact with the scalp and then just getting all the programs set up like i have four different programs running at the same time you know collecting the data transmitting it processing it presenting the stimulus so it would be a little bit hard to set up if you're not fully experienced in it okay more more questions dear colleagues more questions once more questions too okay i say we thank justin for his talk thank you justin thank you thank you all and thank you everybody for coming to this meeting uh if there are no more questions i would say this stock is our thank you again for coming to the final talk of justin kellmark's intern in audio and acoustics research group let me stop the recording and how to do that