Financial Sentiment Analysis with FinBERT & HuggingFace + Analyzing Model Predictions w/ W&B Tables

Key Takeaways

and so now with just one click we can turn this table into a plot that we can visually explore let's check it out [Music] hey what is up everybody iron from weights and biases here and in this video we'll be analyzing sentiment of stock market news headlines as being positive negative or neutral um by influencing a hugging phase model on a large data set that contains stock market news headlines and we'll also be using wnb tables as a way of interactively exploring tabular data and suddenly checking our models and making sure that they perform properly wnb or weights and biases is a machine learning tools platform and tables is one of the tools at our disposal tables allow us to log images audio and text data and interactively explore them right in our browser tables are useful for exploring data sets or still being in this video visualizing model predictions so now let's really quickly talk about what is finbirth so fingert is essentially a version of google's famous bird model which was fine tuned in financial data what we're trying to do is to analyze sentiment of stock market news headlines and those involve a lot of jargon that's very domain specific and just normal english language models they do not necessarily generalize well to the financial domain and so that's why we're going to be using finbirth which was trained in financial data and which can do a lot better of a job with analyzing that sentiment so right now we're looking at a google call up notebook which is used to load the data which in our case we're using a data set that the model was not trained on we're using a calculate data set which is called daily financial news for six thousand plus stock which was scraped from the internet and contains a lot of the stock market news headlines and that is what we're interested in then this call of notebook inferences the hacking phase invert model on that data and logs it to weights and biases and logs the model prediction says wnb tables you can log a wb table by formatting a pandas data frame and then just simply logging it with one line of code and if you're looking to log and explore some data using wb tables you may also find documentation useful to help you get started and i will leave a link to that in the video description so when the model inference has finished and you've logged the data to weights and biases you can click on this link to open the run page so the way wmu works is that we can have a project and we can log multiple runs to it so you can think of this as us being able to log model predictions for different models or for different data sets and then being able to compare all of them in a single project page in a single dashboard like this one in our case we're interested in this specific set of model predictions visualized as a wnb table so let's first take a look here we have our headlines um here we have the stack to which the headlines attributed to and we have the sentiment analysis predictions as being positive negative or neutral first thing we can do is sort the predictions in ascending or descending orders and try to find say the most negative headlines so i'll communications fell 18 percent euros traffic rail down six percent full 7.9 let's try also sorting by the positive headlines hurricanes ratings improved rbg rating is upgrade improved cannot continue revenues increase here we can see the word improve appearing the two most positive headlines so let's use wnb tables to check if the word improve always biases the model to output positive predictions all right so the first thing we can do is to remove the run name column because right now we're dealing with a table that only has the predictions from one brand so this column is essentially the same for all of the headlines so i'll just remove it now we want to create a new column that will check if a given headline contains the word improve here's how we can do it so i'll click insert one right there will be a new column then i click column settings and here i say row headline headline that lower contains improve and now we can group by this column so now we consider the model predictions for a group by whether the headline which was input into the model contains the word improve or not you can see we can see those headlines here but i prefer a way of looking at them by also inserting a new column going into its settings selecting the page size of one and being able to see a given headline and the model predictions for uh that specific headline we can probably just remove this one and we can probably also now remove the stocks column because we can see the stack for a given headline here so now we're going to have our headlines as you can see we have 20 of them containing the word improve and around all the other ones not containing that word now we can look at the value distributions for the positive negative and neutral predictions for the specific group of headlines now here we can see that 16 of the 20 headlines that contain the word improve are highly highly positive so it kind of leads us to suspect that the word improved does correlate at least correlate at this point with the model analyzing the sentiment as being positive but to be sure we can go here and say column settings and we can average the different classes and as you can see now what we suspected by looking at the valid distributions turned out to be true which is that the headlines that contain the word improve are an average 88 positive and only three percent negative and seven percent neutral so there seems to be a very strong correlation where the headline contains the word improved then it's gonna be classified as positive but just looking at the numbers might not tell us the full story and that's why we can make a plot which will contain the headlines and their predictions so first thing i'll do is i'll remove the averages here and ungroup the entries by whether they contain the word improve so what we gotta do now is to click on the gear icon and click plot table query and so here we can see an interactive dashboard which contains all of our headlines and where they fall in the distribution of being positive or negative according to of course the thin birth model predictions as you can see right now the x dimension is the positive column the y dimensions the negative column and the label is our headlines so as you can see we got a lot of headlines so we got a lot of different points on this plot and this plot is really customizable and so what we're interested in finding out is looking at where do those examples that contain the word improve fall in comparison to all the other examples so what we can do is we can instead of the label being our headline we can make the label be headline which headlines which contain the word improve will be our label and for toll tab aka for the text that shows up when we hover with our mouse over a specific point like here we'll set it to just you know being heavily so that we could then hover over a point and read what headline it's actually about and now we can click apply and voila now we can see that all of the yellow points are the ones that contain the word improve and as we can see it's true that they all land like most of these yellow points they land in the area of being really really positive so now let's look at the actual examples if we zoom in here we can see um that this one is shows improved relative price strength improves parkinson's symptoms improved picture improved operational performance neurogenetic improved technical strength so now we can see that the headlines that are model classified as being positive which contain the word improve are actually pretty positive if we read them ourselves and we're not able to see any sort of bias that our model could have towards the word improve so it's ended up being a bit of a sanity check in this regard but who knows maybe one of these days me or you will be able to catch some inconsistencies and be able to diagnose many sort of problems or bugs which may arise in the pipeline of training among models and so now let's look at the examples with the word improve that i classified to not be that positive so for instance this guy here baldwin and hardware learning brand of the hardware and home improvement division of so as you can see here the word improve appears more in the way of describing a an industry and it's not saying specifically that something has improved so here the model also performs properly and now let's look at this point also humana mr for earnings expectations but improves year-over-year analyst block so as you can see here it misses q4 earnings expectations but it improves the year over here and yeah that seems fair to me looking at it so it's not so it's not as positive as say this examples with the word improved because it also misses some earnings expectations so as you can see the model is classifying the examples with the word improved properly and now we can just have the peace of mind that the model actually works as expected so that's it for this video and i hope that at least picture interest to explore more about what wnb tables can do links to the code that is used to generate this table will be in the video description and link to the table docs will also be in the video description so thank you for watching this video smash that like button to let us know that you enjoyed it and consider subscribing to our channel to see more tutorials interviews and talks and thank you for watching this video i really hope that you enjoyed it and found it useful