Exploring a REAL Microsoft Code Base - Industry Code Walkthrough

Key Takeaways

[Music] hello everyone and welcome back to the YouTube video so in today's video what I'm gonna be doing is exploring and walking through a real Microsoft code base now the reason I'm able to do this is because I'm working at Microsoft is a software engineer intern and I'm working on the Python extension team for Visual Studio code so if you don't know within Visual Studio code one of the more popular editors out there right now there's all these different extensions and what these extensions do is just bring different functionality different features syntax highlighting whatever there's a ton of different things that come with these extensions one of the more popular extensions is the Python extension which you can see has been downloaded by just over 21 million people and is supported and maintained by Microsoft now this is an open-source extension which means pretty much all the code that you are looking at or that I'm gonna be showing in this video actually all of it is available publicly online from this repository which I'll leave a link to in the description so here you can see all of the code for the extension that's publicly available because it's open source and this is what I work on every day I work on an open source project which means I can actually show you guys what I've done which i think is really cool so with that being said I will mention that this entire code base is pretty much written in typescript there is some react as well for the UI related features and I mean I'll be talking about that as we go through but this is quite complicated code it's not something that I imagine you guys will completely 100% understand but I am still gonna explain it and walk through some different features and since the extension is so massive I will kind of talk about you know some different random areas and try to give you a taste to kind of everything without going far too in-depth on one specific area all right so we're pretty much ready to get going the last thing I'll say is actually right now there's a really awesome deal going on on a bunch of pre premium programming courses if you guys are interested in potentially purchasing a whole whack of premium programming courses there's actually 33 of them that we have bundled together right now I've teamed up with 26 other content creators authors people that are well respected kind of in the industry and as teachers and they've taken all of their premium programming courses and I have a premium programming course as well and we've bundled them all together this is usually like thousands of dollars worth of courses if you combine all the value of all those courses together and we're selling it for a pretty good discount actually is about 94 percent discount I don't want to like will you with the big numbers you can check out all this stuff in the description down below but I just figured I'd make you guys aware and this is a great way if you want to support me or support some of the other creators to get a really good deal on a ton of different premium programming courses this is only gonna be available from June 22nd to June 26th and I believe there is an early bird discount if you purchase it on the first or second day anyways I don't want to talk about it too much there's a link in the description and feel free to check that out and just see if it even is of interest to you alright so let's actually start exploring this codebase walking through some different areas and yeah just kind of getting into it now I will start by saying that this codebase is massive there's a huge codebase I wouldn't anticipate or expect anyone to be able to understand it even just in a few months of actually using it because there's so much content in here and you'll see myself there's some areas here that I just don't actually know what they do or I haven't been exposed to them yet so I have a decent understanding but I'm gonna give us kind of like a high-level overview of the structure of this codebase then we'll get into some code that I've written myself that I do truly really understand and I'll talk about how that kind of works and that should hopefully give you a good idea of how we've done things inside of here and some of the design patterns and all that so the first thing that I'm gonna do here is just kind of show you the structure so you can see that there's a bunch of different folders here and a bunch of different random files now the core technology used in this code bases typescript so typescript is the main language and what happens is when we use this extension we compile that typescript code actually transpile it into JavaScript and then run that the reason we're using typescript from what I've heard is because it's way better to have a typed language when you're working on a project this large the reason for that is that I can just hover over things right and see exactly what type these items are and I can actually go and look at definitions of interfaces of classes I can click on for example like react thoughts CSS properties and go go to go to definition and I can actually see all of the stuff related to that so it's very powerful and it just makes developing and navigating a codebase way easier when you have strict typing and typescript is nice because it actually gives you the option to set typing or not so if you're in a situation where you don't want to have strict typing you can just not set a type for that variable or for that object or whatever it may be I mean it means you can save a little bit of time right but for example like four grid columns I can see what all of these different items are I can see if they're optional or not and it just makes it a lot easier when you're actually doing development now of course you may have noticed here that we're also using react so we use typescript mainly and then react as well to do all the UI related features we also use redux as a front-end state manager and also to handle communication between the extension and between IB actually you op so that leads me into the first thing that I wanted to say and I want to jump into this SRC folder so I'm not really going to talk about most of the other stuff here cuz it's not that important or interesting but SRC is where all of the source code for this project is inside of here we have five files and I'll briefly define what each of these are so client this is where most of the stuff happens client is where will be called the extension lives so because we have a user interface we need to have some kind of front-end back-end and a separation between them so we have the client which we call the extension which runs in a separate process and then we have the data science UI which is a user interface which runs again in a webview in a separate process so immediately we have two processes what that means for anyone that knows anything about multi processing or multi-threading is that we have to have a very robust communication system between these processes because when you have two separate processes you may have things like just sharing memory is not that easy being able to talk to one another I can't simply just use one of the classes from another process I need to actually send a message to that process have that message handled have a request sent maybe I send that with a payload some information some data and then have some response sent back so it gets very complicated very quick so those are the two main folders you have I pie widgets which is very specific to ipython widgets if you don't know what those are don't worry I'm not really gonna discuss them and then tests so inside of here is where we write all of our automated tests now we have a CLI on the github repository and what that does is make sure that every time you submit a new pull request so you make some kind of change to the repository it runs all of the tests that we've written here and make sure that they all pass so we have unit tests we have functional tests believe there's integration tests all different kinds of tests that test pretty well every feature that we have and everything that makes sense and you go for a pretty high code coverage you want to really test a good majority of the code that you have in the extension to make sure that when you make a change you don't break something and something some test fails essentially right this also means that we don't have to manually test anything every time that we go ahead and make it change we can just wait for that CLI on github to run and it will tell us hey you mess this up hey you need to change this and that whatever that's the point of test so anytime that we write some kind of new feature or we add or make a major change we write automated tests for that feature probably functional and unit tests maybe sometimes either/or depends on how big or what it is and within this we actually use two different testing frameworks one of which I believe is called TS mockito and then there's another one that is called something else was vs code test so I don't know what the other testing framework is forget the name of it but the main one I believe is TS mockito and that essentially allows you to mock objects I'm not really gonna go into all of that because that's kind of somewhat complicated but anyways that is the main idea behind kind of how this is all set up so I could of course go into clients and go into data science UI and show you all of the different features I'm not gonna do that what I will show you though is one area that I am familiar with in data science UI and kind of just talk a little bit about how that works and then I'll go over to the extension side and I'll talk about how we actually handle communication between these two processes and actually one of the new features that I've been adding so let's go into intro active common here you can notice that there's a bunch of different folders and these all represent different areas of the UI the way that this works is that we actually render what's called a webview so it literally is pretty much like kind of a mock HTML page or yeah like a mock browser inside of es code and that's where we render all of the react related components and all the UI related features so if I go here to variable Explorer this is something I'm quite familiar with because I've used it and I've been working on it we can see that this is react so if you look here we have a class it extends a react component it has a variable Explorer props and variable Explorer state now I don't really want to explain react to everyone but essentially there's two main areas or two main things in everything that we call a component now what a component is is essentially something that you render within HTML so you have this main parent component which is kind of like your app or your main window and then within that you render all of these other sub components and the whole point of doing that is that if I make a component for example like a variable Explorer which lets me view a bunch of variables I can render it in one area and then I can reuse it and render it somewhere else so it's just a better way of writing kind of complicated HTML related UIs and all those kind of things and what you can actually do inside of these components is write JavaScript or types great typescript code in this case and you can handle state of these components and properties of these components so properties are something you pass to the component when you render it for example say I want you know the background of a component to be blue then through the props of that component the properties of it I pass that I want to render it with blue then I look at that property inside of this component and I modify the HTML accordingly that's essentially how that works within each component you also have state so for example say you have a height something's resizable that would be a state of the component so you have a width that would be a state so you have the amount of time someone's click the button on that component that would be a state of the component and every time you change state in a component where you change the properties it rear Enders the component so that's the way that this works and this is what we use so I don't really like that's as much of an explanation as react as I really want to give you but this is the UI related stuff we just used react components and you can see for example here I'm rendering a draggable component that has all of this stuff inside of it this language inside of here is called JSX in here it's actually TSX I believe that stands for type script something but it's like a hybrid between typescript and HTML and you can see that I can actually write for example I know this inside of my HTML code and all of my classes are not defined by class they're defined by class name because that's what JSX uses where TSX uses as their kind of syntax for rendering HTML so anyways that's a whole nother lesson on its own is typescript and all that stuff but essentially what happens is you have a method inside of your components called render and that will be called automatically when any of the states are prompt change and then you have lifecycle methods which are something like should component update component did mount that happen when certain events are triggered so say the component just mounts so that means it just appears on the screen then what you do is well you can kind of hook in to this method and and do something so in this case what I'm doing is setting the initial height of the variable Explorer every single time this component mounts on the screen amounts actually I don't think that means it like loads I think that means sorry it means it loads it means it hasn't rendered yet but it loads so I set this before it renders the initial state onto the screen anyways again not super important but some of the examples of something that we do the UI is fairly straightforward it's just using react and if you know react then you understand pretty much the UI component okay so now what I'm gonna do is talk about how we actually send information between the user interface and the extension now the user interface you have to remember and imagine really is only responsible for rendering the UI right maybe it handles things like if you press a button maybe it you know displays information in a nice way but really it's not meant to do any super computationally heavy things it's not meant to store data persistently it doesn't save files it doesn't do a lot of the things that the Python extension does or needs to do so we have to have a way for essentially the user interface to connect and interact with the extension which has a lot of that heavy lifting or does a lot of that heavy lifting now a really common example is something like storing state so here I have this variable Explorer I know I haven't showed you guys what this variable Explorer is but essentially it's just an item where you can look at the state of variables in an I Python notebook now when you open this thing up you can resize it and you can change the size of it and you can toggle it so it's viewed or it's not viewed now obviously if I set it at a specific height and I close it when I open it back up I would expect that it stays at that height I don't want it to automatically resize to the default height every time because then I just have to constantly keep resizing it right so a problem that I solved this start here at Microsoft was let's make that happen let's make it so that first of all we can resize this variable Explorer because it was not resizable before and that if you resize it to a certain height it stays at that height unless you change it right so the way that we do that is first while I'm in the UI here so I'm in variable Explorer dot TSX I'm on the what's called not the client side the data science UI and well I have this method I've called in save current height that says this dot prompts dot set variable explorer height and then I pass the height of the container and the height of the grid this is just what the information we need to actually store the height of this variable Explorer and well I'm calling a method that says set very low Explorer height with that information now you don't need to know where I call this from or kind of how all the details operate but just imagine that what we need to do here in the UI is we want to set that height and we need to do that we need to do that setting from the extension side because we can't set and save persistent data from the user interface we can only do that from the extension which is running in a separate process so we need a way to communicate with it so what this actually does where this method is actually defined is inside of redux so redux is what allows us to actually kind of handle state changes and commute in between the extension and the user interface now there's some other things that do this as well but Redux is kind of one of the core tools that we use and I don't really want to explain too much into how Redux works but let me just go here to variables TS and types TS and show you what I mean so inside of types TS which is inside of the reducers folder here what we have is actually a definition of a bunch of different messages that can be sent between the user interface and other aspects of the user interface or the extension so for example we have ADD and focus new cell this is a message that we could send saying hey you know comment action type this happened send this through the extension let the extension handle it over there so essentially what we're doing is we're sending messages between the extension and the user interface were not for example like I'm not importing a class from the extension and using it I'm sending a message and then that message will get handled on the other side so it's a form of communication some kind of protocol that's going on so we define all of these messages here I'm trying to find where the variable Explorer one is we have toggle variable Explorer first of all that's a message that could be sent and set variable Explorer height so that's the message that we want to send when we set the variable Explorer height from the user interface we want to send this message to the extension so here inside of common action types let me see if I can find set variable explorer height so let's just ctrl F there and we can see that when I have common action type dot set variable Explorer height I am mapping that because this is a common action type mapping to the payload of I variable Explorer height so what this is saying is that when I use this message I want to pass information that is inside of them like that implements this interface to the extension so essentially I need to send data with this message as a payload and the data that's gonna come with this message will be I variable Explorer height now I variable explorers height is a interface and you could see that it's defined here as container height grid height so exactly the information that I'm sending to numbers is what I'm stating inside of here so if we go here that I want to send with this action hopefully that is clear so once that happens once we send it the action ah let's go to do I gotta find how all of this works now there's so many files to uh to explore let me go back to read X here so let's say that that action is sent and if I go back to variable explorer I'm just gonna have to tab back through a bunch of different things here okay so now we're in variable Dutch yes so this file here is inside a redux it's right beside types and what this does is essentially handle when a message gets sent what happens so when I send a message from the user interface so I called that method set variable explorer height from the variable explorer magically I don't want to go this far a message with common action type dot set variable explorer height gets sent it just gets sent out through read X and we pretty much say hey this is the information I want to send redux take care of it so that's what happens in the UI so redux says okay if we sent this message or if we sent this message whatever messages are sent here we're gonna accordingly call this mapping so what this is doing is pretty much mapping all of the messages that we want to handle inside of variables yes to their respective functions that are gonna handle the communication so we're still on the user interface side here and through redux I send a message that says hey we want to set the variable explore Heights with this information Redux picks that up and says okay you know I've got this message and then what it does is that says I'm gonna map this message to this function so let me go to this function here set variable explorer height and you can see that inside of here it's argument is a variable reducer argument that is I variable explorer height what this means is essentially we're wrapping what the information that we're going to send which is I variable Explorer height and we're going to return a new I variable state now this gets really confusing but all you really have to understand here is that I grab the information out of this payload which is what it's called and then what I do is I post to the extension this information so what I say now is okay we handle this message from redux okay Redux has taken this it's grabbed this information it's gonna do some kind of logic here and then what it's gonna do is send to the extension the argument which is going to be the payload here so it's gonna be the data so container height and grid Heights with the message interactive window messages set variable explorer height and then the container height and the grid height so what this is saying is let's communicate with the extension side and let's send them this information so that on the extension side we can set the variable Explorer height so if I go to interactive window messages set variable Explorer height let me go to references for this what I'm gonna do now is bring us over to the extension where this message will be handled so essentially all of this happened on the UI and then Redux said okay let's talk to the extension by sending a message so now we go to the X let's see here the extension I'm trying to figure out where I want to go into interactive base and if we look at interactive base now we're inside the extension side so we're in a different process we're separated away from the UI and there's this big on message event here that happens that it pretty much says hey when a mass message gets sent handle it here and then there's all these switch cases for all the different methods that can be sent or all the different messages that can be sent one of which is set variable explorer height so essentially we say if the message that was sent was set variable explorer height called this method with the payload and ignore any errors that happen there so what we can do now is go to this method so go set variable explorer height and here this is where the logic is handled to actually store this variable Explorer height so we say if payloads undefined then the updated heights equals the payload as this interface here or this object what we're gonna do now is some fancy thing that again is difficult to explain and essentially store this information in local storage so that if we want to grab it in the future we can and in fact right here this variable Explorer height request and what this does is simply get the information that's stored in storage and send it back to the user interface so that the user interface can take States and in rendering that that's what it does now I know there's there's just so much to explain here I'm trying my best to go as thorough as I can but let's go to one more thing now so after I've just done that this communication between extension and user interface let me talk about a kind of set of features that I'm working on right now and how this works so what I'm doing actually is working on a feature to be able to export a ipython notebook into PDF HTML or a Python script they already have Python script and v/s code but there's no feature for HTML or PDF so I've been working on that and I can say gladly that is pretty well fully functioning there's just a few more things we need to kind of iron out so what I have here is this folder called export on the extension side keep that in mind it handles all of the exporting and actually taking a file and changing the format of that file which is not that easy to do so here what I have is export manager dependency checker export manager file open our export manager file picker there's a lot of different things that have to happen when you want to export a file to a different format right and you can imagine that when I write this code what I'm trying to do is set it up in a way that I can very easily add another format to export to you because in the future we might want to export to something different right so I want to set up this code scalably so that it's very easy and I have a really good infrastructure if I want to add something more so I'm gonna start at export manager file opener because this is the kind of entry point into this feature so anytime that we want to actually export to a specific format this class will get called first and this export method will get called now before I even get into all this and start looking at some of this stuff what I want to do is talk about this injection so in this code base we use something called dependency injection and something called inversion of control now this is pretty much just a practice it's you know it's like a design pattern inside of software engineering which allows you to have all these kind of services if you want to think of them that way that run independently as what we call Singleton's that you can access at any time by simply in in them as a dependency into a class so here you can see that what I've done is I've injected the export manager dependency checker as the manager into this class in its constructor I've also injected the I document manager and this is another class that simply manages documents right I've implemented or injected the progress reporter the file system the application shell and the browser service into this constructor what that says is immediately when I start using export manager file opener pretty much find where these classes are and grab them bring them in and reference them using these names which means now I can use this service inside of this class to make a class injectable you simply define it as @ injectable and you add it into something called the service registry ok so I want to show you the service registry here what this does is simply register all of the services that we want to use in our dependency injections so whenever we have a class that we want to potentially inject into another class as a dependency what we do is register it inside of here and this pretty much just tells our program when we go to inject something what we take and where we get it from so what I've said for example in this export manager here is okay I'm gonna add a singleton which I won't really describe which is type I export manager so it implements this interface I can go to this interface here and you can see that all that says is that it must have an export method that takes a format and a model so that's what this type is we're gonna call this export manager and we're gonna use export manager as this class the idea behind this is that I don't really care what class I use when I inject a dependency I just simply refer to it using some name and then I get given the service corresponding with that so for example inside export manager file opener here if I implement or I inject export manager dependency checker I don't actually care what class implements this I just care that it works and it does what the interface states that it does so again we're getting very complicated here but the idea being that if at any point in time I actually want to change the implementation of export manager dependency checker all I do is change the class that this name references and nothing gets affected inside of any class that injects that dependency because it doesn't know what class its injecting it just knows it's injecting a class that's referenced by a name so I know that's confusing but that is how this works that's how the service registry works and this is what allows us to inject these items in here we state them and define them in the service registry and then we can use injection to take them in to any class from any point in this extension so that is how that works on the dependency injection so what I do here is essentially I have this is very confusing to explain because there's a lot to go through which I don't necessarily want to but this just takes some export format it takes the model so the notebook that we actually want to export does a bunch of stuff has a bunch of telemetry happen and that actually delegates all of these tasks now to the export manager file picker so it pretty much says okay we've called export manager file opener this now calls the export manager file picker the file picker actually says hey we need to get you to choose where you want to save this exported file to it gets it to choose that and then this now delegates to the export manager and says hey we picked that we wanted this type of file we want to save it here let's send that request over to the export manager so we can actually handle doing the export the export manager does a switch on the different formats takes the model whatever file the target that we need to save to that the export manager file picker has determined then it delegates to every specific export method so export to HTML is here PDF is here which isn't done yet and Python is here so essentially we have all these different classes they're responsible for very specific things we have an X where base class which has some functionality that is shared between all of the specific export methods and what we're doing is we're saying okay what we're gonna need to do when we export is we need to first of all open the exported file right once that file is exported we're gonna need to open that so we start with the file opener because if we call this opener if it can get an exported file from somewhere then it can just open it right that's all we're doing inside of here were pretty much figuring out hey did this export happen successfully if it did open the exported file great that's all this class needs to know and all it needs to do now export manager dependency checker okay well we need to call this because we need to make sure that we have the dependencies installed on the system before we can go ahead and actually pick files and start exporting something let's say this one works right okay boom everything's working fine then what we need to do is go to the export manager file picker we need to actually pick some file that we want to export to and then from there we need to actually export the file so once we pick where we want to save this file let's actually go now to export manager so delegate the tasks to him he'll take whatever we passed and then automatically say oh okay so if we're doing Python I need to do this if we're doing HTML I need to do this and then call one of these three specific classes which performs the actual export operation and then we go back up the chain and we eventually get you export manager file opener and now we're at the point where hey we successfully have a file we created it and we open that file up and that is the idea behind this export kind of folder and feature and functionality that I've been writing here and that is pretty much as much as I think I can explain without getting too exhausted going through this code base so I think I'm going to leave it here this is a difficult video to film because I want to show you guys everything but I can't write and there's a lot of stuff that has kind of prerequisite knowledge need to understand and I hope that at least I kind of give you guys some kind of idea of what's going on here and how things kind of work you don't need to understand all the specific code but now maybe you understand the complexity in a large-scale software system there's so much stuff happening and just even finding a way to communicate between different processes that are running is difficult in its own right and you even saw there's some messy code here because when you don't do something right the first time or you do it not a hundred percent correct you fall into the trap of just continuing to do it that way until you to a point where you have these massive files with a ton of stuff going on and it's too late to even change anything so with that being said I think I'm gonna end the video here if you guys made it to the end give yourself a pat on the back I'm sure not many people will make it this far and remember there is that big course discount happening right now for one week so June 22nd to June 26 again more information about that in the description so if you enjoyed make sure you leave a like subscribe and I will see you guys in another YouTube video