Internet of Things Optimization

Key Takeaways

hello world it's Suraj and the internet-of-things it's definitely a buzzword these days but it's a real thing and I'll demonstrate how to use reinforcement learning to optimize electricity consumption amongst multiple devices in a smart home mark and driessen's quote that software is eating the world still rings true today more and more devices are coming prepackaged with internet access that wouldn't normally be that includes smart soul-shakers smart rectal thermometers smart mugs smart chairs the list goes on and because these devices have a connection to the world wide web they can communicate with the outside world and each other sharing information and even learning from one another this internet of physical things is becoming increasingly common across the globe whether it's a smart home a fleet of self-driving cars or a pipeline of assets as they are shipped across the globe having a web of interconnected physical devices has a huge range of use cases in our modern world for businesses and consumers but a major concern regarding these devices is the amount of energy they'll need to consume silicon chips need electricity and the more devices we have the higher our energy cost will be it turns out that we can use reinforcement learning to create a system that will minimize our electricity costs substantially this can apply to a whole range of industries where an interconnected system of devices is required to perform a task agriculture manufacturing and even a personal home which will be our demo google recently used RL to reduce the amount of energy they used in their data center by up to 40 percent just like a laptop generates Heat their massive racks of servers generate a lot of heat but too much heat can damage the servers so a cooling system is necessary to help maintain a certain temperature and because it's such a dynamic environment a learning system is required to set that temperature a whole host of unforeseen scenarios like changes in weather and power outages can occur also humans interact with the equipment in unexpected ways regularly probably doing the floss dance on them and each data center has a unique architecture and environment so their system needed to be able to adapt to multiple unique environments last week we introduced the fundamental way of framing the reinforcement learning problem where an agent is interacting with an environment to maximize a reward the Markov decision process the goal of an agent is to learn a policy so it knows given a state the best action to take in order to maximize a reward and in a complete Markov decision process where all of the environment variables are known we can use dynamic programming to learn an optimal policy but what if we don't know all the environment variables beforehand give up no then it'd be considered not model-based RL but model free RL in model free RL the first variable we miss is a transition model so we won't know what's going to happen after each action we take beforehand this tells us the probabilities associated with various state changes the second thing we miss is the reward function which gives the agent the reward associated with a particular state beforehand so when we don't know either of these Markovian variables dynamic programming won't work we need to instead use a different type of method called Monte Carlo Monte Carlo methods are a broad class of algorithms that rely on repeated random sampling to obtain numerical results the key word here is random Monte Carlo methods make use of randomness to solve problems which turns out to be very useful in mathematics and physics Stanislaw ulam invented it in the late 1940s while working on a nuclear bomb as part of the Manhattan Project then john von neumann decided he liked it and programmed a machine to do those same calculations they decided to name the codebase Monte Carlo as every super-secret project should be named Coulomb's uncle happened to be losing lots of money in the monte carlo casino in Monaco so that's why uncle's in fact deep mines alphago used what's called a Monte Carlo tree search to help it play against the reigning go champion resulting in move thirty seven more on that at the end of the course Monte Carlo techniques have several advantages over dynamic programming first they allow for learning optimal behavior directly from interaction with the environment without needing the transition or reward function defined beforehand second it's easy and computationally efficient to focus MC methods on a small subset of the total states and third MC can be used with simulations so let's say we have a home that consists of a bunch of Internet of Things devices we've got a smart TV a smart fridge a smart dog and a smart giant server in our room all of this equipment requires a lot of electricity to run but it also requires cooling or else my room would get too hot so I have a cooling system now let's say that we have access to our electricity usage logs thanks to partnering with a data friendly provider and our smart thermostat can set a temperature accordingly depending on the type of system we build we can imagine electricity flowing into all of these devices creating a closed loop system and this constant stream of electricity data it can definitely be utilized it's giving us the electricity price cooling demand and electricity consumption as variables using this we can construct our Markov decision process the goal of our system is to minimize our electricity bills more money for GPUs lamelle our agent will perform an action in this environment that action will be to either increase or decrease the temperature by one degree Celsius my fellow Americans most of this audience uses the metric system the state then that our agent can be in will be a measure of both how much cooling demand there is as well as the price of electricity the reward can tell us whether we are saving money or not by switching states by calculating the total electricity consumption multiplied by the price of electricity and depending on if that's greater or less than what exists at the time step before we know whether or not that is a positive or negative reward there exists an optimal policy here such that if we were to give it a state in this case that would be the cooling demand and the electricity price it would know exactly what temperature the thermostat should be set at such that we are optimally saving money on electricity by cooling our room as much as necessary when necessary our adaptive real time reward based system needs to learn this optimal policy and since we don't know the reward function or the transition function beforehand we have to compute our rewards and transitions as they happen in real time we'll want to use a model free technique like Monte Carlo to learn the optimal policy the basic idea is to calculate the value function of each state backwards with the reward received after the end of the episode there has to be an ended then the task has to be considered an episodic task for us to use Monte Carlo in our case we can say that an episode lasts a full eight hours while I'm away at my office working diligently if we move from the initial state to the terminal state according to the given policy will receive a reward at each time step we'll remember all of those rewards and when we get to the terminal state will loop back and calculate the value function of each state in the case that there are multiple episodes then Monte Carlo just averages all of the returns we know that the return is the sum of the discounted reward in the context of Monte Carlo though we switch it up to obtain the state value function we take instead the expectation of the returns not the sum we can define a state s to be a discrete random variable which can assume all the stats with a certain probability every time our agent reaches a state it's like we are picking a value for the random variable s for each state of each episode we can calculate the return and store it in a list repeating this process a lot is guaranteed to converge on the true state value function in Monte Carlo RL we are estimating the value function for each state based on the return of each episode and the more episodes we take into account the more accurate or estimation will be notice though that a possible problem could occur what if we visit the same state twice in a single episode well there are actually two types of Monte Carlo policy evaluation first visit and every visit will focus on first visit in this video first visit only recognizes the first visited state every second visit does not count the return for that state visit and the return is calculated separately for each visit Monte Carlo includes randomness because when it updates every episode depending on where it originated from it's a different result depending on which action we take in the same state because it contains these random elements Monte Carlo has a high variance when we graph a simulation of our agent solving a problem we'll see that eventually the policy will converge and then our system will know exactly what temperature to set our room based on the electricity related variables there are different kinds of Monte Carlo techniques that can do all sorts of cool things but we'll talk about those later on three points to remember from this video in model free reinforcement learning as opposed to model based we don't know the reward function and the transition function beforehand we have to learn them through experience a model free learning technique called Monte Carlo uses repeated random sampling to obtain numerical results and in first visit Monte Carlo the state value function is defined as the average of the returns following the agent's first visit to s and set of episodes please subscribe for more programming videos and for now I've got to try a new sample so thanks for watching