Multiprocessing in Python - Advanced Python 17 - Programming Tutorial

Key Takeaways

hi everybody welcome to a new python tutorial in the last two tutorials we learned the difference between threats and processes and how we can use the pert and threading module in this video we will go into more detail about the multi-processing module so we will quickly recap how we can create and start multiple processes then we will learn how we can share data between processes and we will recap how to use locks to prevent race conditions and how to use queues and at the end we will learn how to use a process pool to easily manage multiple processes so let's start and let's quickly recap from the last video how to create and start processes so we say from multi-processing import process and then down here we define a number of processes so a typical good choice for this is the number of processes on your machine and you get this with OS dot CP you count and then you create your different processes with proces equals process and this takes a target is a function argument and it the target is a callable function this is that is then executed by this process so we define a function up here this simply square some numbers and then we give this to our process here and then for each process we call process dot start and also process dot join so this says says that we want to wait for all processes to finish and block the main program until all these processes are done so this is all we need to setup multiprocessing and now let's go into more detail and first let's talk about how we can share data between processes so in the last video with multi-threading we learned that we can easily share data between threads with a global variable and now with processes processes don't live in the same memory space so they don't have access to the same public data and because of that they need special share memory objects to share data and there are two shared memory objects that we can use we can use a value for a single value or we can use an array so we say from multiprocessing import value and import array and down here let's first start with a single shared value so we say shared number and this is now a value and this takes two arguments first we have to give it the data type as a string so we give it an eye for integer and a starting value this is now 0 and first of all print this so we say number at beginning is and then we say we access the shared number with or this value with shared number dot value so now if we run this and we see that this is 0 and now let's create two processes that should modify this number so we say process 1 equals process and as a target it gets a function that we call at 100 so let's define this up here so let's define at 100 and this gets a number and then it should modify this number a couple of times so we say for I in range 100 so 100 times it should say number dot value plus equals 1 so it should increase this by 1 and we also want to modify a behavior that takes some time so we say time dot sleep and 0.01 and so here we give it this at 102 hour process and it also needs arguments so we say arts equals and this is a tuple so here we give it the shared number and be careful since this is a tuple with one element it also needs a comma here so that Python knows that this is a tuple and then we create a second process that will do the same thing so proces to that should add 100 to our shared variable and then we say process 1 dot start and process 2 dot start and then we wait for them to complete so process 1 dot join and process to the join and now at the end we again print our number so we say number at and is and then access it with shared number dot value so let's execute this and see what happens so the beginning is 100 it's zero and now we got lucky now let's run this a second time and now it's not 200 so it's only 168 and why is that because here a race condition happened and I will not explain this in details so please have a look at the previous video there I explained in detail how race conditions occur so a race condition occurs when two threats or processes try to access and modify the same shared variable at the same time so in this case both processes try to read and write them into this object at the same time so some operations might get lost here so in this case it's only 168 and to prevent this we must use a lock so we say from multi-processing import lock and I also talked about this in the last video so please check this out so a lock prevents another process from accessing this at the same time so to use this we create a lock object so we say lock equals lock and then we give this to our function to to our function yes so this now also takes a lock and a lock has two important methods so first we say lock dot acquire and then at the end we say lock don't release so as soon as we say lock dot acquire it will set this in a locked state so this means that while this is running no other processes access to this code and can execute this part here and then when we unlock the state again with lock that released and the second process can also execute this so this is all we need to prevent multiple process to modify this at the same time so now let's run this and now we get 200 let's run this again and yeah still working and the better way to use locks is to use locks as a context manager so whenever you say lock acquire you always have to call locked release then at the end otherwise this will lock and your program cannot continue so don't forget this and you can use a lock as a context manager so we say with lock : and then your code so this automatically will acquire and release this for you so yeah this also works and now this is how we can share a single value and now let's share a array so we say shared array equals arrey and this also needs a datatype so in this case let's give it a D for double and here give it a list as initial values so we save put in zero point zero here 100 point zero and 200 point zero and then we say our array at the beginning is and let's say shared array and then we have to access each element with inside brackets and then with the index and we can also use slicing here to access all in the indices so let's just put in a colon here and then again print this at the end so at the end we want to print our array so array at the end is this and now we have to change our function so this now takes multiple elements that this takes numbers and then we have to go over each number here and increase it so first of all let's also change this parameter here so let's say shared array and now in our you know function what we want to do is we want to go over each number and increase it but be careful here so we cannot say for number in numbers and then simply say number plus equals one so now if we run this this will print our array at the beginning and at the end and this is still the same and this is because this loop here will create a local variable called number that is then in so this is nothing to do with our shared value object so in order to do this we have to say four and let's say for I in range and then the range has the length of our array and then we access each element with numbers dot I and say plus equals one and now let's run this and now we see that it got modified but we also had race conditions here so don't forget the lock so we say with lock and then our modification operation and now we increased each element in our array by 200 so this is how we can use the shared value and shared array and we can also use a queue to exchange elements between processes and I also already show this in the last video so a queue can be used for process safe data exchanges and so in the last video we set from queue import queue and there we have to use a slightly drift different queue that is formed from the multi processing module so this has all the same methods and except the the task done and the task and the join method so a queue is a linear data structure that follows the first-in-first-out principle so the first element that you put into your queue that is then also the first element that gets retrieved when you want to get elements so let's make an example to use a queue and exchange data between multiple processes so in this case let's say Q equals Q and then create two processes that should do should access and write to this Q so we we have a process that gets as a target it gets a function that we call Square and as arcs it gets some numbers and our Q and then we create a second process that has a second different function here so we call this make- and it has the same arguments so it will write the same q and now let's define our functions here so we say def Square and this will get some numbers and a Q and then we say so in this case for I in numbers we calculate the square and put it into our Q with Q dot put I times I and then a second function make- and there we also it also takes some numbers and aq and here we also go over our numbers for I in numbers and then we say Q dot put minus one times I and then let's say and let's start our processes so process 1 dot start and process to the start and then process 1 dot join and process to the join and here we don't have to call queued up join because there is no method cue the but what we can do is we say while our queue is not empty so while not Q dot empty and then we want to print each element so print Q dot get so this will return and also remove the first element in our queue and yeah now let's run this and see what happens numbers is not defined I'm sorry I have to create in numbers variable and I will say this is a range object from 1 to 5 so or 5 should be included so I say from 1 to 6 and now this will print each element and we see that both processes have access to this queue and can write and put elements into it and then in our main process we can also access the queue and get the elements back so this is how we can a can use a queue and now as a last thing let's talk about a process pool so a process pool can be used to manage multiple processes so a process pool object controls a pool of worker processes to which shops can be submitted and it can manage the available processes for you and split for example data into smaller chunks which can then be processed in parallel by different processes so let's have an example how this works basically a pool takes care of a lot of things for you so you don't have to consider a lot so we simply say from processing import pool and then down here we create a pool and we say so we say pool equals pour and then it has two or four nets and it has four important methods that you have to know and for the rest I would recommend to have a look at the documentation because there are a lot of more methods but the most important ones are map apply join and close so what we want to do is we want to create multiple process that should access a or execute a function so we call define a function cube and this takes a number and returns the cube so it little will return number times number times number and now here we can simply say or we create some numbers so numbers equals a range object from zero to ten or ten or only to nine and then we say pool dot map and now we map and we have to give it a function so we give it the cube and the numbers and this will return a result that we can then print so print our result but first of all we have to so this will what this will do this will automatically uhm allocate them the maximum number of available processors for you and create different processes so typically this will create as many processes as you have a course on your machine and then it will split this iterable into an equal into equal sized chunks and submit this to this function and this function is now executed in parallel by different processors by different processes so this is all you need to write and then the pool will take care of the rest so this will allocate the pools it will split the data and then run this method in parallel and when it is done it will return the result and we have to call pool dot close and then we can call pool the join so this means that we want to wait for the pool to to process all the calculations and return the results and we have to remember that we should call pool that close before so now if you run this we can print our result and we can see that it has the cube here so yeah this is how we can easily use the the pool to to run different processes with a function and if we simply want to have one function executed by a pool then we can say poor dot apply and then also the cube and then in this case it will only has one number here so we can for example say apply numbers the first element so number zero so this will execute a process with this function with one argument and you so this is the most important things about pools and there are also asynchronous calls to this map and apply functions but I will not cover them here and yeah for now that's all I wanted to show you about the multi processing module I hope you enjoyed this tutorial and now feel comfortable working with multiple threats and processes and see you in the next tutorial where we will learn about function arguments and parameter passing in Python