Synchronize Multiple Applications with File Locks in Python

Key Takeaways

what is going on guys welcome back in this video today we're going to learn how to synchronize the file access of multiple different applications using file locks in Python so let us get right into it [Music] alright so whenever we have multiple different things accessing the same resource at the same time we usually need to do some synchronization especially when writing access is happening so when the content of the resource is being changed because we cannot just have multiple threads multiple processes or multiple different applications access the same resource at the same time without coordination without synchronization because this will lead to inconsistencies in the data this will lead to changes being ignored some changes may be being duplicated it's just a very messy thing and you need to have some coordination you need to decide who's allowed to do what at a given point in time and doing this with threats in the same application is quite easy and the same process is quite easy and I have made a video about this already I'm going to cover it here briefly again but today's video is more about file lock so about accessing the same resource from different applications and how to manage that and in terms of code that's actually almost the same so it doesn't really differ too much but you will need to have uh you will need to use a different package for this a different approach for this because of course different applications don't share the same memory space so they can't be coordinated as easily locally so let us get started with a simple example I'm going to create a text file here I'm going to call it counter Dot txt and I'm gonna just enter a0 in here so that's going to be a txt file that has the value 0 as a string in it and what we're going to do here in our python script is we're going to say import time import threading and we're going to just uh Define the following function increase now my pycharm is lagging increase counter we're going to pass an amount which is going to be an integer got to do the proper type hinting here and we're going to say for placeholder in range amount and we're just going to increase this um with a pause in between so we're going to if we have an amount 10 we're going to increase it one by one with a pause of one second in between so that we have we can see how multiple threads work uh simultaneously concurrently so we're going to say here with open counter txt in reading mode come on in reading mode SF what we're going to do is we're going to read the content from that file we're going to save it as current we're going to type cast it into an integer uh and then we're going to increase current by one and we're going to open up the file again counter txt in writing mode SF and we're going to write current to that file and of course we want to write string current to that file so not the number but the string value of that and with every iteration we're going to sleep for one second just so we can artificially slow this down so that it can happen somewhat simultaneously and what we're going to do now is we're going to run 15 threads at the same time executing this function accessing this file so we're going to say here uh four placeholder in range 15 for 15 threats and I'm going to say threading dot thread the target function is going to be the increase counter function and we're going to pass the arguments here 10. as a tuple so we add a comma I'm going to add a blank line in between here I'm going to add a blank line here uh and then we're going to run this and you're going to see um we're going to see nothing because of course we need to also start these threads and you can see we have some errors here because uh things just get messy and somehow uh you know it is increased but it's not increased if you if you let this play through it's not increased 150 times some of the threats fail because they cannot access this file properly and um what we can do here now is we can use a lock from threading to synchronize this in this application so we can say lock equals threading Dot Lock and of course you can also use lock acquire and lock release manually but it's easier to just say with lock to basically acquire and release in this block here so every time we want to read and write read from the file and write into the file we acquire a log afterwards we release it and this is done by every threat and if we do it like that let me just reset this to zero here if we do it like that uh you don't see any output here in the command line but it does happen here when I click on that you can see that this actually increases the counter and I can also remove the sleep here reset this to zero restart this and then it's going to be done uh instantaneously 150 as you can see so this is not what we're talking about in this video this is just the introduction example here this is how you do it locally with threats now let's say different scenario I have multiple different applications I'm going to call this now app.py even though it's not a flask application and what we're going to do now is we're going to have just this one app it's not going to have multiple processes or multiple threads it's going to be its own application and what this app is going to do is it's just going to say four placeholder in range 100 it's going to open the file actually we can copy this from here it's going to open the file increase by one and right into the file so read from the file increased by one and right into the file back so that is the counter let's reset it to zero let's run this application it's done and you can see now we have a hundred here so this is what this one application does now this application could be a hundred different applications to do something similar right so I'm going to use this one python script here but you can also create five other Python scripts that you can run at the same time one might increase by one one might decrease by one one might divide one might clear the content whatever you have multiple different applications that don't know each other that don't really communicate with each other and they want to do the same thing here they wanna uh increase the counter or decrease the counter they want to access this file and change its content now to simulate this I'm going to use this one script I'm not going to use multiple scripts I'm going to use this one script and I'm gonna just run it 10 times uh simultaneously how am I going to do that I'm going to do this with my Linux skills here I'm going to move to the directory that I'm currently at which is this one and I'm gonna run a simple bash command let me just zoom into it I'm going to run 4 I in I'm going to use here from 1 to 10. oh I'm gonna say do python three app Dot py and done so you can see we get the same issue here we get 10 processes started the same application is around 10 times and some of them already have an error message some of them are increasing the counter but you can see it stopped at 199 even though we would expect 1000 if we run this 10 times this is because they're not synchronized of course they're all trying to access uh this at the same time that's not good so what we can do here is we can acquire a file lock we can use a file an external file as a lock and for this you're going to need a python package called file lock so pip3 install file lock and this package is going to allow you to import file log and then all you have to do is you have to say log equals file log dot file log and of course all these applications even though they're different and they don't know each other have to use the same file log so they have to use the exact same file otherwise it doesn't work and this file in our case it's going to be counter Dot Lock for example and then all we do is we say with lock and then we have our code in here and of course you can also do acquire and release manually so you can say lock dot acquire and log dot release if you don't want to use this context manager here so you can just do it like that um yeah and if you do it like that now it's zero and if I run the same command here you can see that all of this was executed and I have a thousand here it's a value so all these scripts uh were running at the same time and again if you don't believe me just create five different script running at the same time uh with different functionality increasing decreasing dividing multiplying whatever try it out it works with different different applications as long as they all refer to the same file log and this is how you can synchronize multiple different processes but also multiple different applications because for processes for multi-processing you have different locks but if you have multiple different applications that are not uh the same application they're not coming from the same script this is a very nice way to synchronize those things all right so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not a single future video for free other than that thank you much for watching see you next video and bye