How to Scrape Telegram with Python

Key Takeaways

telegram is the social media of cyber crime and a lot of other really weird wacky and like questionable or legal things so for some threat intelligence and just knowing what is out there it might be worth monitoring what happens on telegram so in this video I'd like to show you how we can scrape telegram channels messages and all the things in Python and to help streamline that process for us let's use the python Library ton this is available on GitHub at lunami we/ teleon and it's a pure Python 3 Mt Proto API telegram client library that works for Bots too super duper easy to install and get started honestly it's just a pip install single command but it is phenomenal now I do have to say and they include this disclaimer here look as with any thirdparty library for telegram be careful not to break the terms of service or get your account banned be smart be responsible do it this what you will but I want to show you how it can be done by the way this video is sponsored by layer I'll tell you more about them later alongside their own telegram data collection cyber crime and dark web thread intelligence and info stealer malware logs so I am inside of my Cali Linux virtual machine and I just wanted to be in Linux for my development environment you could use any other distribution you might like or you could be on Windows but it's super duper easy let me work in a directory that I could work with I've just created this telegram scraping directory currently empty but what we could do is PIP install ton now if you have python installed whatever version you might be able to get PIP as the package manager up and ready we could go ahead and install that now before we start writing code you might know that I am a big fan of pointing folks towards the actual documentation hey read the information from the source so you can take a look at their read the docs resource for more in-depth explanation with examples troubleshooting issues and more useful information from the documentation itself that is online at doc.on dodev and from there we have an awesome reference for all the things that we might want to do and how we could do it obviously hey starting from the start if you're new here you can jump straight to the installation which we have just cruised through using pip getting that installed if you want to get any development versions to play with it you can certainly do that or add any other dependencies that you might need then we really need to get into action with telegram before working with telegram's API you do need your own API ID and your own hash this you need to get from telegram itself so you would log into your telegram account with whatever phone number of the developer account you want to use but that is online at my. telegram.org so when I hop over to my. telegram.org you will need to log in with the phone number that you have associated with telegram that will send you a confirmation code via Telegram and not SMS or text message but at least that logs you in securely so I'll enter my phone number here and after I've entered that confirmation code we'll navigate to the API development tools if you haven't done this before it will prompt you with a create a new application window and that is where you could fill in your own application details for your app your scraper your tool how you'll use Python to actually work with telegram you can enter any URL or you don't really need to in the first two Fields just giving it a name and a short name but those can of course be configured and that will generate an API ID and API hash now those need to be secret they should not be shown they should not be shared they should not be given to anyone cuz anyone else might then be able to use your telegram account for whatever they might want to do that's absolutely confidential with that said you'll eventually find yourself at this page the app configuration that will display and include your app API ID and your app API hash and whatever name or short name you gave it obviously I have these hidden so they aren't displayed to you and we could set those up as something to work within our code or ideally have something to work with as environment variables that might be used in whatever application setting you're using this in the teleon documentation includes this hardcoded in the script or python code that they might run but I think it's better for us to do that as an environment variable so now let's get started writing our first script to play with telegram let me go ahead and call this just scraping dopy opening that up in Sublime Text my text editor of choice you of course can use whatever you might like VSS code Nano Vim emac whatever the heck but let's work with the ton package and from that Library let's import the telegram client alongside that we know we'll be working with some environment variables so let me import OS so I could easily grab that and with that we could set some variables for our API ID and our API hash now we will of course set these up as environment variables but we'll use that os. getv function and method to be able to get that value let's call this telegram API ID and we can copy and paste that just tweaking it slightly for the API hash but just for a sweet proof of concept let's try to Stage our client given the telegram client from ton passing in a session name Telethon uses some cool management of sessions that you might use as you interact back and forth with the library and with telegram so you could call that a non or Anonymous or whatever honestly it doesn't matter it could just be session name or anything but then of course we'll pass in our API ID and our API hash as we have staged that above and now I would like to build out an asynchronous function so let's use async def to Define Main and with that that can be all the operations that we might do with our tooling that we build out so just as a super simple example a little proof of concept demo to make sure this is all working let's actually try to work with our client and it is a synchronous so bear in mind you'll need to use await as a prefix here we could try to send ourself a message client. send message as a function or method we'll want to call we can pass in a string me just for myself in single quotes or double quotes noting that as a string and then the message that we want to send let's do hello from Python and now outside of our main function we do need to actually call it but remember it's all a synchronous and it's working with our client so let's use a with context manager there and then let's use our client loop. run until complete as it will handle a lot of the asynchronous operations that we might want to use and still somewhat of a procedural method until it's done so inside of that we'll include the call to our main function that we just defined up here now fingers crossed hopefully I didn't get anything wrong we could try to see if this will send us a message through our python script in the ton library but before we get this all going to se an action remember we need to actually add those little environment variables for our telegram API ID and our API hash so I'll paste those in and I'll hide it cuz you shouldn't see that and now that I have that done I have telegram open on the left side of my screen with the chat with myself just for the simple proof of concept can our script properly send a message to ourself fingers crossed ooh first time you end up running with this thing you may very well need to enter your phone or bot token I believe ton does end up using that to help stage and prepare your session so again you're using this for your own use you can have whatever I don't know threat model risk assessment your own obsc you'd like in the mix here but for the sake of showcase let's get the phone number in the mix that will prompt you for another sensitive login code so just be cognizant of that but now we should be able to try and run our script and see if it sends us a message let me hit enter on our pythons scraping dopy and there it is Hello from python now obviously that was just a super tiny and small example we'll dig into a heck of a lot more but again I do want to use the the documentation as kind of the best reference for that it gives you all the functionality if you wanted to try and sign in as a bot account even using a proxy in the back whatever you'd like the next section their quick start gives you a little bit more insight on how you can get some information about yourself or your own account pull out some properties work with other things that you might have conversations and dialogues messages that you might be a part of or just sending messages to all anyone and of course using markdown and all the messages that you send if you end up using it to send messages but remember our purpose purposes are actually using this to scrape to listen in to Monitor and to see what else might be going on in other telegram channels you can actually add some other triggers or like updates different events that might happen if you see a certain kind of message or it has a specific keyword you could build this all out with different event handlers and build out some really slick stuff in your python code but again gentle reminder be sure that you're working all within the realm of the terms of service and just kind of being smart with whatever you end up doing with this scripted automated process and doing it with a bot or using python in the mix but now let's build out the functionality to actually join a telegram channel so we could see some messages and then be able to pull out the contents try to think of like I don't know a cool use case where we almost have a telegram crawler that'll recursively find messages and one after the other join a new channel pull messages join another Channel pull messages lot of cool Runway there but we need a couple other new Imports honestly just one we do want from ton and they have a little subm module here.t L and then they have functions and channel is what we're going to be looking for we want to import join Channel request and actually that should be channels s plural forgive me there but now let's build out another asynchronous function that we could just call async def join Channel this will just be a convenience function for us so we would end up passing in our client that we're working with and then the channel link like you need the username or the invite link to actually gain access to that channel as you would naturally Telegram and you may or may not actually get into the channel so in which case we could just have a simple small little try and accept Handler here but inside the tri block let's have our client remember we'll use await because it is a synchronous use a join Channel request just that type that we just imported from that new little subm module import and then we'll include the channel link that we pass into it if this were to succeed we would obviously continue on in our code inside of the tri block so we could just simply display maybe an F string to say look we successfully joined the channel and then our Channel link included there if it didn't work if our try and accept statement were to fall through and again I know this is Scrappy code we'd probably be better looking at the actual exception so we don't miss any accidental exceptions like syntax errors whatever mistakes we might make we could just say failed to join Channel given our Channel link but that's all that really takes so rather than sending us a message as just a simple tiny proof of concept let's try to await joining a channel given our client but then we need an actual Channel link and some place for us to go and a channel to go take a look at now flare is the sponsor of this video but I know people get a little hyper sensitive with ads so don't worry I'm not going to be rambling for like 10 minutes again I just want to get in and get out and grab some of those sweet telegram links cuz flare makes this pretty awesome remember You' got that Google search basically looking for all events across the dark web elicit networks maybe if you want to drill down into chats to see any specific telegram forums or just search for them look we could be tracking maybe red team tools or automated pentest or hacking Frameworks out and about for sale maybe you're looking for some known thread actors that are associated with ransomware denial of service whatever operations and of course it's pulling all the sweet pictures out too holy league is a good example of that and uh cyber vulk just as well maybe some other operations that do defacement and other whatever nefarious cyber crime or you're interested in exploits new cves vulnerabilities that are being taken advantage of out in the wild all sweet stuff that you could help track down and at least some behind the scenes this idea and concept of scraping and looking for messages across telegram is what you could build out just as well so for our own telegram scraper in Python let's go see what other weird wacky channels we might be able to join and then lurk in and piler some of the messages out obviously they include the room link or the invite URL in each of these you could see stuff like oh Nova Market Anonymous Services weird weird stuff in languages that I can't read insta Market the Big Apple Continental Service New Market money market Hunters exploit hacker group oh we got to find ponors with the Z I know they were pretty happy when I uh wanted to Showcase them for a bit Market shell oh I like that one scammers hell chat group okay let's go see what's going on in scammers hell chat group we can grab the room link and then let's use that in our code by the way flare had made some really awesome open source tool to be able to do just about everything that we're doing in an even better faster smarter way so I'll include that in the end of the video but if you'd like to see a big shout out and big thanks to them for all their support and helping things out here let's go ahead and add in our Channel link that I've just staged as a variable and now let's go see if we'll smartly join our scamers hell group chat let me get telegram off to the side let's get back to our terminal and let's try to run our scraping script one more time and there we go successfully joined scammers hell chat group oh thanks so much oh what is going on here okay thank you thank you for all of this and whatever we find ourselves uh digging up now look take this with a grain of salt obviously I'm sure this is all kind of trash garbage maybe some of it's legit I don't know I'm not going to validate but maybe we've got the cool capability to do that in mass now with stuff we could build out for our telegram scraper but obviously we haven't done any of the good stuff yet we are just joining a channel we now need to grabb some of the messages so this is not hard ton makes it super easy and obviously we've kind of already built out some of the structure some of the process some of the methodology when we might want to build new capabilities to scrape messages just another asynchronous defined function where we will include our client as usual the channel that we're interested in and then maybe a number of hey maybe a limit as to how many messages we'll scrape we can make that configurable but it's handy in case you'd like it with a default argument so all we really have to do is Loop through the messages and then pull them back let's use asynchronous here so async 4 each message in the client capability when we iterate through messages given a channel based off of a limit that we've provided then we can validate look if that even has text if message. text exists if there's a real property and it's not just an empty string what we could do is print out the message now obviously You' make this smarter I don't know you could have some class objectoriented programming make this a whole Factory machine as you'd like but for our simple showcase and example I think this works well so in our main function as remember this is all the capability that we actually want to do using the functionality we've built out let's try to now await scrape messages given our client and the channel link that we're working with and the limit that we could just say is one in this case let's just get the first latest message that I can track down now let's get back to our terminal let's try and run this thing I'll use our python scraping dopy script it'll join the channel and oh I broke oh it should be each message or honestly we can just change that to like message in our iterator that doesn't really matter uh anyway ha I'm human I make mistakes let's do it again and okay we got a lot of stuff here because look this is the entire object in Python these are all of the things that it will include in all the specific data fields properties perer id id date message all the stuff we'd like to clean that a little bit better and there are some convenience things with ton to help display that in a pretty way we actually get back to our code let's use the message. stringify and that will make that beautiful for us so we could probably more easily track down what's going on where there we go now we can see all of the properties in a little bit more of a readable way and we could see everything that might include like the stupid emojis and the text itself if we wanted to print that out along with its ID the peer Channel and the send Etc so let's make this dumb and stupid let's just try to print out the message text for as many as we want and let's go to like 10 uh I will in this Loop and again I know it's messy in its own Loop but let's try to just add I don't know a little divider maybe 40 hyphens at the end or something and that will just turn out as many as we'd like for the messages in that channel but think about how now you could process the message text maybe carve out and find new channel links to join or go lurk in any of those other spots and I know of course we aren't really dealing with with opsc or the operational security here of you just joining channels uh this is for showcased it is for Education look uh your things you might very well be able to do with a bot very differently but this is doing okay let's now try to run this again and see what stuff that it'll churn out here uh everything that we just saw within telegram but now in a pythonic way for us to be able to do a little bit more processing so I hope at its core I know this is baby Basics right we've got 30 lines of code but that is a jumping off point if you'd like to play with this and explore a little bit more there are a lot of other sweet features of ton that you could play with but to Showcase just a little bit more sweet Easter eggs I would love to show you that cool tea hunt or tea hunting tool that flare put together for us this is public and you can find it online on their GitHub github.com flaired and then look if you want to take a look at some of their API documentations or sweet bindings with python or go or whatever to use with their API what we'll dig into though is tea hunting and it is public you can go ahead and pull it down you can play with it but let me show you all the cool sweet stuff that it can do and this is a great example of making more of a fleshed Out tool and capability rather than just the I don't know 30 lines of code that I spit together in our showcase here look they've got some sweet color look they've got some cool capability with ARG parse in the mix and even using some of the natural language toolkit to try to determine the sentiment and try to figure out hey is this going to actually mean one thing good bad or ugly with a lot of the conversations that are going on here it does have that channel manager as a class that I mentioned some of the sweet objectoriented programming so Stuart big shout out to him I believe he built this one phenomenal researcher and developer over there got all the capability to join channels look for messages hey have a little bit better uh exception error handling I don't mean to scroll through this cuz I know that's hard to do just off the cuff kind of overwhelming when I'm just not looking at the code and making the text go by but let's do it let's try the thing out I'll get back to my Cali instance let me open up another terminal let me get into that telegram scraping directory and let's go ahead and get cloned down our tah hunting tool and let's see it an action looks like they actually include uh the requirements. text probably worthwhile to install pip tacr requirements. text and I am curious what this config.js file is that they have in that directory maybe something a little bit more hey capability o yeah to be able to look for different channels maybe you supply it'll automatically join and work through and then try to see any specific message keywords that you're looking for any messages that contain hack carding malware expl tracking cool stuff that you could Define and if you wanted to dig into that code you could see how it would work with it and then use that as a component to determine where to Branch off to and to crawl next that's really all this is right some sweet crawler to see telegram channels in action we can try and run this thing let me use tah hunting and I do want to use Tac H to see the usage capability they've got a sweet little banner up here oh yeah I dig that telegram content crawler okay so we could provide the config file if it were in a different path Maybe configure or toggle the message depth or the channel depth how far we want to crawl and dig through and of course we'd include our API ID and hash and maybe the phone number for it to be able to work with it so let's get back to our config.js and then kind of set up these initial Channel links or where we would like to start looking around and seeing what weird stuff we could crawl through I do want to hop back to flare just to grab some of these URLs we could use hly league as one of the examples for the thread actor group seeing what they're up to let's do the same for cyber vulk let's grab them and let's get one for exploits digging into those cves or threats out on the horizon exploited in the wild I've just been pasting these into our config.js and now that that is all staged we could fire it up and use our tah hunting script remember our usage information here we could Supply a different path to the config but I believe it'll use the default Json in the current directory first uh message depth we could toggle Channel depth will set those I have touched up the codes that will work with the API ID and API hash from our environment variables but now let's try to run our python tah hunting with tac tac message depth let's start small let's just do like 20 for the start and then Channel depth we could say maybe two yeah I don't want to go super duper far and we don't need to specify the config so fingers crossed this thing will uh keep us cruising okay final up we got the cool logo yeah we'll need to authenticate okay I started the thing off and now it is cruising now remember this is all based off of our user I'm not doing any of the obsc worry concerns here I just totally cool with telegram account that I've got jumping in joining those channels and seeing what is up parsing out some users though seeing all the messages back and forth getting the Emojis out all right a lot of this I can't read whatever cyber crime in the languages that I don't know pass that to chat GPT that's where we use our sentiment analysis right and uh the natural language tool kit and hook it to some AI models right that would be really cool to help translate but we've got all that power now thanks to being able to do this within python how many channels have I joined okay scraping completed uh took about 50 seconds just to Showcase scraped about 112 messages process 17 channels oh goodness outputed these all and sweet CSV files for me to examine sentiment report and then a couple high alert stuff potential threats potentially positive ooh we could dig into all that we do have the saved output but I might leave that as an exercise for the reader if you would like to go try this yourself how bad does my uh telegram look now if I bring this in what else do we got I know we started in our all scammers hell chat group right with more messages that were coming through as we got started uh but what's happen all right yeah we've joined overflow case all that all right I am a little curious what was going on in those reports though I do want to open up hey this current directory see what was in our sentiment report uh analyzing 112 messages and then you know neutral situation no significant threat granted it was crawling through kind of very vastly different things um concerning messages of fools trying to escape me Mato from AP Zone all right recent rch cyber attacks cyber power um yeah there's not a whole lot I could just say you know surface level on that but it's pulling out the session of course and the messages that would seen from the CSV file this is stuff you could get into some Google docs Google Sheets Excel whatever you want to do to analyze it what the hell was that all right this is why I say you know you got to take it with your grain of salt um it is cyber crime it is Shenanigans it is weird wild wacky illegal elicit stuff out and about on the interwebs but look I do want to show you all it's out there and how you if you're interested could totally take a look and use some of the tools maybe build out your own Python scripts maybe use tellah hunting and the sweet open source option with that but big thanks so much for you tuning in to watch this video for you playing along if you were willing to try and create craft out your own little script in Python to be able to scrape with this on all thanks to ton hey making this easy for us so thank you so much for watching this video please do all those YouTube algorithm things like comment subscribe and seriously please do give some love to flare I know I sing their praises all the time but that is just genuinely because of how cool they are and all the incredible things you can do with it and big thanks to them for helping streamline and put together some of the stuff to Showcase in this video seriously I'm just grateful for all their support Link in the video description jh. life/ flare and I will see you in the next video