Self-Learning Reverse Engineering in 2022

Key Takeaways

how would i learn reverse engineering in 2022 to learn anything you need to practice and spending time on something always trumps looking for the best tutorial or tool just get started but there's one thing i believe that really really helps especially when you get started so let me show you [Music] back in 2016 i worked on my binary exploitation playlist and of course before we can go into buffer overflows and other memory corruption exploits i had to provide a quick intro to programming and assembly we somehow need to understand how computers work in order to attack them and especially episode hex10 reverse engineering c programs i show a technique that i find invaluable and that is simply comparing the source code to the assembly code that is generated this video was mostly about comparing 32-bit and 64-bit but more generally this is about the process how you can figure stuff out for yourself you know they're kind of like two ways how to do tutorials one kind just tells you the facts teacher center teaching but the much more effective technique is one where the students themselves research and discover a topic so here for example i wrote a basic c program and then compiled it in 32-bit and 64-bit and then we looked at the assembly code and we can simply look at how the c-code translates to the different assembly code but not only do we learn about how c code looks in assembly we automatically also learn how to do reverse engineering for example here i'm creating an integer a with the value hex1234 and we can see that in the assembly code as well so we have here a move instruction that moves the recognizable value hex one two three four into a memory location relative to the stack in 64 bit at a negative offset from the base pointer and in 32bit at a positive offset from the stack pointer so by simply playing around like this you can learn reverse engineering try to remember that a line like this probably means in the c code you created a variable and this process i think is very powerful so now let me show you two amazing tools that make this process even more playful and effective let's start with godbolt.org the compiler explorer so on the left side you can put for example c code and on the right you can see the assembly code generated by various compilers so let me paste in one of the example codes i used in my binary exploitation playlist a very basic license key check see how amazing this is so let's make sure we have c code selected and we can see we use the gcc compiler version 12 to compile it hovering over the lines also shows you which c line is responsible for what assembly code hovering over the function name and parameters we can see that all of this is just junk code it's just function setup code so if you want to reverse engineer the actual algorithm you know code like this maybe not that important the first actual line of code is the comparison if rxc is 2 and here's how that is implemented in assembly the word pointer might be confusing ignore dword and pointer but these brackets are important this means it's referencing a memory area and the value inside is the address imagine it kind of like the memory of a computer being a huge array then to access a value in the array we have this index and the index is calculated from rbp the base pointer register this contains an absolute address but it takes an offset from that minus 36. so it compares that memory value with the value 2. and then we have a jump not equal so either the code continues here or we jump to l2 which is all the way at the bottom if we pass this check we go into the if case and we have a printf and now we can also see here how the function call to printf is prepared specifically how are the parameters passed to printf and at the end we can see a call to printf so the cpu will now jump to the printer function implementation but the arguments are prepared before it one argument is for example the print format string itself checking license and it's here moved into edi so now we know on 64-bit if we call a function the first parameter is probably going to be loaded into edi or rdi so next time you see a call look at the assembly before and if you see a move into rdi it's probably the first parameter of the function anyway you can spend hours on here just looking at how the different c code is looking at assembly and it's really really valuable but it can do even more we can add another compiler here for example an arm compiler this was just intel assembly but nowadays more and more devices run on arm for example mobile phones or the new macbooks so maybe it's time to learn some arm assembly and if you are already familiar with intel assembly you can use this to learn how to read arm and you don't even need any arm specific tutorial just look at the comparison this space is a bit limited on a small screen like this so make sure to use this on a bigger monitor but look at for example the comparison if rxe equal to so apparently arm cannot do a direct comparison of the memory area instead we load rxc from the memory into the r3 register and then we compare r3 to the value 2. and then we have a branch not equal which is equivalent to the jump not equal another tip is to look at not only the assembly but the actual binary being created so here in the compiler settings you can enable compile to binary and now you can see the actual binary code that corresponds to each instruction and now all the jumps are not based on these assembly labels but have actual function names and offsets and addresses and also the printf call we can see now the actual address of the printf format string being moved into the edi register but we can do so much more let's look at the impact of the o3 compiler option this enables aggressive optimization in gcc and look at the result the complete function setup part is now missing the compiler realized it doesn't really need it also remember how we learned earlier that the first parameter to a function is passed in edi or rdi well main is also a function so the value of rxc is of course in rdi and in the non-optimized assembly we can see that at the start of the function edi is moved into this local variable location on the stack and then that location is used to compare against the value 2. but of course this can be optimized we can just directly compare edi to the value 2. it's amazing how much you can learn just from playing around with this but the real reason why i want to make this video is because of another tool that was just released by vector35 and hex-rays they are the developers of binary ninja and ida pro to get the tool you have to reverse engineer got bald it's a bit tricky but what you have to do is take these three letters g o d and then reverse them to d o g there we go dog bolt the decompiler explorer it's kind of similar in the sense that you can compare different decompilers and here you really need a bigger monitor but we can select here a ctf challenge example from the samples and look at the different output binary ninja gita and x-rays produce and a few other tools so let's go to the main and have a look all three tools decompile the same ctf challenge binary sometimes these decompile outputs are a bit hard to read but again you can learn here how to read this code and if a line is confusing to you you can also check what other tools did for example as a beginner this call to stack check fail might be confusing what does that do but it's missing an ida's output this kind of makes sense because this function is actually part of an exploit mitigation stack canneries so to reverse engineer this function it's not really useful to see and ida nicely hides it on the other hand maybe you want to see it it all depends on the use case but i'm sure there's a setting in full ida pro to make this show up as well anyway we can see that the code calls fgets so the user can enter a string and then it is in this variable and then we call encrypt and if the output of encrypt is true so not zero then we call print flag and here is the encryption function ida again produces a pretty beautiful minimal decompile output and binary ninja kinda looks scary but you can also just clean it up by hand and remove these type cards and then it will be pretty similar anyway i'm too lazy to solve this challenge now so i leave it as an exercise to you but i hope you agree both these tools are really amazing especially when you are just starting out it's also awesome for basic ctf challenges big shout out to those companies creating such a free tool this video is not sponsored by the way i just think this is extremely useful i hope this helps you learning reverse engineering in 2022 [Music] you