Prepare for the AWS SysOps Administrator Associate (SOA-C02) – Full Course to PASS the Exam

Key Takeaways

hey this is Andrew Brown your favorite Cloud instructor bringing you another free Cloud certification and this time it's the adabs ssops administrator associate certification also known as the S SOA C02 and the way we're going to obtain certification as always is going through the lecture content uh doing the work for real in our own inabus account utilizing the Hands-On Labs instructions and as always we provide you a free practice exam so that you can go aset exam get that certification put it on your LinkedIn a resume to better get yourself a job in cloud or in a Dev hops rooll I just want to remind folks here that uh this content is made available free here on free Camp though if you want to support more free Cloud certifications just like this one I strongly recommend that you purchase the additional study materials which comes with things like additional practice exams layered content technical support and more um you know me probably by this Point I've taught a lot of different types of uh courses around Cloud here adus Azure gcp terraform kubernetes Linux and more um so you know you're in good hands but let's just jump into the course uh and get to it ciao hey this is Andrew Brown and we're at the start of our journey asking the most important question first which is what is the sis Ops admin associate so the ssops administrator systems operation administrator is an administrator certification when I say administrator I mean like I think it administrator uh Cloud administrator um and it's going to teach you things like automation of cloud infrastructure via scripts troubleshooting Cloud networking issues uh performing automated or or Endor no touch maintenance on compute uh monitoring and observability of cloud resources and the course code for this certification is the s o s03 I'll get my head out of the way so we can see the full uh code at the time of this video the um solution Architects been s03 for quite a long time why is this not s03 I don't know ad us really should have updated the course code by now um but I just want you to know that if you are taking the SO3 I bet this course isn't that old but as always make sure that the course code matches for the certification you're taking so you have the latest content uh to best pass the certification um this specific uh certification the ssops admin associate has has been considered the most difficult general knowledge associate exam for ads at the associate level um I've always said it's the developer but it really depends on your background I I think a lot of people struggle with um it networking understanding uh how networks work and so that's where this becomes a challenge for a lot of people but if you have a developer background you might find this harder because you don't have the networking background if you have a networking background you might find um the developer harder but uh this one is generally the one that is is least taken but has extremely valuable knowledge so I really strongly recommend that this one's taken alongside the solution architect associate and the developer um but who's the certification for well consider taking this specific certification if you want to work as a junior devops engineer um ssops administrators is not a common job job title I will see it sometimes but generally we call these Junior devops Engineers um so you know just look for those rules online if this is the kind of stuff uh that you want to do um this is a role if you enjoy maintaining existing Cloud infrastructure and providing technical support um think of what a junior devops engineer does a junior devops engineer supports a senior devops engineer or just a devops engineer so um a lot of people want to get a devops because it sounds fun automating all this infrastructure but understand that this is more of a supportive role um to other devops engineers and uh it can be a customer facing role um that would lead into that so just understand that difference there so if you enjoy the following tasks like Building Systems that support applications working with Linux working with Cloud networking or uh learning just enough coding to work with basic scripts well this is the rule for you but let's take a look at the full path of um certifications so even from the solution architect associate which I just released less than a month ago this has changed dramatically uh because adus has done a bit of a shuffle um they used to have more Specialties but what they've done is uh they scrapped like I think like four or five of their Specialties and they've now uh introduced uh the AI practitioner the data engineer and the Machine learning engineer as of this video they're all in beta but I suspect that they are going to come out so I'm going to treat this as if this is what it actually is now because I think that's what's going to happen um but there's a lot of paths that you can take uh when you are thinking about your journey and there's no wrong path these are just suggested paths that I I think that you could take for specific roles so I going get my pen tool out here the idea is that you almost always want to start with the Cloud practitioner never skip the certification even if you don't want to set the exam do not skip the content um in my courses I make a lot of content that I do not bring over into the associates other creators will uh pack that stuff in there but the problem is is that we need to spend as much time at the associate level doing labs and that fundamental content needs to stay in that fundamental certification so if you have yet to do so please go do my cloud petitioner first it's not as long as these courses but we're going to cover stuff uh especially pricing um uh in that one pricing and billing so make sure you take that one first uh there's obviously the new AI practitioner which would lead you to maybe data engineer machine learning but let's focus on the ssops administrator where we are here uh so the idea is that when you take the cloud practitioner a lot of people will go take the solution architect first there is no wrong one to choose first whether it's Sops the developer the soltion architect associate the only correct answer is to take all three of them at the same time because they really have a lot of overlap there's like 30 40% overlap um between each of them and really if you don't take all three of them you're just going to have incomplete knowledge uh so some people just think that oh well I want to become a devops engineer so let's skip the ones that are not necessary but I'm going to tell you the associate level you want all three of those general ones but anyway you can go take the soltion architect then the ssops then the developer or the developer in the sis Ops and from there you can go on and do the devops engineer if your goal is the Ops engineer the sis Ops administrator and the developer are a must absolute must because uh the combination of those two certifications is the devops but expanded in a much more uh deeper um curriculum okay and then if you want to uh specialize or uh uh do additive things you can go over and add the security certification or Advance networking I do want to point out that ADA certifications do not validate programming technical diagram code management and many other technical skills that are required for obtaining technical roles ads assumes that you are grabbing this information elsewhere um so you definitely need more than just these certifications to land your roles okay and I'm going to repeat that so you absolutely know that how long is it going to take to pass the certification very similar to the solution architect associate we're looking at 60 hours on begin or end and 20 hours on the experience end um so you know if you have your Cloud pred conditioner but maybe you haven't done your solution architect associate yet you're looking at 60 hours if you've done the um solution architect associate you can cut that down to 40 to 30 hours um for those who are very experienced already know ads very well and they're just trying to get the certification to prove they already have the knowledge you're looking at 20 hours or less just because there is a lot of um little things you have to learn doesn't mean you can't do the work but you need to prove that you know uh uh the knowledge of it of us fully and so you're going to have to dig into a lot of very specifics of services um so an average study time would be probably 24 hours 50% lectures in Labs 50% practice exams though I should really give more emphasis on Labs because my labs are getting more long longer these days uh recommended study is one to two hours for uh 24 days it really does take a month with each of these associate certifications if you're doing them individually if you do all three generals at the same time you can uh shave off a month so that's why I strongly recommend doing all the three Associates at the same time uh how how are we going to pass the certification while we're going to watch the lecture videos and memorize key information you're going to do Hands-On labs and that's the key difference between uh the S OA c01 course that I made and this one is that I've added so much more Labs that's what makes this course so darn long because I'm adding very very high quality labs for you to follow and I really want you to do them in your own account and I would strongly recommend getting some practice exams we have paid ones we also have a free one make sure you get your free one by signing up on exam amam pr.co uh even if it looks like you you have to pay do not worry just click through you will get to your free prac exam there and just follow through there and we have a bunch there for you okay in terms of the content outline uh it says four domains there's not four domains there's like six or eight so that must be wrong let me get my pen tool out here and we'll adjust the number here in a moment but as always the domains have their own waiting and this determines how many questions of a domain that will show up so we have domain one which is monitor during logging and Remediation domain two reliability and business uh continuity very hard word to say conty I think people know what that means but I can't say it then we have domain three deployment provisioning and automation now it says deployment here but honestly uh the deployment knowledge is more for the developer not so much as the ssop so it's extremely light um they're not talking about application workloads they're not talking about code Pipeline and things like that we have domain four so security and compliance domain five networking content delivery domain six networking network uh or sorry cost and performance optimization these things domain six domain 4 They Don't Really cover that much in the exam which is bizarre but you're supposed to have this knowledge and again that's why you need to take my cloud partitioner because that's where I shove all that knowledge and I'm not going to uh go over it more than once because you're already supposed to have that knowledge uh networking content delivery is a bit unusual because um that is something that is more really for the developer but uh we do cover um some of the uh uh Network like the content delivery stuff networking is very very heavy but content delivery not so much for this certification where do you take this exam well you're going to take this at either an in-person test center or online from the convenience of your own home itus delivers these exams via Pearson View and so they have an online Proctor system so you can do that from home or you can go to one of their many test centers um that they are networked with to do it in person if you have the option do it in person it's so much less stressful um because the environment is very controlled um and uh you know just it is a better experience but if if you can't then take it from home and that's what you're going to have to do they used to utilize PSI online um for whatever reason they've stopped using PSI but uh to be fair PSI hasn't been very good these years I think um uh GitHub GitHub uses PSI and it's just an awful experience so I think that's why itus has moved away from PSI and that's why they're using Pearson um you need to understand that these exams are proctored that means there's a supervisor or person who monitors students during an examination I'm not exactly sure how true that is anymore there's definitely someone that checks you in and stuff like that uh with the rise of AI I would imagine that they're probably leveraging uh some level of AI to automate it um as I found that when I'm taking my exam they've been a little bit more flexible with uh things that are happening not to say that I'm cheating but I just just mean to say it doesn't feel like a person's watching watching you the entire time but act and treat it as if there is because they do capture that information and if they decide that they don't like um uh your test uh your your your test your test taking they can revoke your exam and so sometimes I see on Reddit people people pass to get their exam and then two weeks later a month later they revoke it because they decided somewhere in that footage uh there was something questionable so make sure uh you present yourself as honest as possible uh during your examination okay uh for grading it's the same as the solution architect um and so that means you have to get about 72% to pass we say around because it's totally possible to fail with the 72% so aim to get higher than 72% on your practice exams get 85% get 10% over in a bit so that you have that wiggle room uh to make sure that you do pass for real response types here um is multiple choice and multiple select with 65 questions there are 50 scored 15 unscored so you can afford to at least get 15 scored uh questions wrong actually more than that um if you count the unscored one so you can get quite quite a few wrong there are no penalties for wrong questions so always answer them and again the format is multiple choice multiple answer these uncored questions the reason they have these on your exam uh is mostly because they want to uh introduce new types of questions another reason is that it can um help determine how to make the exam easier or harder because if people are passing with the very easy unscored question then they might consider adding them or adding more difficult questions and I believe that it aids also in detecting if people are cheating because um they might be administrating very specific questions to um specific areas and so it helps them narrow down where dumps are being stolen from I can narrow down who is doing the stealing so uh just understand that's the purpose of them and if you encounter a question you've never seen before don't stress out so remember you have 15 questions you can get wrong um and they may be an unscored question for the duration of this exam you have about two two hours uh two minutes per question so 130 minutes is your exam time your SE time is 160 minutes when we are talking about SE time this is the time you need to prepare uh for the whole exam meaning you're going to uh uh have time to review the instructions show uh online Proctor your workspace uh read and accept the NDA complete the exam provide feedback verify your identity um so yeah always be 30 minutes to an hour prior to your exam and uh factor in uh any problems that you could have okay because it's your money you don't want to uh waste that these exams are valid for 36 months and it's three years before you need to recertify most people don't recertify because once you have that base knowledge you don't really need to get recertified um they might have a free method for recertification but um you know it's up to you and up to up to your employer but if you're a firsttime uh person you obviously want to uh pass passy certification so you can show that you have that base knowledge at some point in time I want to have a bit of real talk with you because I just want to strongly strongly emphasize that if you pass this exam that doesn't make you a devops engineer okay devops engineer is a senior role that is given to those with years of experience and deep knowledge of implementing technical workloads so you could obtain something like a junior devops engineer role um but not again solely with this certification you have to consider there's like about 200 250 to 500 hours of additional work that is outside the scope of cloud that you need to have alongside with Cloud to obtain these uh these um these rules and I just want to point out that adus does not care about ad certifications for hiring so if you're trying to get a job at ads you say look at all these ads certifications I have they don't care they want you to also have those fundamental skills because they know if you do get H adabs they can then internally uh provide access and get you trained on certifications if that is required in your position um certifications serve a structured way of learning with a goalpost that's not to say that um they don't matter to uh in um to employees so some employees do care about them but I just want to make it very very clear that this only teaches you the cloud component and there is so much more that you need to know uh besides that but luckily for you I try to um put as much of that in my courses and that's why my labs are so long because I'm bringing those missing skills that they're just so you don't it's not required to pass but it it's required for you to do the job and you really want to have those skills so Cloud certifications expect you to have these foundational skills programming scripting SQL it networking Linux and windows servers project management developer Tools application development skills compi algorithms and more um to fill these technical gaps leveraging you can leverage the free Camp large catalog for General technical content content to get skill ready and job ready look at the examp Am pro supporter subscription because I am creating uh projects specifically to Cloud that brings all of these skills and I'm making isolate courses that also uh bring these skills specifically for cloud um so look at those two options and that's going to really help fill out those missing hours those those missing Gap skills okay I just want to talk about how we do our um our Hands-On Labs because it's very different from other providers or other content creators in our Hands-On instruction we do our best to try and fill the missing gaps we might spend considerable time before using a service developing these Gap skills Labs can be long because I want to show you everything and and labs are not heavily edited okay uh so some hands handon Labs might end in a failed implementation but are left uh uh left in to experience troubleshooting or giving an accurate reflection of what it is uh what it's like working with that service so some adaba services we have to learn but are not recommended for use and I'm very honest and open about what is a good service and what's a bad service and what's the likelihood that you would actually have to use it for real so if there's a service that I think that is just in the exam that we have to pass but it's not going to benefit you in your career we're not going to go heavy into it and I'm going to make it very clear uh in the materials uh we do uh we try to do our best to clean up costly infrastructure but you should always be proactive and check if resources are left running you are responsible for cost and spending your ad account we cover in the cloud practitioner thoroughly billing cost management things like that we're not doing that heavily in these courses it's assumed at this point that you took my cloud partitioner and you know how to responsibly monitor your spend um and you'll see me sometimes I'll I'll miss some resources they're minor resources but if you can't afford the pennies if there's a bucket or uh some alarms left open and you get charged a dollar or two um just understand that uh I'm not going to uh 100% give you a guarantee that all those things are spun down because it's your responsibility and you should know how to do that okay so always check your spend go into the cost Explorer and check check check check okay but uh there you go what we'll do next is just take a look at the exam guide so we can understand the contents of the exam specifically [Music] okay hey this is Andrew Brown we're going to take a look at the um exam guide for the CIS Ops administrator associate what's interesting here is they're talking about um in March 2023 like last year of the time right now is that they used to have um Hands-On Labs so they' have like one or two real Labs that would spit up en's account to make sure you knew what you were doing they got rid of those and they've gone back to multiple choice multi- select which I think is a mistake I think that they were going in the right direction but maybe it was a cost or uh I don't know for whatever reason they've gone back to just multiple choice and multiple selects so that's what it is if you want to download the exam guide it's right here there's also sample questions which we can take a look at though I would just say take the uh sample questions with a grain of salt because they're not very reflective of the real exam as I've always found um so if you don't have confidence with this don't worry about it take the real take take a real practice exam like my free one and you'll have much more confidence or an idea of what the exam is going to be like but anyway let's go here and I have the exam guide open um and we'll go all the way to the top here so if we scroll on down we can see we have our um domains so we have domain 1 2 3 4 5 six and what's really interesting about this this one is that I read through it all I've taken the exam multiple times um and it's not as reflective as it should be with the actual exam it's not as bad as the suan architect associate which is wildly uh wildly inaccurate to the actual exam but this one is okay but in some areas I would say that it's uh not accurate so you don't have to stress out about some of these things so I'm going to get my face out of the way so we can see exactly what we're talking about here and I got to hide my face not the screen and let's zoom in here so we can see a little bit closer here and let's take a look at domain one so domain one is logging um logging and Remediation and I'm just going to get this tab out of the way here so Implement metrics alarms filters by using a monitoring logging Services identify collect analyze export log so in my cloud partitioner I teach you how to do logs so we're not going to or sorry not logs but I teach you how to do um alarms uh so we're not going to go really deep in alarms it's something that you should already know at this point same thing with cloudwatch dashboards that's something I would do in the cloud petitioner um but we what is very important is learning how to filter metrics working with the cloudwatch agent which I have a very long video on this one we spend most of the time building an app and then configuring this this one's a very high value uh lab that I gave you but we have cloudwatch logs uh Cloud watch insights cloud trail logs and we cover all those in laps then down below here we have configure notifications this is really straightforward there's not much to say about notifications other than when you create a um alarm you can tell to notify something it's very straightforward so there's not much to talk about that one we have remediate issues based on monitoring availability metrics so troubleshoot or take corrective actions based on notification alarms um they don't really need you to cover this in the exam content uh it's very straightforward uh like what it would be to troubleshoot or correct an action um I can't think of an example off the top of my head but I'm just telling you that uh they make it sound like you have to learn more than there actually is invoking an event Bridge rule uh so yeah we definitely need to know how to use event Bridge uh so we cover that one very very thoroughly um and if we don't have a lab in the event Bridge section we definitely cover event bridge in one of our other labs multiple times just because it's a core service that you're going to run into when building out uh serverless pipelines so that's definitely something we're good to touch we have uh adus systems manager automation runbooks I believe that we cover this more than once um so this is something that we will cover a few few times here it's saying take action based on adus config rules if they're talking about Abus config I never saw this whatsoever in the exam uh I it's like doesn't show up whatsoever so I I wouldn't really worry about it was config uh we have Implement scalability and elasticity so create and maintain a auto scaling plan so we have a lab on that very straightforward Implement caching so possibly we're talking about elastic cache and memory DB memory DB is not required but I put it in there because it's a newer service so I imagine at some point they will do that Implement RDS replicas and Amazon aora replicas um you can just learn the lecture content for that I think it's a little bit very involved to uh learn the uh how to set up replicas to be honest and in practicality most people are starting off with RDS and not really working with uh a so I think the theory knowledge here is better than the implementation knowledge Implement Loosely coupled architecture um just using cloud services we're already doing uh Loosely coupled architecture so there's not really much to say there it kind of feels like fluff text that they threw in here uh differentiate between horizontal scaling vertical scaling this is something we cover in the cloud partitioner it's not something and the solution architect associate it's not something that uh we need to explicitly call out in this certification um horizontal scaling is when you add more servers vertical scaling is when you make servers larger that's it so I'm not sure why they have that as a point on here Implement higher availability and resilient environments so configure elb with rough 53 health checks um again I think you just need to know about those I I don't think that it would require you to actually be able to do it we do cover R 53 and we have Labs on elb elb is more so covered in the developer than it is in the umis Ops so uh that's why I always say to study all three of them because it'll fill any gaps and you might studying one area for a different certification but you'll have the full knowledge of of these Services um and so that's again my recommendation is to do all the associates and the developer will give you really good elb skills ASG skills and things like that but here they're talking about differentiate between the use of single a and multi uh multi- deploys um so that's pretty straightforward and lots of lecture slides cover that Implement fault tolerant workloads so we have uh EFS elastic IP um Implement Route 53 routing policies you don't really need to learn how to do this uh Hands-On but I did make a Hands-On lab just in case implement backup and restore strategies so we have automate snapshots backup uh based on use cases here they really want to uh put large focus on A's backup I added Amazon data life cycle manager but it's just a life cycle system for uh EBS so once you learn about like S3 buckets right U and their life cycle then you start to understand life cycle feature in every service restore datab so you should understand how point and time restore works you should try to back up um even if I don't have a lab on it I think I do but if I don't you should try to back up a database and then restore it and see how long it takes because it takes some time for it to restore uh Implement versioning and life cycle rule so that stuff's pretty straightforward uh cross region replication we covered that in the solution architect associate it's it's repeated in this content here for you perform data uh Disaster Recovery procedures um I mean I think it's more so about there's like a um there's like a a diagram that they they have that looks like a thermometer I'm forgetting what it's called off the top of my head but if you understand what that thing is that's what they mean by perform Disaster Recovery procedures provision to maintain Cloud resources so create and manage Amis so we definitely have a lab on ec2 image Builder create manage troubleshoot a cloud formation we cover that extremely thoroughly because it's useful for both actually all certifications you should thoroughly know CL inform the only course where we kind of left it uh thin was in the solution architect associate but we had lots of labs and Cloud information but we didn't go through the details of the specification of the language provision resource resources across multiple abis regions and accounts um so we have Ram stack sets stack sets don't come up a lot but if you conceptually understand what these are you'll be in good shape uh select deployment scenarios and services blue green rolling uh Canary deploy so yeah we cover deployments but again deployment is more so in the adist developer associate so if you do have any questions uh that is going to round it out a lot more and it's not even being very specific about what is being blue green like rolling Canary so this is kind of weird because usually youd say like in the context of you know is it code pipeline it like what is it and so they're just generically talking here this is where I keep saying that they're adding junk lines in here um identifying remediate deployment issues so for example service quota subnet size in CL information errors permissions again this is another junk line but service quotas is something we cover in the cloud pratitioner so we're not covering it in this course as it's already in that one subnet sizing this is not something that comes up a lot but I guess what they're trying to say here is that when you choose uh to build a network that you have to make sure that you have the right size of subnet um but honestly this is a problem for Enterprises and this is not this is not associate level content so and not going to show up me your exam um other than just understanding sizing of subnets but there's like like identifying and and deployment issues because of subnet sizing we have automate and manual or repeatable process so use dat services like System Manager Cloud information Implement automated patch management and they're specifically talking about Windows servers here schedule automated task bya Services um we have security and compliance a lot of this stuff again I cover in the cloud petitioner and and I don't know why they they did this so heavily here because um it's not like they ask you so many security questions so again I'm not exactly sure as to why they did this but um anyway we have the content and and we've brought a good chunk of it over into the um sis office administrator but if you need more do the cloud partitioner it won't take you that long uh but there's some key ones like KMS so any or ACM we definitely have uh dedicated videos for ACM same thing with parameter store at Secrets manager Secrets manager is something that will definitely show up on the exam so make sure you know that one um we have base videos for this we probably includeed in this course here networking and content delivery so it's more so about networking than the content delivery part as content delivery is going to be more covered in the developer um but we have VPC we just have a very thorough VPC section we include in all of our Associates so you're going to get that uh private connectivity um it's more so about knowing how to do it as opposed to implementing it because some of these things are expensive or difficult to implement so just understand we're limited for what we can do in labs for those we have a configur network Protection Services WAFF Shield so we can't really use the paid version of of Shield we can only talk about it um abos Waf is pretty straightforward and we do cover that there for ref 3 we have hosted zoner records routing policies resolver which is not easy to use we learn about what it is O AC's or origin access controls there's o AC's there's o AC's OAC versus ois I covered this in the course I just can't remember which one is the newer one um yeah so O's is is the new one and we we definitely walk through cloudfront example of O AC's we have static website hosting which is covered more so in the developer and the solution architect and we even cover in the cloud pratitioner so it's really bizarre that this is this is in the list but you should understand about it so if you haven't done it just understand the concept about it it's not going to be very thorough about checking you on that one troubleshoot networking connectivity issues so interpret VPC configuration so just understand how to configure vpcs collect and interpret log so we definitely uh do these uh in the in the course identify remediate cloudfront cashing issues so we we cover the most basic uh cloudfront issues it's not too difficult troubleshoot hybrid and private connectivity issues that is not a fair thing to put in this exam because again it's very hard to set up hybrid and private connectivity issues so you'll conceptually learn about those things but we're not going to do those uh from a lab perspective cost and performance optimization so all this stuff I'm not even going to touch here because again this is stuff we cover in all the other ones it's implicit why they have a dedicated section on this I have no idea it's an absolute mistake in this course let's go down the appendix um so there are services in here that I put in the exam in the exam and that's why my course is so darn long because even though they don't show up on the exam I never saw them and I don't even know why they're on here they're in the list and because they're in the list I just want to cover our bases so we have Open Source service event Bridge SNS sqs so this one I don't know why it's on here but it's here um these are should be known by every single certification so there's no reason to explicitly cover those but they have them here um yeah these are fine this is fine they don't list memory DB but I've added memory DB into here the adabs tools and SDK I thoroughly thoroughly do a job on the CLI and SDK tools more important to the developer but just as important as this Ops one understand that it is overkill but it's not for the certification it's for your own good so that you are good at using AWS cloud trail Cloud watch those we cover extremely thoroughly compute Optimizer um is not a very important service very simple um but it we've added it to the list ad config uh very straightforward Service as well more important in the um in the uh security certification but it's here control tower is not something we're going to implement um we should know about it we have a slide on it it is a pain to set up it is a pain to use customers don't like it um so that's why we don't thoroughly cover that one AIS Health dashboard and license manager those are CL predition level stuff and we brought them over in this course even though it's they're already covered in that one is Management console is the thing that you use so there's no point in listing that there organizations we uh cover but again mostly in the cloud partitioner and it's a pain to set these things up um so just understand that the labs might be limited for that one service catalog we conceptually talk about it um it's not a it's not a very good service it's more for the Enterprise level and something again we cover in the comp petitioner so carried the lecture content over for that systems manager we cover thoroughly though I noticed that when I went in there they did make a bunch of updates so understand that I might not be covering the most up to-date uh UI but I'm covering all the conceptual things because those things have not changed and we thoroughly cover them like how to build documents and do run commands and automations so there might be a bit of discrepancy with um the UI because they're always changing it there but I wasn't going to go re re uh redo all those slides because they they changed superficially some some things there trust advisor is something that we cover in the cloud pratitioner it's no point in covering in any associate or anyone beyond that you should have that at the base level there uh data sync was not something that ever came on the exam for me or anybody else same thing with transfer family but we covered them and I actually have a very complex lab on transfer family that uh covers um I think it's transfer family no no I'm thinking of uh the FTP Service the transfer family is uh something else so so forget that well I don't know I guess it's in the course you'll find out when you take my course here uh cloudfront we cover elb Global accelerator we do that one I was surprised how easy it was I was I was uh last time I used it it wasn't so easy and so it's really a nice experience now ref 3 you'll have to buy a domain if you want to utilize it if you don't want to buy domain you can just watch me and that should be sufficient enough Transit Gateway which is not easy to use VPC uh VPN okay ACM we definitely do a lab on ACM detective which is cool to look at but not much to talk about directory service which we do a lab on and is very difficult to use so if you don't want to do it you can just watch it and learn firewall manager which is expensive so we don't really run it for real guard Duty which is very straightforward we already covered in Cloud petitioner so we're not going to thoroughly cover that one again IM we thoroughly cover IM uh so we're in good shape there access analyzer that's a very simple service inspector that's a very simple service but they made the offering a lot larger so I didn't cover the entire service something I would do in the security one but uh it used to just be for scanning um uh scanning uh a machine like an ec2 instance to see how hardened it is um but there's a lot more offerings there KMS which is very straightforward Secrets manager we cover thoroughly uh security Hub we check out uh Shield we can't use for real wa we we utilize to to some point um we cover it was backup that one was a pain to learn but I I learned it and I'm teaching it to you um personally I wouldn't find much use for it but it's more of an Enterprise thing because you can consolidate all your backups in one place EBS we thoroughly cover e EFS uh actually had some nice improvements it's even easier to use than last time so liking that service but very straightforward FSX um where you have your own file systems we do a lab on that and the lecture content it is hard but uh we do cover it um so just do your best to get that that theory down S3 which we absolutely cover in very thoroughly same thing with Glacier same thing with storage gateways and there's a lot of services that are out of scope so hopefully that gives you kind of an idea of what is going on but again I think that there's a lot of stuff in here that you're just never going to see on the exam so don't get too frustrated with the amount of content uh do your best to go through as much of my content that you have and then do the practice exams and if you're scoring good in the practice exams you should be pretty comfortable with it and again don't skip that cloud partitioner I will see you in the next one okay [Music] ciao all right so we're on to the introduction uh to cloudwatch here and so cloudwatch is a monitoring solution for your ad's resources and it's really an umbrella service meaning that it's a collection of monitoring tools and there's a lot under here starting with cloudwatch logs here so this is for any kind of custom log data that you want to centralize such as applications logs engine X logs any kind of logs you want then you have cloudwatch metrics these represent a Time ordered set of data points so think of it of a variable in time that you want to monitor maybe uh CPU usage memory usage Network in network out uh then you have cloudwatch events these triggers an event based on a condition so take a snapshot every hour the service has now been rebranded as Amazon event Bridge um but some people still refer to it as cloudwatch events then you have cloudwatch alarms these trigger notifications based on metrics which breach a defined threshold then you have dashboards these create visual visualizations based on metrics you have service lens this visualize and analyzes the health performance availability of your app in a single place and this one really pulls in uh all three levels of observability uh for you which is interesting we'll talk about that when we get to that section we have container insights so this collects Aggregates and summarizes metrics and logs from your containerized apps and microservices you have synthetics this test your web apps to see if they're broken and then we have contributor insights and this views the top contributors impacting the performance of your systems and applications in real time so you can see there's a lot of services but the ones you really really need to know are the top five that's logs metrics events alarms and dashboards these other ones service lens container insights synthetics and contributor insights are new uh they're not going to be so important unless you're going for the devop devop Pro or you're going for the CIS uh sis Ops administrator uh but it's good to know all of them anyway way and I just want to emphasize that all these cloudwatch services are built off of cloudwatch logs and I want to emphasize that here just to show you what I mean so down below we have logs so again logs is a place to store all your files and lo and Metric uh leverages logs to turn uh to turn them into monitorable variables and then the dashboard leverages metrics to turn them into graphs if you want to create alarms it it it uh builds off a metric if you want to use service lens it builds off a metric but service lens actually pulls in uh multiple services so that's not exactly clear but it's just the way I fit it into here then you have events these trigger actions based on event uh data you have contributor insights uh and then you have uh or sorry container insights then you have contributor insights which is all over it's by itself it doesn't use logs and synthetics which doesn't use logs but there you go so that is the introduction to [Music] cloudwatch so before we jump into Cloud watch let's take a look first at the pill of observability so we understand the utility of all these cloudwatch services so first let's define what is observability this is the ability to measure and understand how internal systems work in order to answer questions regarding performance tolerance Securities and faults within our system or application and to obtain observability we need to be looking at three things that's metrics logs and traces and they need to work together to give us observability we can't just use them all in isolate and expect that we have observability they to work together so let's look at those three now starting with metric so metric uh just think of it as a number that is measured over a period of time so if we measured the CPU usage and aggregated it over a period of time we could have average CPU uh then you have logs so these are just text files where each uh line in the actual text file contains data about something that has happened at a particular time uh then you have traces this is a history of requests that travel through multiple apps or Services where we can pinpoint performance or failure uh and there's one uh extra bonus one here especially regarding AWS which are alarms sometimes considered the fourth pillar of observability but when observability were defined alarms were not part of it uh so I'm just adding it there and I like to make the joke that it looks kind of like the Triforce so they should have called it the Triforce of observability so there you go all right so we're going to take a look here at cloudwatch logs and this is a service used to monitor store and access your log files but specifically it is a centralized log Management Service uh and it does a lot of things it has a lot of integration so let's talk about them right now the first is that you can export your logs to S3 so that you can perform custom analysis maybe use Athena for that or um uh some other service you can stream your logs to lassic search service so you can have full robust or uh robust full Tech search or use the elk stack uh you can stream cloudwatch events to cloudwatch logs and this allows you to to uh then uh uh uh analyze or leverage your cloudwatch or cloud trail events just just as if they're Cloud watch logs uh it's secure by default so all the log groups are encrypted using SSC you can use your own uh customer master key with adus KMS uh so you have a bit control over there uh you can uh do log filtering so it has a filtering syntax um and cloudwatch logs has a sub service called cloudwatch insights which we will cover uh in this uh section here uh for log r mention uh all logs are kept by default uh indefinitely and they never expire but you can adjust adjust the retention period for each log group keeping it indefinitely Or choosing between one day and 10 years um and most Services integrate with cloudwatch logs uh sometimes you have to turn some services on or require IM permissions to use cloudwatch logs but more or less AWS will do the heavy lifting for you to uh you know set those permissions and turn them on so there you go now let's take a look at cloudwatch logs long group so a long group is a collection of log streams and it's a common to name the log groups with the for SL syntax which we'll see here in a moment uh so here uh if you were to open up cloudwatch you'd have log groups and you can go ahead and create a log you would name it using that for/ syntax um and so it would then go appear here uh the reason you'd want to do that is then you could scope it based on maybe your production environment or whatever convention that you want to use um you can see that the retention is never expire uh but you can change that from uh never expired to 120 months here which we can see here which is 10 years and that's log [Music] groups so we were just looking at log groups and log groups contain log streams let's talk about what log streams are so log streams represent a sequence of events from an application or instance being monitored and you can create uh Lo log streams manually but generally you don't have to do this ads will do it for you um but anyway I just want to show you some examples of different log streams and again if if you have a log group you click in here and this is what you would see so for Lambda function you can see that the uh the log streams are uh named based on the time it ran and also it has uh like an ID on on the end of it on the end of it there uh if you're looking at a log group for an ec2 instance what you're going to see is that it's going to make the log stream name just the instance ID and if you were to use adz glue which is a service that uses um cloudwatch it's going to name it based on the glue job so the convention for log stream names is all over the place but you can see certain Services have certain conventions um but they can be whatever we want them to be [Music] okay hey this is Andrew brand this video we're going to take a look at cloudwatch uh so what I want to do is work with logs and I want to create a log and send data to the log and so I'm hoping that we can do this completely programmatically I actually haven't ever tried to send individual logs um using the uh uh the SDK or C so I'm very curious to see if that uh that could be accomplished worst case we could always just spin up an instance and uh stream data but let's find out what we can figure out here so um I'm utilizing our ad examples repo I'm opening this up in G pod you can use codes spaces or whatever you want to utilize just remember you need to configure your adus credentials uh for your account mine should be already preconfigured for this and I just set those as environment variables um but I'll wait for the terminal to open here um so we give that just a moment here you see I have a lot of folders and once terminal is open here I can go ahead and create myself a new directory we'll call This cloudwatch And since we might do more than one thing in cloudwatch I'm going to go ahead here and just CD into that Direct three and we'll just say mkd logs okay and then we'll go into logs here and again we might do quite a few things here I'm not sure maybe we might not do much but I'll just go basic for now Amazon Q has moved to its own extension I don't care I don't find Amazon q that useful um and I probably not going to use it anytime soon anyway so let's go down below I do find um I do like GitHub co-pilot and I do pay for that so I kind of want to bring that in here uh if you don't mind so just give me a second I'm going to get co-pilot installed I don't even see the option so I'm not really sure if we can actually uh do that it says this extension is deprecated because this runs on um uh the open vsx Library so it's like uh they'd have to publish to those ones and I'm not finding it here right now I guess it' be part of the GitHub extension but for whatever reason it says it's old so I guess I'm just not going to use any AI assistance I don't really need it um and uh if you don't need it don't bother with it as well it's up to what you want to do here but when we say co-pilot something about GitHub co-pilot let's go ahead to that um uh to here I'll just make a I'll just touch a new readme file here readme.md and on left hand side here we'll just expand this and I'm going to drag this into the basic folder and so the first thing we want to do is create ourselves a log so that should be pretty easy we'll go ahead and tip it CLI cloud watch logs and see what options we have and we have create log groups we've clearly done this before we could easily use cloud formation for this but I think for this one it's so easy we're just going to use uh the CLI here today okay so we have this and there's a pattern to the naming where you'll do a forward slash I probably cover those in the slides I'm actually just looking at my slides here if I do talk about it but um uh and like here's just like a slide here but you can see like the scope can be whatever you want so here this is like based on date this is log stream um so maybe log group here yeah so you know you can do the forward slash and have whatever you want it just depends on what you want to have so I'm going to go ahead and just call this um uh example uh log uh app okay so or we going to say example app I suppose there is no app per se but maybe we'll go basic here we'll say basic app and that'll be kind of the scope again you don't you could just name it one thing if you want to but I'm going to do that here today just because it has forward slashes in it I don't trust it so I'm going to give it double quotations we'll go ahead and copy before I do anything else I'm just going to make sure that uh I actually logged in here and which account I'm logged into it examples okay great and I believe that is the correct account so we'll go ahead and copy that good habit to do that once in a while and I think uh last time I was fiddling with here and I changed my font sizes so I'll just quick fix that you're just say font size terminal increase and what is it this HCK key this HCK key I'm trying to figure out what the hcky is here control shift p I'm pressing that I'm because I'm in a browser it might not be acting the way that I want it to work so I'm just going to have to do that a couple times like this there we go one more time and that feels comfortable to me anyway so let's go see if we created our log so I'll go over to Cloud watch and we'll see what we have I might have a bunch in here I don't know we'll go to cloudwatch log groups and so you can see I have a bunch but the one we're looking for is example basic app so if we click into this okay you can see we have no log stream so nothing has been sent over here all right but what might be interesting is just to take a look at what it looks like when we create a log so I'm just going to click this I'm not going to create one here but notice we have a retention setting so we can say when it's going to expire we have a log class whether we want to save money with the infrequent access um we can apply KMS key I would assume that I would think that they'd be encrypted by default but I'm not certain uh uh for certain but you could add that uh KMS key there if we go into it there might be more configuration because sometimes there's configuration that you set time of creation there's stuff that you can set afterwards so if we go here it looks like there's things we can create on top of it so we have a metric filter data protection policy um which is kind of interesting but for now we'll leave those Alone um and what I want to do is go ahead and create a stream but before we move on let's just go take a look at that retention option because I wonder if they even have it in here on the create it might be a separate action so it says it it will not expire so we actually would have to set it separately I think that's a good idea to set up a retention uh policy so I'm going to just click back here to the logs and we'll look for retention so that's just like one of those places where the um uh the console is doing two things but there's actually two separate commands so we go down below here and I'll change this um because logs can add up and cost you money so it's probably always good to set a retention period I'm going to set one day here and then I'm just going to grab this here and paste it in as such and we'll go ahead and copy this command let's just say create log and then set retention log all right and we'll go back over to here we'll go to our log groups we'll go down below [Music] and did not change now maybe maybe it's cuz we have to do a hard refresh here sometimes that happens where if you're clicking around it's not pulling the latest data we didn't run into an error so that's fine so go back and check again one day okay great so that's what I want so the next thing I want to do is create some stream data now again uh usually there's things you integrate like the agent and uh it will be the one that is sending the data but I'm just really curious if we can just just directly send data uh to it and probably probably something we'll want to do is create a stream because that is a component so there is create log stream so let's go ahead and do that next and we'll go down below to examples and I also just curious can we create that via the uh we can it's just name okay so there's nothing special going on here say create loog uh log stream this will actually be creating a log group so that is good there we'll grab the name and then we can give it whatever name we want so we could say like a name right now we might want to give it like the current date I'm just going to see how do we get that as um UTC or uh times so say uh I'm looking for date in uh Unix format Linux so I'm just looking for it here I want Unix format this website has a lot of ads so it's like really buckling buckling my uh memory here we'll go down below here I just want to know how to get this in Unix format okay I'm going to go ask chat GPT I don't have time to go on these uh adrid websites so we'll give this a moment here I'm not sure what this is up here oh it's just my logo here so let's just say how to uh get uh Unix timestamp in Linux as a command it's probably like some kind of formatting we have to apply on in so that's probably what we want so go ahead and copy this I'm going to paste it in here hit enter and that looks good to me so what I'll do is put it right here and we might have to wrap it like this I think that's what you'd have to do and so hopefully what that's going to do is create one right now so that's probably how we'd want to have a stream we'll go ahead and copy this and paste and hit enter and we'll take a look and see if it did what we wanted it to do so we'll go ahead and give this a refresh and there's our stream if we go into it nothing in here so the question is can we push data to this this is what I want to know and so we'll go back to logs here and I'm taking a look here I don't think deliver is that that might be like importing can you import a delivery is a connection between a logical delivery source and a logical destination okay so maybe there some kind of connections I'm not 100% certain what that is right now probably I have to go add that to my cloudwatch section here but I'm just going to carefully look through this here delete resource policy describe get delivery probably be like a put I would think put retention policy put delivery Source create or update to logical delivery source so I'm not really sure but let's find out can we use the adabs CLI or ads SDK to directly write to a stream because I think that'd be interesting to find out oh it says you can well what's the command put record ah there we go definitely going to update my um lecture content you'll probably already see me have shown it to you and then it'll seem like it's like why don't you know but we don't see put record here so where is it is it lying to me so we go over to here no no no no no no no no I I sorry I wrote it wrong here cloudwatch log stream not Kinesis I mean there has to be some way because there's an API right um but maybe it's just not possible and we have to just use the agent I'll give it a moment to think okay so here they're suggesting the the ads SDK for JavaScript has a method here and down below it's suggested there Cloud watch real user monitoring oh they have rum now oh yeah I remember there's rum now okay let's go take a look here and see what they say so it says create your log group that's what we did put log events so that's what it's called okay so I didn't see it there okay oh it is there okay great so that's what it was that's what we're looking for so let's go down to the examples and it looks like we can just take a file and send it over um so we'll go here and just say uh send logs to uh logstream all right and so now we can have a log here so we have log group name my log log stream name 20 2015 uh 601 Etc okay so and then we have our file so maybe you can send multiple files here I'm actually curious about that so we'll go back over to the API here and I want to check that specific parameter where it talks about um log events so we go ahead and we'll say log events um represents a log event which is a record of activity that was recorded by the app and so it's expecting a time stamp and a message the raw um event message no larger than 256 kilobytes so I guess the question is like if you have logs do you include the log in the message or do you separate that out um let's just look at some log formats com like common log formats or is that actually a type of log format called common log format for uh for log management common log format is a standardized text format when when generating server files uh for web analysis or web an analyzer if there's a standard that's great I'm not the best at remembering these but here it chose the IP address um the user identity the person and stuff like that so I would think that the challenge with this is that uh we would have to it seems like we'd place it in both I think that's what's happening here let's going an extended log format this is another standardized format that is used by web servers in comparison EF is is provides more information and flexibility okay so what I'll do is I go to chbt I'm just I just want some data so uh generate say generate genen genen generate out log data in elf format with 100 records uh from a web server make it look like real data so we'll generate that out and then and if you don't have chat GPT or something else I'll just copy it out just just don't do that with me here today well you know what actually let's do it let's let it do that I was being difficult I just wanted to give me a file but you know what let's actually make a file that's going to generate some fake fake data I think that's actually not a bad idea first I was mad now I'm not so we'll give it a moment okay personally I would have preferred Ruby but I did not know it was going to generate python I know folks like python so I guess we can utilize that here it looks like it was missing something here so generate a some something so I'm not sure if it just didn't finish but what we'll do is we'll go ahead and make the script I can use Python that's fine we'll just say log. piy or I just say I don't know if that's a reservoir so say my log. py and I'm going to go back over to here tat PT hello wake up come back white screen give me a second okay there we go it's back so so we have um it's back here and hopefully it sticks around so I not sure if it's missing some but we'll find out here in just a moment so I'm going to copy this content again you can just run these files but you can just see that you'll be part of the process of how we would actually go ahead and get uh create this but I'll go back down here below and this should generate the logs so I have a feeling that something's missing here I can't really tell just yet um but we have generate time stamp method path status path status method user agent IP uh no looks like it's all there um the only thing is that this generates all the logs but it doesn't write it to a file yeah it's not writing to a file so I'll just go ask it like okay um show me the code to write okay this should write the logs to a txt file with each record on a line oh never mind it's telling us how to run it okay but I kind of prefer if we just ran it and it it would create it but I guess we could um send the logs to an output like that okay we just try this out that'd be interesting because this would this would print out to um uh St out anyway so if that works that'd be very interesting so we'll go up here and just say uh create logging data and we'll assume that we can run it as such so I'm going to go ahead and we're in logs we're not in basic though do we need to do pip install here is there anything special we're using no it looks like all standard libraries so we'll go ahead and copy this paste this in below and it says it's complaining it's still generated out something here so I'm not sure what it's complaining about but it says can't open file generate logs. py oh because it's not called that it's it's called well you know what I'll just rename it because that actually is a better name and I like that it's using an underscore so I'll just rename that as such okay and at least it made the file which is weird there we go now we have some data which is pretty cool and so the thing is is that in order to bring this data in let's go back here it says timestamp log message string so the problem is even if we have this as a log and like even if we had it like this we'd have to take that file and then iterate through to do it or we'd have to modify our python script to um uh utilize this but what we will try to do here I'm going to try again use chbt so we'll say um say uh take the uh take so take a log file that is in elf format and let's iterate let's write a bash script that will send the logs to cloudwatch log stream and so what I'm thinking is what we'll have to do is it we'll have to uh parse it each line pull out the date and then Loop through it and send the data there okay so I think that's what it's going to do and let's just take a look and see if it actually makes sense okay so here it says get the sequence token for the log stream um why is it called sequence token okay we it didn't talk about sequence token when we looked at the CLI let's go back here and take a look here so sequence token the sequence token obtained from the response from the put log events does it describe it anywhere here each subsequent call uh requires the next sequence token provided by the previous call to be specified within the token okay that makes sense so it says the following command puts the events to the log stream and then we move on okay so let's go back here and take a look see if we can make sense of this so we start the stream or we start the put events I'm going to copy this so we can just take a look at it and if it doesn't work that's totally fine I'm not going to run it blindly I never run things blindly and so I just want to call this uh put um put logs and then we'll paste that on in here and I'm just going to chmod this so say chamod U plus X generate logs I not sure I think that's the right one I can go here and just say where is Bash it says it's user bin bash so I would think that it has to be user bin bash we usually have older scripts so I could just go check one of them I can find a um a one here I'm just trying to find any that might be a good idea here so look at like deploy script user been user bin EnV bash I mean I think these are the equivalent but I'm going to do this one because I just know that's more portable anyway so uh let's take a look here more closely so log group name stream name log file and we create the log group we've already done that so we don't need these to okay the sequence we are probably going to need so I'm just going to go ahead and bring these on down like this it's a little bit easier to work with and then we have function to send log events which is fine read log file and send logs to cloudwatch so it's saying it's making a for Loop and it's doing what I thought it's extracting out the Tim stamp because that format was asking for time stamp and message so it takes the whole line with the date in it and then it separates it out and then here it's calling that function um and honestly I don't really like writing functions by hand with bash so it's totally acceptable to use uh generated services for this um I don't like how long these are I think we could do this I'm going to just see if that works by the way if we do this I have to do this as well here where that's going to mess up let's just make that a little bit easier to work with again I like my stuff readable and easy to work with so I'm going to go ahead and do that this is probably why like in the past we never saw really good um thorough tutorials programmatically because there so much work to write these by hand and people didn't want to do that so it's really nice that we have generative AI for that stuff what's interesting is we could take this um sequence token and put it probably in a function as well but we will just stick with this code because I don't want to make things more complicated than it is and even me doing these backlashes probably might make a mistake but I want to carefully read this so we have send the log events so we have log events here and it says dollar sign one um so I think that's referring to the first input because when you have a um a b script use dollar sign one dollar sign 2 to bring in parameters right and so I'm assuming this is in the context of what was passed here which is over here and so we have this log event here and then it's saying if there's no sequence then start it from scratch otherwise let's go ahead and create it right and then it says go get the last sequence now this might actually return the last sequence so I'm not sure if this is inefficient so notice here like you put the log event it will actually return the last sequence so this additional call is probably not needed um next sequent event so I'm going to tweak this and I'm going to go here and do this and do this and we'll go bring this down here and we'll say um query and it should just be that and that should get the next one okay I can't remember if I need a period or not here I always kind of forget but I think that's all we need yeah I think that's all we need and so then we can do this and then called this sequence token like that and so that would be fine now the other thing is that this one probably returns one as well I think they all do so if we go to uh put logs here it's the same story next uh sequence token so we'll go here and do that that's why it's good to know what you're doing and read it not just take code blindly because you can get better results that way so I'll paste that in here as such and then we'll go here and paste that in his such and I think that's fine uh yeah this has the wrap around it it doesn't seem like this one does and this one's supposed to have this on it otherwise it's not going to select it correctly and that's supposed to have that there okay great so I think that this will do we want I'm going to go down below here and so we have this one here what's the point of getting it if we always get it somewhere because even the first one we don't get it so this seems really silly to me I'm going to get rid of this right and let's go back and take a look at these logs again yeah this one is the initial one yeah and so that's a lot cleaner that makes more sense I'm seeing this as red as if it's confused this is here I think this one's extra there we go because that that one matches to that one that one matches to that one that's fine okay so we'll go that back down below and and carefully read this so get the date use a to to get the value or sorry is the one that's out I'm hoping that this works correctly then we have message line which is fine whoops then we're constructing the Json which is fine that looks fine to me and we'll send it that way we could use uh JQ if we wanted to constructed as well but um a string is totally fine as well and then we have the log file as it's iterating through it right so I think that's the input that's coming in I don't know what ifs is unless it's a uh thing there there so I'm hoping that works so yeah I think my script is okay let's go ahead and we'll hold on before we move on we have web server logs. log and we called this what the problem is well it's not really a problem but we'll just have to grab it manually so you're going to have to change this for whatever yours is we could pass in parameters for the script but I don't care we're going to hardcode this today there's enough we're learning here right now and so I'm going to go ahead and paste that in here right so I'm hoping that this just works it'd be awesome if it does so I'm going to go ahead ahead and say put log no this is not autocomp completing so it's telling me this needs to be ched I thought we already CH moded it so say you plus X put logs okay and so now do forward slash like that put logs and right away the dates the date's wrong the dat's wrong okay so invalidate 30 May 2024 input is not a terminal um and so the thing is like when chat gbt is gener out it doesn't necessarily know that we're using Ubuntu or something else um so I think what would be useful is where is this messing up so what I'm going to do I going to comment this out I just wanted to Loop through it right and I want this to just um Echo the time stamp so we can see what it is because I don't know where this is failing this could be failing um when trying to send it or could just literally be failing on this line so that's what we're going to find out right here another thing we can do here is if it fails we can set set hyphen E I think it is and that will stop where like stop on the first issue um so right away saying date invalid it didn't even get to my echo hello we'll try this again and so that is no good um what I'm going to do here is just Echo out the line so we can see it first and so this is our line and so the idea is that we could take this line okay and we'll just say test lineals this there might be double quotations in there so that might be hard for me to assign okay so that's not going to exactly work but clearly it doesn't like something here um the challenge is that there's double quotations in here and so I have to escape them all to assign them I could put singles around it I might be able to do that let's try singles here so I'm going to go ahead and just say um copy this and say say test uh test what they call there line test line or I'll just say tline and then I'll put singles around it I think that might work we'll hit enter and so I'll just say Echo dollar sign T line okay so we have that and so now what I can do is I can copy this here and run it and we know it's going to fail but we now we definitively know that it's failing here tline okay and so that's fine so just give me a second okay all right so yeah we now have this part so we'd have to step through it individually but I think the thing is that uh it doesn't know that we're on Ubuntu so I'm going to go ahead I'm just going to say this doesn't work um I'm on Ubuntu because the tools like a and stuff like that they vary based on uh what you're utilizing but while that is thinking I'm going to go ahead here and just try to run some of these lines so if we do this here like this you can see that it's grabbing um the fourth part of it and then if we do this here like this so I'm just trying figure out what part of it not working here let me take that off there and so it's extracting out the date which is perfect so the issue is like this date can't handle this date so if I go ahead and do this like this and then we paste in this here it's saying date's invalid um so yeah it doesn't like something there so it's probably going to tell us something about the date command what the okay hold on here so here what is it doing it's changing the way it's extracting it out so extract the time stamp so this is another way that it could do it and then it doesn't look much different looks like it's doing the exact same thing so like if I go back over to here what is what is different so it grabs it puts it into a stamp convert time stamp to milliseconds but okay and the flag is now just D instead so now we have d right and plus percentage s percentage 3n it's the same it's the same thing so look it's saying date inv valid date let's see if it understands so I think it's just it's misinterpreting how the date supposed to work this is where we might have to look up how the date function works so now we take a look at here extract and convert a format to a date that it can part so now it's suggesting this so I'm going to copy just this part out and we'll type in clear here and I'm going to go ahead and paste in this line and so now what I want to do is try it with the tline that we have here T line and so all it's done is it's added the plus z0 so that might actually be enough for it to fix it so let's go ahead here and try that out so we'll go ahead and try this still invalidate so I'm going to make a new one here I don't like this conversation um and we'll try this so now it's saying that so we'll try this now okay okay let's read the man so let's go ahead and do this chat PT can't do it for us it's dumb as bricks display the time described by string not now okay that does not tell me much uh so I'm just carefully looking at this here here date date display time described by string and then um I guess the question is like do we need to parse it well it was it was converting to milliseconds right so let's go back to the API here let's take a look here and yeah it wants it in that millisecond [Music] format okay so look we'll say uh bash convert date string to Mill seconds because I mean this is all chat PT is doing and it's obviously doing it in a very bad way um so yeah they're kind of using it here let's use this as an example and see if it works I'll go here and just say date like that and then we'll go ahead and try this okay so that converts it over um and if we just try date here like this hold on here I want to go here what does it what does it do if we just do this it just prints back out the format okay so if we go here back to our format and I'm going just do double quotations around here I just want to see what format it's going to take right so if we do this it doesn't like that if we do this it doesn't like that if we do what do zero on that it still doesn't like that what's the format of the one above here this one um this one here I already kind of forgot what it was go over here huh okay so what date the question is like what date formats what day formats will will date hyphen D accept that's what I want to know while that is figuring that out I'm going to go back and check the man because this is the Crux to our problem and that's the formatting convert seconds since the epoch show the time zone sometimes with these tools what they'll do is they'll have a um like you'll have to specify what the format is but it really tells us next to nothing which is kind of annoying so here it's suggesting that it can handle these so let's just try some of these here but it says it's invalid and this format is an American style date so uh date hyen D does not work with American style date so a way that we could work around this problem is instead of doing that we could just change the format a bit so I would rather have it an ISO ISO because I feel like that one would be more reliable so we'll go ahead and try this here and we hit enter and that works fine so what we'll do is we will need to change we can parse the string right uh here so we'll go ahead and say um I'll go a line above here we'll just say uh time string and we will go and grab this one which is fine I don't think I need the doubles and then the thing we want to do is take this format here and say convert this string to ISO date format leave in the time and don't use date to convert so it might use SD or some kind of other tool I'm not really good at writing by hand and and so what I'm hoping is that it will no no no no no no no no no I want it with bash okay we'll go here and say use bash to convert so A and S is what I assumed it would do and so what this will do is extract out the parts and then reassemble it I'm not sure why we need the whole month here because the month will already be numbers but maybe the point is is that maybe it would be like if it's it says may that's why okay fair enough so what we'll do is um okay turn turn it into a bash function could probably figured that out ourselves but we'll do that anyway and so my question is if I had this with this work could this be parsed so we go ahead and try this so can parse that format which is it's not exactly what we uh we did earlier and so what we'll do is copy this function okay and we'll go here we'll paste that there and then we'll go back down below and we'll grab this here and so hopefully this is what we want to occur okay so I go ahead and paste this here and this will be um now in the format they want so we'll say ISO so uh date time and it we just say uh original date time okay this would be original date time like that and then here we will keep the date we'll take this part out like that and then we'll say ISO date time and so I'm hoping that that will get us what we want so I'll say Echo timestamp like that and so hopefully that will get us our result so I'll go ahead and run this again or I'll just type it manually put logs and you can see it is printing out the line so that's a little bit hard to see what's going on so just comment that out for now and I mean that looks good the question is are all the dates the same so if we go back to here and we look at our times 2407 2414 they're clearly not all the same so it's like it sets the first one and then it just keeps using the first one which is not what we want to happen so first what I'm going to do I just want to see if they're different here so I'm going to say um Echo original date here first we'll try that because if these are all different then we know where our problem is original date time sorry okay so these ones are all different which is good and so the next one I want to do is I want to convert to ISO and see if these are all the same so we say ISO daytime I guess I have to spell it right for it to work maybe that was our problem maybe we spelled it wrong and I mean that one doesn't really look like what we had there so I'm a bit confused why does it look cut off original date time try this again that is not what we were expecting right because when we looked at this they showed an example of this converting out to this will output that if we go back to our output it didn't make that so the time zone supposed to be on there so i' have to copy this and go back to here and say um and this is just Linux it's very finicky to figure this out so we'll say this turned into this using convert ISO fix the script okay what I'm thinking it's going to do is is fix the uh a here so we go back up to here time zone I'm going to go up to time zone here so this is time zone and I go down below to this one no I want the time zone I'm saying the old one did not work uh it did not output a time zone give me a break here watch it's going to give me like me the same script that looks the same to me so I'm G to go back over to here let's take a look says 1 2 3 4 five this one says oh no it has a six okay well we'll see right so we'll grab this again and I will go ahead and paste this in as such and then hopefully this time it will do what I want it's like it's not listening okay let me go figure this out I'll just fix this and uh I'll be back in a second okay you know what I think the problem is I'm looking at this date here and it doesn't have that plus z00 Z on the end so earlier I think we were doing this one here it had an alternate one and so that's probably what I'm not bringing in here so I think it's this this one that's missing I was like debugging it but I was just like staring it a little bit closer and I think this is our problem right here okay so let me try this just fingers cross that this works and by the way if you're trying to do an echo within a function you have to do Dev TTY so that you can uh see it we'll go ahead and we'll put those logs invalidate convert all right let me fiddle around okay you know what um I think what we need to do is just add in the plus and I think what that will do is set it to like green Wich time I'm almost certain that's what it is so just to make our life a lot easier I'm go ahead and do this okay because there is no time zone here doesn't have one I don't think the original string has that right but when we did it before it looked like it extracted it out so that's why I'm confused right I could have swore we had a VAR A variation here where it did it but we can go check the logs go look at the raw data it has it right here right so basically that's telling me that maybe it's not this function it's our it's the thing that actually extracts it out okay so I think this one should be fine we'll go back and put this back in here we could hardcode it in but I want to just get that regex that will do it and so our real problem is this thing here this thing is not doing what we want it to do okay that's our problem um so yeah it's just go back here and try one more time okay all right so I think I know where this has gone wrong so I had to read about o because I really didn't want to get into it but this hyphen f says to use a regular expression and wherever it encounters any of these characters it's going to uh create a break okay so it's going to um uh basically like if you had a string and you did a split on it's basically splitting on there and so the idea is that one will be 30 then it will break on the or split on the for Slash and then may split on the for slash 2024 split on the colon which we have here um and I mean we don't have any Square braces in here but it seems like maybe it's Square braces or that's part of that syntax I don't really know I don't really care um but what I know is that when we print this out to time Zone's 39 and 39 is really the the the seconds here and this is the milliseconds so I'm thinking what we need to do because the time and the time zone can be kind of together I suppose or like I guess it could be separate but for time I'm thinking maybe we can do print I'm not sure if we can do this print four five and six I'm not sure if this will work but I'm going to just try it here and we'll do this and so now the time is getting closer to what we want so I'm going to go back here and say four basically bringing that format back five and six okay and so we look at our time it does not like that so go back here um probably because do doubles here wonder if what would happen if doubles here like this it's not exactly printing how we wanted to print but the idea is that there should be some way to reassemble it right so that's what I'll go figure out here okay all right so I've gotone somewhere I had to look it up and this is how you do it it must be like a sub syntax of a I'm not sure um but anyway so we look here our our time looks right we're extracting our time zone I had to put the plus back in here and I don't think the these squares matter because we are not uh I mean I guess it's a range right so that actually does make sense if we're saying a range of things that's probably what it means as a regular expression so maybe I will leave those in there but technically we don't need them for all of them well we might because then the rest would get appended here but anyway so I think now we have a format uh that should be closed here so let's go run this again and um it's kind of freaking out but things look like they are in better shape so what I'm going to do is just comment this stuff out here because that's just our debug stuff and then we'll try this again so it's not airing out which is good so that indicates to me that it's Lo it's it's iterating through this and we're having a better success here okay so um what I'm going to do is now take this one take this one off take this one off and want to see if it actually produced the correct time stamp and it looks like it is okay great and what we'll do is go next here and I'm going to hope that this just works that'd be really nice if it does so it's sending what's interesting is that it's sending one log but the I guess the question is could it send multiple logs because it's showing that there's an array here right so if we go back to um and it also suggests that it's plural because it said log of events right represents a log event which is a record of activity that was recorded by the application and each has to 256 so it's not really stating whether you can do multi-line but if it's doing squares here it makes me think that it's sending an array and it might be more efficient to send 10 at a time but I just want this to work okay so I'm going to try this out here and another thing you have to consider is that there could be rate limiting so right now we are going to just Spam the the API and see if it works um but you might want to put like a sleep in here or something like that because it might be too fast to send them all like that or we should be sending them batch I think ideally it would probably work in batch let's go ahead and see if this even works as we will find out here so I'm going to go ahead and I think everything's uncommented now right and so we'll go ahead and hit put logs input is not a terminal fd0 okay so um I think at least we're getting to send event logs right and so what I'm going to do is just say um Echo send log events I don't know if I need to do this here but I'm going to do it anyway I'll say Dev TTY okay so it tries to send the log which is fine I'm going to Echo the log here so we'll say dollar song log events I'm going to do Dev TTY and I don't even see anything there so I don't see any logs being passed to it interesting well here's a question if we go down to here can we print out this we'll say Echo log event so that will be this one up here I just want to make sure we don't get confused which one it is I'm going to do this here just put a few of those in the front of it and does that turn into something so that clearly is one that we have here so I'm just going to carefully look at it so we have the time stamp which makes sense um and then the message which appears to be escaped though I'm noticing here in the message it starts here but wouldn't it immediately end here if it's not properly escaped so this is what I'm thinking that this is not uh really correct I don't think that is doing what we wanted to do so I think that we should probably assemble our string using JQ so that's what I would rather do so let's say I want to assemble a uh a safe Json string using JQ I need to create the the message I need to create me uh a array of a single Json object with a field timestamp and a field message and I will supply those values with two separate n bars hopefully it understands what I'm asking for okay we'll give it a moment all right so here we have uh JQ and args timestamp message Etc and so hopefully this will cause us less issues using this now we'd have to have JQ installed and I think I have instructions installing JQ somewhere but let's go take a look do we have JQ already here we do but you might have to install just so you know and JQ could vary based on your machine but JQ is generally a more reliable way to generate out uh this here so I'm going to go here and do this just to make this a little bit more readable I want to bring this onto different lines here so we have this Arc here and then this Arc here um and then this one here okay great I like to bring that down like that let's just be ah we'll leave it up there it's fine okay great so the idea is that this should generate out our log event and so I feel like this would be more reliable I'm just going to comment that out here and this should be more reliable for that generation okay so it's generating out and it looks fine I don't really need this to be multi-line so I'm not sure maybe it's just wrapping is it because it's wrapping yeah it's just wrapping it's not actually multi-line um and so let's go take a look and see if it prefers that and I'm just going to go ahead and say log event here like this okay and we'll type in clear and we'll try this again and we still get this error now the other question is like we go back up to here so that other one might not have been the issue but at least we're logging here which is fine oh you know why this didn't CU that spelled right which is fine but I would have preferred to use JQ anyway I have more uh trust in that one but notice here that it is just printing out the single one here so did it even send this whole thing over is what I'm wondering so what I'm going to do here is just uh maybe do this here see if this helps it's called log event we go back to this one I'll type in clear so now it's showing correctly okay so it did have to be um in those double quotations and so now we are here and so that one is fine so now we get to this part here where it's we're trying to put a log and by the way let's go take look and see if we've actually put any logs here yet we'll give this refresh and we don't have any logs here yet so there is a way of printing out let's say print out all it um commands in bat and this way will allow us to actually just copy it there's a way to do this um I think there's like a here it is Trace and I'm not sure if we need to put the hyphen X there but we we'll give this a go and see if this will let us trace it and we'll run this like this and the reason I'm doing that is because I'm trying to see if it'll print out the a CLI command so that we can just manually paste it and try it so I'm going here and this is honestly a bit of a mess so that's not going to work but what we need to do is now solve this part here right so we know what the log event looks like I'm going to just type in clear here and try this again so put logs and I need to print out the log before we pass it in here so I just say uh Echo log event uh there might be an easier way to be honest if I go up to here to uh this command here I'm just going to copy this here if we said Echo here and then I did singles it might just print this out for us see what that does that it uh did did not we just make this one line because again it's like we don't want to have to like manually bu this in we can save us some trouble here so maybe this one we'll do it oh you know what probably should do uh Dev TTL TTY I'm just close these out so we can see this longer line here um and I want to put this up here so there's more of a chance that this will print out I'm going to put this as the first line oh you know it has to go after this one it's not going to know what it is unless it's after line 15 uh no such F directory I have to put a forward slash in the front there like that and I think TTY stand for it's like T like it's for the teletype machine so TT what what does TTY stand for yeah t t t type writer because the way computers used to work just so you know the way this is in my Vim course by the way if you took my Vim course but when you go and you um like back in 1969 when they had really big computers like the PDP they didn't have a monitor and the way they they would put input into the computer is they'd put it into a teletype machine uh and basically it looks like a typewriter you and you type and it would show it to you on your paper and then you hit enter and it would send it to the computer and the computer would print back the results and so that system's still there and so we're saying dump out to TTY um which is basically to print right and that's why the print print command is called print okay we'll go ahead and we will try this again and so now we can just copy this command the part that we want and then we can debug it CU maybe my quer is messed up like we don't know what's wrong here the only thing that kind of sucks is that it's not printing out the actual um uh the these these ones here uh which is kind of annoying so maybe those aren't accessible there and that's the problem so what if I go ahead and I because we only use these within the function so I'm going to go ahead and just grab these like this and print them out like this and then we could set them as local so I don't think we use it outside of that we don't okay but they probably would work in the function and and keep them at the top but I'm just trying to rule out possible problems that this is causing but I would assume that this would have printed out uh the string but maybe it's because we have it in single quotations it's not interpreting them um so I'd have to do does it have any doubles it has one doubles in it so it's not that bad so I'll go here I'll do this I'll just have to esape that right there okay and then I'll try this again log file ambiguous redirect line 95 oh yeah I don't care about that I wanted to see if it would print this out here and so for some reason this Echo is not printing out the dollar like why is it not printing those out it's kind of frustrating um it could be because we have uh this one here so I'm going to go ahead and just remove this one here just take that up like that now does it print it out oh you know what it's not even um it's not even proceeding to the next one so I don't think that's the problem line 95 95 sorry line 95 I'm just going to take this back up here like this maybe that other one wasn't the problem we'll go ahead and check this again and so maybe it was that dollar sign yeah I think that's what it was it was that so I'm going to take these back up I'm GNA move these back up here hold on we'll just undo a bit we'll move these back up sorry and that was all of them I think and so what I'm going to do here is I'm just going to take this part off because it's trying to interpret it okay so we'll try this again and so now we have our put log that we can test so we can see if this is the issue oh it didn't print them out it doesn't print them out it's driving me crazy okay we'll just manually assemble it just would have been nice because like if we're debugging it it goes a lot quicker um you know so yeah whatever whatever okay so I'm just going to go ahead and just copy these manually but I don't want to print that one out that's going to be pain there has to be like it print it out just a sec okay I'm going to pause I'll figure it out okay it's cuz I got rid of the double quotations we got to put those back in there so those matter those matter and so now I think that if I do this one like this like that it's printing that one out now great okay but why is it not printing out log stream um I don't think it's doing that one is it no no it is okay so they're all there and that was the only issue there okay so now let's go ahead and copy this this one to here and we'll paste it and it has a problem with something here so it thinks it's incomplete so I'm going to try to run this again okay and then I'll copy this I'll just make a new scratch Pad here just say new file here paste this in because it it can't be multi-line right don't want any so maybe we get rid of the brakes is what I'm thinking here um I really would like this to be a single line so go back here and um JQ should not format uh should output the jcon all on a single line no line breaks okay okay let's see if we can tell to do that literally a single flank it says hyphen C okay let's see if that actually does that we go down here this is what it's like being a cloud engineer or or whatever devops it's like fiddling with these scripts till they work witha stuff okay so in here we'll do hyphen C okay and so what I'm hoping for is that this will print out a little bit nicer there we go that's looking a lot nicer and so that's going to make it a little bit easier when we copy and paste this because then we're dealing with line breaks and that's another thing we have to rule out so it still has a problem with something missing here so I'm going to copy this here this command and I'm going to carefully look for where the problem is we're going to pretend that our company does not give us an AI assistance tool and so I go to the beginning of this line here I'm going to tell this file to wrap so I can see a little bit clearer what's going on here edit selection wrap WP where's WP do we have a WP where's a wrap wrap WP WP come on word wrap here it is I use macf I don't use this every day all right so I'm going to carefully look and so I'm assuming that the problem is somewhere within here okay and so I'm thinking I know what the problem is immediately it's because we have double quotations and the interior has double quotations we can probably solve it with single quotation so if we go back to here here the problem is probably um uh the the doubles the doubles which is up in our function up here right and so it probably we just need to do singles and that will fix our issue okay we could also tell JQ maybe we say JQ can you uh could you not do that here so if we did this we'll just change it right now right to this but we'll interpret that if single quotations don't know well we'll try this but that might not solve our problem okay so I'm going to go ahead and do this again and so I don't think the singles fixed their problem I think it's that we need to escape Escape it so I'll go back here and say um please escape the uh double quotations for the string uh when outputting JQ and I maybe there's a function for that we'll find out in a second all right and so they're suggesting this one line here um I said to only escape the double quotations it's like escaping a lot more than that all right I don't know anyway we'll try this and see if that fixes our issue here because we we're going to have to stick with those um those ones there so if we do this if we paste this in here like such okay and then we go back and we place this here and then we do this doubles now I'm not fixing the ones below I I expect it uh to mess up that's totally fine what I want to see is that there and so it didn't print out anything oh because we have to do this here like that here we go we'll try that again and it looks more reasonable so this one is just escaping the interior ones but we don't have any doubles on the outside of it um probably because this is a little bit different so I'll go ahead and do that there we go just because we are uh echoing it out that's why we have to do it that way so I'll go ahead and copy this and pit enter it still says there's an unexpected one here so I'm going to go ahead and copy this here go back to here we'll look at it carefully it obviously is something to do with our our log events so I don't even care about the rest up here I'm just going to go to log events here log events end of word um delete to the start there we go so I can see this more clearly and so clearly there's something wrong here I'm seeing again here this is not escaped so this is escaped over here this is escaped over here that's escaped that's escaped why is this not escaped doesn't seem like it's escaping everything okay so let's go back over to here it takes the whole thing it's supposed to escape it right but it clearly is not doing that let me fiddle with this a bit more okay all right so I just want to show you I try just taking it and I'm trying to just format it till I can get it to work and so I'm trying the shorthand syntax it's still saying a comma is missing or something even though I've I've gone through it so I'm going to just try to put it in the Json format now all right so I got it to work it it worked in this format and so now that I have a structure to which I I know that's going to work then I'm going to change this so instead of passing a Jon object I'm just going to pass these two values and then place them in here so again I'm just trying to speed through here so I'll just get quicker to the solution here okay so you don't have to watch me do every little thing I just just don't want to skip this part here so this part should be pretty easy because now we have um uh we don't need to do JQ or anything like this I don't need to construct it here uh which is basically we're going back to how we was before but we're just kind of simplifying by doing it in the command as opposed to making a string and then passing the string um and so here we want to do timestamp and then message okay but I'm going to pause here and figure out the next part I just simplified it I just passed the line in because it was the message here okay so we're getting closer all right so I got it to work uh kind of um basically back top I was like Hey and you know what it did is like put it into JQ This and like we're bringing back all the stuff we had before which I thought that's what we'd have to do um but a few different things is that it told me I mean obviously for the sequence we should have done output text because um that would produce the thing we want but basically it said just tell it to ignore the error and I went really tell it to ignore the error that seems like dumb but for whatever reason so it says here we C both the output the war the message warning should be suppressed so it's suggesting that it's not a proper problem it's just the warning the command should execute correctly which is true because I go here and it's in the logs okay um so I guess it's just a warning but anyway now that I have these tweak the question is could I import the whole thing also I started with a sequence and I didn't continue on with the last one so I'm not sure if that will run into an issue I'm going to go ahead and try to run this uh I got to get rid of the the set e because it actually is aing out when it hits any kind of error so this might work is that working we'll go back over to here refresh this so I don't see depending so I'm going to stop this here uh let's see how long does it take for cloud watch logs to show up in cloud well you know what I think that it says 5 to 10 minutes but I don't think that this is uh correct I think this is actually when we got it working before remember that I tried it manually and it worked that's probably what this thing is so it's probably still not working I don't think we can ignore the error I think that's chat jpt being really really really really stupid here um so I'm going to ignore that and I'm going to assume that's not going to work I'm going to wait around here just to see if anything shows up so I'll be back in 5 to 10 minutes here but I'm going assume that it's wrong okay also um I'm not waiting I'm done waiting here but also didn't update the bottom one here as this one's still just doing this but again um I think like if it worked we would see another record right so I'm not convinced I guess what we could do is we just tear this down for a second and and make it again as I just want to make sure we are uh ruling this out here so we'll go ahead and delete this log group and I'm going to go back over to here I'm going to go back to our read me which which is down here I really thought this was going to be way easier but like everything with Cloud everything's much harder than it always has to be um so I'm going to go ahead and do this again call this app 2 just in case the Old One's still there I'm going to go ahead and copy paste and we'll set retention to one day here this is now two copy paste and I'll go down uh to this one here okay and so I'm going to go ahead and try running this again well before I do I got to change this to two and this is also going to generate out a new uh stream so I got to go back here and check again and we we could have coded this in Ruby or python or something else and maybe we would have had less issues um because we could have used the SDK but I think a lot of common use cases for devops would probably be utilizing A bash script for this so that's why I kind of focused on this even though we're having a lot of frustration with it um but we'll go ahead and try this again okay so it's running and I'm just going to ignore it I'm going to let it run a few of these and then I'm going to stop here and I'm going to be back in 10 minutes okay all right so I'm back and um I'm G to go check here and I don't think it worked we have no logs so that error that we were getting when I kept Googling it kept talking about Python and the reason why is that the a CLI is ridden using boto 3 and so I'm thinking okay obviously there's some kind of syntax issue in bash it's going to be really hard to figure out and honestly I've always had issues with that and I think everybody that works with bash files do and chbt and these tools cannot figure it out because they're just not nuanced enough to do that um and so we could write it using the SDK using python but since we already having that terminal I just want to avoid python in this case and I've actually uh have a ruby script here so what I've done is I just told Chad gbt give me a Ruby version of it and then I formatted it because they had it like this and I don't trust their code so I just turned it into a class and so the idea is that um I like to write classes as stateless classes meaning that um you have individual functions you know exactly their inputs or outputs and this makes it really easy for testing debugging so here we have a function called run and we create a new client all right sorry I had a phone call come in so that's why I'm a bit U confused um but anyway so the idea is that yeah we we do this we create a client to Cloud watch we uh we don't have to do it here but what this does is it will check it will attempt to create the uh the log group if it it already exists it will error out and we'll just say ignore the error we'll do that for the the stream as well so this way we'll have a guarantee that will create those those two here making our other CLI commands not needed then we have parf log file so we go down here and then we need to provide that log file path which apparently I did not supply here so we say log file path you don't have to obviously change it you'll just run the code AS you'll find it here in the repo and so looking here it looks like it's attempting to parse it grabs the time converts it to milliseconds gets the um this is message here oh it takes the line and strip strip just will remove uh forward or trailing um spes uh or um or even possibly uh new line carriers which is fine and then it'll turn it into uh a a ruby hash and then it'll push it onto this so this could actually be push if we wanted that instead I'm not sure why it does it that way but same syntax I think that one's a little bit more readable so we'll switch it over to this okay and so the idea is that we are uh we have an array here we're collecting that new formatted stuff and then we will have the log events here so we go back up to hear it and so now we're outputting it here right input output this makes reading code I do this in all the languages it's not just Ruby um but this just a method of having functional code that makes your life a lot easier so here if there are no events it'll say it'll raise an error and say hey there's a problem and what's interesting and I think we kind of saw that earlier when we when we had our um bash script um but um apparently what you can do supposedly until we execute this is that you can describe the log group and you can see if there are any streams and then you can go ahead and say okay get me the first upload sequence um and so what that will do is it will grab that upload sequence if there isn't it will air out and it's rescuing I'm not sure if exit one that'll actually exit it out so that would kind of suck but uh yeah so here we go because I think the thing is like you create a stream you have to upload to it and you can't start over from scratch right you have to uh have the stream complete right so anyway um if it notices that an upload sequence exists um and we have a sequence token it will assign us a sequence token which is great so I think what I I need to do here because I didn't do it up here I'm going to go here and just say nil actually I think what it's going to do is it's going to check if there's a sequence token and if there isn't then it's going to just pass an empty one so I was I was misspeaking there how that worked but we'll go ahead and say return so the idea is that it'll check if anything's been uploaded before if it has give us a sequence so if it's not then it'll be nail and then it will just start and that one will turn one we'll go down to our logs so we have our put log here and now we go down below to here to to put log here on the right hand side and so that will take in that sequence here and we'll put our log events and then it's passing them in bulk so I guess you probably can do them in bulk and that would be the more efficient way we had a bit of uncertainty there with the bash scripting because the way TPT did it out and the and the CLI did not tell us otherwise but if this is Ping from Ruby examples it's probably correct and so the log events I'm making sure they're coming from that and so then this will upload the only thing here is that it will return back a sequence but if we don't need to you know what I'm saying like we're not batching this right so how would we know if we we we were over the 250 limit that's something that this doesn't factor in in our script um but what I'm going to do is I'm going to attempt to run this and if it fails I will tweak it I assume this will fail first attempt but the thing that I'm missing here is the um run bundle in it we need the gem file we need to include a couple things so I'm going to put gem ox or noiri noou no Kiri which I just recently means saw in Japanese and you know why because I've been doing Japanese woodw we'll say here we'll put fry in here Ruby is also made by uh it was made by a Japanese person uh so it kind of explains like when you see Japanese words in here we have gem pry and then this is the inabus s k Cloud watch logs we'll do a bundle install and if this doesn't work I will pause the video and I'll get a fully working version come back here okay I promise this time and and so we'll do bundle exec bundle exec Ruby put logs okay no errors that's interesting but you know what I didn't do oh this doesn't actually work because we didn't run it so to run it we could do I'm G I'm going to use rake for this I don't always do that but I'm going to do rake here today rake me uh is similar to make but it's the r is for rake so if you know um make files you know what I'm talking about here and we type a rake file here it's just a way of executing stuff using rake so I'm going to go here and just say task put logs do and um or we'll just say log and then the idea is I can just call this function so we go ahead and do this here and I'll just say require relative and we'll say uh this is called put logs put logs RB and I just copy that so I could easily do this so now we can specify all these things here in fact we could just make it whatever we want um with the exception of the log file this has to be web server logs. log and so this could actually be something new so we could go here and say like example uh basic app three here and then this could be um time. now toi I don't know if we'll have to require time for that it should be uh built into Ruby but the idea is that will get us in milliseconds I think the um the value here so what I'm going to do here is um I'm just going to go ahead and because I want to make sure that we actually get data here and so I'm just going to go ahead and just say puts log events because I want to just see something thing okay I don't want it to get through here and then find out it doesn't Parts properly so I'll go ahead and we will go now instead of doing ble exact we're doing ble rake logs or log sorry did I do a bundle install after that by the way after I installed rake I don't know if I did that and what do we have here expected log stream name to be a string and got an integer instead oh that's because this uh this has to be that so we'll say 2s try that again and put logs the parsing is not working okay great so I'm going to go ahead and update our docs here Ruby Ruby SDK put SDK logs and so we have a bundle install and then bundle exec uh rake log just so you know what it is and I'm going to go solve this cuz I can definitely solve Ruby you know I'm good at Ruby I'll be back in just a moment all right so I ran into one little snag again I'm going to get this fully working but where I ran into an issue was uh this time parsing again the parsing is messing up um and it's just because Ruby time parse can't handle things uh in a standard way I just put a pry here to uh find out that that was the issue um but uh there is a library called chronic and I know it for many years because anytime you want to par something this thing is like a really good to par so I'm hoping that this works but yeah I'll just continue on here okay and I'm actually surprised it's the first time it can't parse something so I just wanted to show that I attempted to parse it but maybe if I provide its format that might um that might help but I guess I should have gone down here and checked and so oh it should have parsed it it does not look that format but you know what it's not hard for me to um manually parse this I'll be back in just a second okay all right so so chronic did not work and so I just had to convert it I had to tell it exactly the format and now I have in the format okay so I'll continue on here all right so uh all I did was fix that time time issue ran the script and it looks like it's working so let's go over here and see if it h happened to insert so I don't oh you know we're not on app 2 anymore right we um ours is now called app three so down below here and we are seeing logs what's interesting is that it's showing these as different [Music] times you know why it's because I ran this multiple times and every time I picked up the time right and so you can see here that these ones don't have any events in them so they they are failed attempts right um and so that's where you know you might want to make sure you have the logs before you create it so we could change our script so it's a bit better here but it instantly logs we didn't have did not have to wait and so there we go so I guess we'll just use our Ruby script and and and that is that um it's uh unfortunate we couldn't get the scrip working but those are the two points of contention is parsing times and dealing with escaping of characters so that's not an uncommon experience with for uh like if you're actually doing logging you probably again use the cloud watch agent and you would specify what logs you want to log and what format they're in um we probably should explore the types of format so I might go back and update the uh course course to have um different formats in there but um that solves our issue so I'm pretty happy with that let's go ahead and commit this just say uh Cloud watch logs basic and yeah that was a longer video but what what can we do here I want to um well we don't have to get rid of those logs they'll just delete on their own after a certain amount of time but if we want to just clean up here we might as well go ahead and do that so we're going to delete this log group we'll say delete and we'll go all the way down to the bottom here and get example here and delete and you can delete whatever you like and I'll see you in the next one okay ciao [Music] okay so we looked at log groups and log groups contain log streams and then inside of a l stream we have log events so let's look at log events so a log event represents a single event in a log file and a log event can be seen within a log stream so that makes sense so if you were to open up a log stream here are log events each of those uh uh each of those lines is a log event uh and so what you can do is you're able to filter um uh these log events with a simple pattern matching syntax so if I wanted to just put in D which stands for debug possibly it will then pull out all the debug lines uh or the uh log events that are for debugging okay but we'll look at some more richer uh options for filtering next with Cloud watch logs insights all right so let's take a look at cloudwatch log insights and this allows you to interactively search and analyze your cloudwatch log data uh it's more robust than what we saw on the last slide it's less burdensome than exporting your logs des3 and analyzing them via Athena and it supports all types of logs and so the way it works is you're going to log into the adus console and go to cloudwatch logs logs insights and that's what you're going to see you're going to have this little language you can put in there uh and you can run the query you can select the log groups that you're going to search across uh and so they do have this own little query syntax that they have it's not too hard to learn it's not exactly SQL but it's uh again it's not too difficult uh and they actually automate automated a lot for you so you can click a bunch of buttons and it'll just generate it out for you uh a single request can query up to 20 uh log groups uh queries time out after 15 minutes if they're not completed and query results are available for 7 days so let's just look at a little bit more stuff here uh so obviously that query language uh looks very complex but the great thing is adus is going to provide you a lot of great examples to get started so what I can do if I go to the the left right hand side and I click on queries they're going to have a bunch of sample queries and they pretty much meet all the use cases you'll need um I I rarely am writing uh queries by hand I just go to the sample queries and then tweak them from there so you click apply and so uh now you're able to then visualize your information it's cool you actually get a little a little visualization graph uh there as well and you can also save your queries um so if you do have a query and you feel like you're not going to be able to write again just save it and you'll have it there for later [Music] so we said that cloudwatch log Insight supports any kind of log file and the way it does this is through discovering Fields so when uh cloudwatch insights reads a log it's going to analyze the log events and try to structure the content by Jing fields that you can then use in your queries so cloudwatch log insights inserts an at sign symbol at the start of a field that it generates and it has five system fields that will always always be generated no matter what uh type of log it is reading from and so we have message this is the Raw on Parts log event we have timestamp this is the event timestamp contained in the log events timestamp field we have ingestion time this is the time when the log event was received by cloudwatch logs we have log stream this is the name of the log stream that the log event was added to we have log and this is a log group identifier in the form of account hyphen ID colon log group name uh and then we we had the systems Fields but let's look at the actual uh fields that it automatically discovers from different services that use cloud watch logs so if you're using Amazon VPC flow logs we're going to get uh these ones if we're using R3 we're going to get these ones if you use a Lambda we're going to get these ones if we're using AOS cloud trail uh we're going to get these ones but notice that we're going to have to look at the full list uh and I believe do I believe we do look at this um but uh cloud trail has a lot of information there so we'll have to see what it actually can pull from it then we have Jason logs so these are these the fields of a Jason log will just be turned into fields and for any other types uh uh that it cannot discover you can just parse the command to extract and create uh etherial fields that it uses in the query so there you go oh just one more thing here I just want to show you uh uh what it looks like uh so when you use a log if you click on fields you can see all the discovered Fields there and then you click on them it'll just add it into the query for you so there you [Music] go hey this is angre BR this video what I want want to do is take a look at um log events so the idea is that we create a log and we want to be able to easily search the logs for information uh so what we'll do is we'll go back to our repo and um what did I not change here I guess I forgot to push this commit you're probably going to want me to push that commit so I'm just going to open this up and push that commit but we're going to work in our it was examples repos as per usual so just give me a moment as I uh commit our last changes and we'll continue on from there all right so I'm back here and we're going to go to our cloudwatch directory and this one is going to be for log events so I'm going to just make a U CD into this directory Cloud watch logs and we'll say mkdr events I'm not sure uh if there's anything to do in this one like in terms of writing new code but in case we do I'll just make a new read me here because I don't know if we can filter um uh logs using the CLI that'd be really interesting to find out because we can filter like this right but maybe we can try to download our logs or see if there's some kind of filtration method I know there's like a way that we can monitor our logs using another tool but let's go ahead here and the first thing I want to do is generate out our log so in our basic because we already have that one working pretty well we'll go back to basic here and we'll do bundle uh bundle install because it doesn't automatically install the uh dependencies and if you remember this one from last time all the script does it's actually our p python script we want to run is that it's going to generate out um an elf formatted log so go ahead here and say um generate say python generate logs and so that's going to Output the logs which you actually already have right here so we didn't really have to make another one but I'm going to just clear that out and we'll do that again whoops just go ahead and delete this file here I forgot that it was committed so we didn't really actually have to generated again but I'll I'll do this again here and I'll give this a refresh did our script oh you know what it is that's not how it works the way it works is we have to Output it to a file like this okay we'll say allow and so now we have our log file here it's the same thing but at least we'll have updated time so you know if you're doing this you might want to just delete it and run it again and so the next thing I want to do is run my uh bundle exec rake before we do that I'm just going to go over to my rake command here and we can just adjust this appropriately so we just say like um Events app like this okay it'll do bundle exact rake log all right and so that should generate out some logs for us we go back to cloudwatch uh and we search for example we now have this one here okay um if we click into our log we have all of our data here it looks like they're all expanded so maybe it remembered from last time but normally you can tell it to yeah collapse all the rows and by the way you can uh copy the results here download the CSV if you want to I've never downloaded a CSV so I'm just kind of curious It's not usually how I work with logs but we'll go ahead and open this up in Excel for fun okay and so we downloaded it it's just the time stamp and message nothing super interesting there but what we want to do is we want to and by the way you could tail this so like if let's say we're pushing this and things were changing then we could see in real time if something's uh there just to be a external plugin for that let's go ahead and we'll type in something so let's say we wanted to filter all the login Pages we say for login and so that didn't exactly work how I thought it would let's go look at the filter patterns I was looking at the individual slide and it didn't show any filter patterns but clearly there is some uh patterns going on here so we have some regular expression stuff so me putting a forward slash probably messed it up a bit and what's interesting is like if you have we're doing we're working with unstructured log data right now so if you have unstructured log you just put a term in two will give you multiples um opt optionals or phrase with parentheses if you don't want it to count as separate on so that's interesting but I think what would be interesting is actually generating out some Json data and um filtering that because you get some Advanced options for that or improved options for that so anyway uh what we'll do is go back here so if we want login we're just going to have to put in login here so type in login and I was really expecting it to match this so why is that not doing that login or let's try HTTP get that out of there so say login contact it doesn't really look like it's filtering how we were expecting it to filter so just give me a second here to read um but it usually is just as simple as write it in give me two seconds all right so I have no idea why but I typed in home and now all of a sudden it's deciding to filter so I'm not sure why that was having an issue it could be something with the URL at the top here so noce that um I'm not sure again if it filters out all the information but we'll go back here to this log here I'm just going to go back in here give this a hard refresh and we'll try that again give it a moment here to reload so we'll go to the top here we'll type in home enter so now it's filtering for home why wouldn't it do login let's try login again now it's doing login and we could use a regular expression so if we wanted to say home or login maybe we could do this um that's usually how maybe we have to put squares around it uh nope not exactly no um I mean we really could just put two here right we could just say home and log if that's all we wanted so we have home in here log in please and so you can see this thing a little bit finicky and uh not very reliable but supposedly works sometimes I think really has to do with how it fil filters out this into the top here but um yeah so here's Define a character class that it be like a bunch of characters or the or and so that's what I was doing was I was giving an or but maybe we have to put percentages in front of it let's see what happens see do um percentage home pipe or this we'll try that so hit enter okay so that now giving us the behavior that we want all right so pretty straightforward uh but it's not the funnest figuring out that syntax but what I would like to look at is Json because Json uh filtering is is extremely useful so it says filter patterns to match terms with Json log events and then we have using filter patterns in match terms in space delimited log events so there's ones that are space delimitated which actually looks like um the one that we have here because uh this one says unstructured right so that means like any pattern above unstructured but um space eliminated is actually what we're utilizing so that's probably we should have been looking at this so here it says you can create filter patterns to match the space of limited events the following code snippet shows a space elated log that contains these fields characters between brackets and double quotations are considered single Fields okay but what's what what are we looking at here so let's create a filter pattern do [Music] this yeah so I'm guessing what it's suggesting here and this looks like it's really looking at like the ellf log so it's good that we did that before but if we go back to our logs here let's just expand this for a second yeah I'm not exactly sure on how how to work that but uh I'm more interested in doing the Json so let's go ahead and uh figure out Json so what we'll have to do let's go back here and we'll have to generate out some Json data so I'm going to go over to chat gbt to save us some time I guess we'll do it in Python because we've been doing everything in Python so uh using python write us a uh script that will create um mock web server logs the logs must be in Json format okay so we'll give it a second there to see what it comes up with all right let's see what it came up with it's apparently going to use Faker which which is totally fine I'm totally comfortable with that so I'm going to copy this and we'll go over to here and we will go into our Vents and we'll call this um generate Json logs. Pi I'll go ahead and paste that on in here and looks fine to me I guess I don't know and um we do need to install Faker so I'll go here and we'll say uh we got to go into the right directory I don't know that's going to add it to a pip file though oops I don't want fake I want Faker so I never remember how to do this in uh python but we'll go here and we'll just say uh Faker that way we'll know that it's included and so what I want to do here is attempt to run this what does it need they probably gave us instructions it's probably similar to the other one it doesn't show us how to run it this one in particular is telling us to save to a specific file up here so it doesn't work exactly the same way as our other one but I'm going to call this just uh uh web server logs Json I think it's totally fine we could even uh increase it to we'll leave it as 100 right now I'll type in clear and we'll say python generate logs hopefully that works and we now have logs we'll go over to here yeah looks like something so that's fine uh so what I'll want to do now is upload those logs and the thing is that we have our put logs here so we'll have to make a new file here we'll call this put logs json. RB and we'll bring over our rake file so we'll say rake file and we'll bring uh we'll have to bundle AIT here and it shouldn't be too hard to adjust our previous script so we'll go into our gem file here copy this we'll go down to our gem file here and replace it and our rake file we'll want something very similar so I'll copy this I'll paste that in here I'll call this put logs Json put logs Json and we will grab this one here and paste it here so what's going to be the difference here well first before we move on let's go back to our rate command here it's already says it's already called events here so that's perfect and but I'll say events Json app so something a little bit different there so we know what it is and um this one's a bit different because what we're doing is we're we're not parsing uh parsing Elf or parsing Json log file is that what it's called ef ef log file okay it is I keep saying LF it's just spells elf I haven't said elf once I'm not sure if that's frustrating anybody but whatever so the idea is that we're going to bring the file into here and what we want to do is we want to read the file log file path probably not the most efficient way to do it here they actually iterate through the lines but what I don't know is if the uh the log file here is actually um uh per line so what we'd have to do is we'd have to check the logs here not those ones but this one and probably what I'd actually want because if you got logs it would actually probably be like line per line right so I go back here and say uh the uh the logs should be adjacent object per line not in an array no commas between the items of the array please because that's actually how a log would look like the other question would be if it generates it out so it does the dumps right so I'm hoping that that will be in a better format it's like writing me the script like three times is it a dummy here today we'll give it a second all right I think chbd is having a hard time because it just kept cycling and cycling and cycling I think what's happening is it can't make it to the end and that's where it's having an issue so it keeps trying to complete it like why is it doing multiple versions of it I don't understand um but anyway what I'll do is I'll just drag this off screen so I can kind of figure it out and just uh change it here so I'm going to try to see what they change so the idea is that we get to generate logs and honestly I'd rather that this works like our last one I don't really want it to uh dump a Json file so I'm going to do is go back up to here and uh in here all it does is we have log entries and then it iterates through them which is fine so here it says logs D logs which looks probably similar to our other one and so what I'm thinking is that we can do this here okay and is this one still called generate log entry it is okay so that actually will work fine we don't get to specify our thing it's just 100 here so I'll just take this out of here and this out of here and this out of here get out of here and um so I'm just trying to look at this one this one here generates them all out and then it assembles a string okay so this is actually assembling a string whereas this generate logs this say generate log entries but that's just repeated here so I'll take that out of this this will actually return back a a log entry object here so that's fine but I don't know how this is going to print it out because I'm not sure what it would print out here so I'm just going to try it I'm not sure what's going to happen and we'll update our read me so it's more like our other one so this will be gener web server logs Jon actually no we just call this Json that would make more sense no yeah it's fine we'll just call Json it's not technically a Json file because it's not format correctly but I'm got to call it that anyway okay so just understand that I'm uh taking some Liberties there so we'll go ahead and try this and we do have an issue because I'm not very good at python um and this is saying cannot open file generate log. Pi that's because this is called Json here and we'll hit enter and so now we're getting the format that we want so this is what I really wanted and I don't want commas on the end and stuff like this like when you're working with big data this is generally the form format that Json will be presented and will be pared parsed upon which is single line and stuff like that the only thing that I'm seeing here is that we're seeing single quotations I'm not sure if it matters if it's single or double so just give me a second and so uh it's saying here it has to be doubles okay so it is printing these out but the problem is that this is not um uh the correct format so we can't just print it like that so I'll just say uh let's say print Json single line JS it's probably stringify but oh I got did I say python I don't want a pretty print I just want normal printing maybe it is pretty print and you can just format it different ways so okay let's take a look at dump here default it says indent none so let's go back to here maybe the other one had that on there and so we'll just go uh Jason dump entry and let's see if that does that works maybe Json like that nope it's just Json is it imported y it is okay missing one positional argument give me a break python dump single line Json come on why is this so darn hard this is why I don't like python just tell me without me having to read a thousand things you know give me a second okay chbt says it's this it better be this okay so go ahead and paste that as such I still think we'd have to print it right so we'll go print here and doesn't like that probably doesn't need parentheses around this let me thinking I'm in Ruby here we'll go ahead and enter again um this is the entry here and I'm going to go here I'm just going to delete this so I can make sure this results works we'll hit enter we'll go back boom okay so do I have the logs in the format that I want yes I do excellent we'll go back to uh Ruby I can't remember if we finished that or not but all we have to do here is um because I didn't want to use the read here I'd rather it for the file because that's actually a better way to do it and so in here the idea is that we need to get the time out of here so we look at our logs which is here we have a time stamp and so that format looks like ISO so it looks like this is something that I would think that Ruby could par so say time parse this but I don't trust it so I'd rather get the um St stpr time for this I'll just say I'll go back over to here and say t stbr Ruby for this date time format because that will save us some trouble here we'll hit enter I hate that it explains things I just want the information come on give there we go there we go that's all I want I just want this part here thank you thank you go away okay let's go back over to here I'm not sure what I'm more abusive to Alexa or chat GPT it's hard to say I'm going to go ahead and paste this in here there we go uh I don't think the other one had singles no they didn't so I'm going to go ahead and do that I think it's just getting doubled up by accident could be singles or doubles here it doesn't really matter and so that should get us the time stamp in the um proper format this is per line so I would say line or this is the same here but we want to take this here and say line timestamp so that will get us our time stamp and then we have our message and our message is the line okay so these are the two things things that we need to return and so I'll go down here delete out the rest yeah the other one does time samp and message so that's perfect and we actually want to yeah push this we'll say log events push like that these are now have to be colon because that that makes more sense and that looks good I'm going just put like do line on this so I don't forget what this is a file for each I always do in the end just I don't get mixed up and so that will get us that now the thing is this line well it's parsing each line but each line is Json so before we can do that we'd actually have to parse the Json because I think it would be treated as a string and so I have to say parse line and that will get us the Jason and then we do this and then we provide the line because the line's just going to to be raw or like a jifi uhj a stringified Json stringified Json that's what I think it's supposed to do okay and I think we're supposed to have a comma there in the end or it's going to error out so if that's changed I don't think we have to do anything different to the script it was just that parsing that we had to change so um I'm going to go type in clear here I'm going to go back to our rake file this one is using web server logs Json and we'll type in clear we'll do bundle exact rake log and it says it doesn't know what it is because we need to change the name here there we go we'll try this again and it has a problem with put log put logs Json put logs Json put logs Json it's the same put logs Json put logs Json oh here right so these are changes as well there we go um and then this one as well I don't think we can use self inside of self like that so that's why I do it maybe we can and I'm just like paranoid and I I over I'm over verbose here but that's how I just do it okay and so if you scroll on up to B integer got time class instead okay so what I'm going to do here I think I have pry in here I'm G to go and put a binding pry right here because maybe maybe it's already in a date time format maybe it was interpreted as a date already and so we go here and I type in I type in Json q and then Json and the Tim stamp check it here Tim stamp it will say class see what it is it's a string so it's definitely not it's definitely not a time stamp so what do is I'm going to grab this here credit and then I'll put it above you know what this is this thing's actually fine it just has to go to 2 I I think that's my problem that's our problem okay so I'm going to go and type exit exclamation mark down here exit exclamation mark clear and we'll try to run this again it says log events in a single put events requests must be in chronological order and I think the reason why this is I didn't know that that's an issue we have to have them in chronical logical order that's interesting but I think the thing is that as we iterate through it I'm trying to think of like would it make it go backwards if we start in the first one it was the first one it would push no no it would still end up in the the correct order let's go back over to our Json here and it's suggesting that's not chronological so I think that's an issue of Our Generation script here so yeah here it's saying random times let's go look at our old one and I think it's just that this script sucks is is basically what's happening here so we're going to go ahead and copy this and the other thing is that this one doesn't even use Faker and it does the exact same thing which is really annoying um so I really don't like the fact that this uses faker cuz then it's like we we have to be reliant on that I kind of prefer this one here and the only difference is that um is how it's generating the log entry so you know what I'm going to do I think we're going to have to just take this one and and bring this one over here sorry we'll go down like this I just want to be very clear which is new and which is old and I'm going to go ahead and paste this in below and we'll close this out close this out close this out close this out I'm going to split this over here and so I I'm going to work with the top one and the bottom one so the bottom one is the old one which is better or sorry like this is the one we want to keep and so all we want to change in here is the generate log entry so if we go here the key difference is that it's just returning instead of this it's going to return this okay so we we'll have um time stamp is time stamp I mean we can just go down the list here we have IP is IP that's how we do it or no you know what I think it's equals I mean that one's using colon okay sorry and then uh huh hold on here let me go back for a second I'll keep this around as a reference here this other one here okay so I'm going to try this again so we have [Music] IP just put two here so it doesn't I'm not sure why it's showing that error that's kind of annoying but um I'm going to drag this on over IP is IP user identifier is user identifier I realized I'm missing the commas I'll come back to that in a second user ID is user ID time stamp is timestamp method is Method path is path protocol is protocol protocol status is status bytes transferred is bytes transferred referer is referrer user agent is user agent okay so this has to have commas around it's going to complain like that this one can go away this one is now log entry um need comma on this one here okay so that's good the other thing we want to bring over here is um this other one so we scroll on down here very similar except this one is just doing this and the other one's wrapped in a main and like you should have a main for um whatever but I don't care it still works without it so whatever and then I think what we can do is we can get rid of this top one and now what's really nice is we can get rid of our requirements.txt because I don't want to have that there okay so I'm going to go ahead and delete this we're going to go back just type clear here we going to type in Python I'm going to go grab it from here the command and so hopefully this works without issue enter it doesn't of course it doesn't and it's saying uh Jason name Jason is not defined we go down below here we just have to require it so say require Json here or import Json and I'm going to go ahead and hit up and so I'll check my logs now and looks fine I'm not sure why our our user identifier and ID are blank and it refers blank is it because it doesn't actually generate those out okay so it's like basically saying like hey you don't have these right so they were first and that's fine I don't care um but now we have data that is more structured the way that we want I kind of remember oh because it wasn't doing it in the right time order right so now I'm assuming that this time stamp will uh go in the order that we want also the other thing is that now it's matching the format of the other one so now I have to go back to our Ruby file and then in here there was this uh formatting and we got to go back to our older one or puts log and we'll go down to this one here and we'll bring it back this thing in here and actually you know what I just noticed this is hardcoded it's not even bringing in the timestamp so our old script has been mucked up this entire time and I never even noticed which is bad um but maybe you notice that as you're working through it so I mean we'll fix it now which is not a big deal but uh this here yeah how would it how would have this one worked before time stamp time stamp yeah how did I not notice that [Music] before okay well it's lesss of a problem for this one but we'll have to fix both so let's just do this one really quickly this one uh all we had to do is get this format here so we'll grab this and then we'll go ahead and grab it like that and so that will fix our issue but the question is like did this log actually do the thing that I thought did because I thought maybe this one uh generates it in order the um time stamp random time now random minutes um so I will just say here like change this function so every subsequent call it will increase the time uh from the previous time because we don't want random times and I don't want to have to do a sort function here okay and so we have a function within a function cool whatever sure why not and so I'm going to go grab this one here and now what I'm going to do is go back and delete this one we're still going to fix our other script lucky for you if you're if you're in the uh if you're watching this we'll fix it so you won't even notice that I made that mistake generate Json logs python log um Json and we'll go up here objective type function is not a Json serial serializer all we changed was one [Music] function okay I'll undo that for one second here let's try that again no problem so it does not like our new function we'll paste it in here what's the difference from the last one face I mean it looks like the same thing try this again all right give me a second all right so now they're just having this external variable here so I guess we'll just adjust that here and we'll place this here and we'll go back we'll grab this one I mean I don't really like the idea of having a function inside of a function but we'll try this and I'll go down here we'll delete this one permanently we'll type in clear and I'm going to go ahead and generate this again and let's see if the time actually increments it's now incrementing correctly okay so that should be less of an issue I'm going to go over to cloudwatch because we're probably creating um maybe the same log but I'm going to go down here and just go for SL examples and see what we have okay so we have a couple here these two here I'll delete these log groups and uh we'll go and type in clear here and I'm going to see if our other one works now and it looks like it worked perfectly fine so we'll go over to here and we'll go into here we'll click on our stream refresh I don't see any data so it looks like it worked so I'm not sure what the problem is there so we'll give that a little wait but we should go back and uh cuz sometimes there can be a bit of a delay here so I'm not sure if that's what we're experiencing right now I'm going to go ahead and delete this one here delete loog group and I'm just going to rename it to like two or something so we'll go down to this I'll try that again and we'll go back over to here refresh no log data okay let me figure this out all right so I went in here and it's passing the log events and there's no error I added more error handling here and it's not showing any problems it's telling me that it's going to Cloud watch logs it's going to cloudwatch logs where is it then I don't see it here it's like it's lying to me um and so I'm not exactly sure what's going on one thing we could do is we could try to generate out fewer logs or just cut down our log file so let's go here and let's just cut it down to like 50 because we know there's a limit to how much we can send and maybe it's just too large okay but it should have raised an error told us something right so try this again and we know that this works almost instantaneously here it's nothing okay let's cut it down to like literally two lines and if you're thinking that it doesn't work I'm going to go ahead and show you this going do binding pry here um my other thought is here is like that's putting out the events which is not very useful it logs it goes down here it definitely calls this function and maybe I'll put a binding pry after here and just see what happens because maybe it is airing out but it's not throwing an error okay so I'm going to just show you I'm going to type in log events Q has like Vim Vim keys there so log events you can see we have message and time stamp those look correct to me we type exit to go to the next stage and we're going to write out the response uh I have to hit Q There R response so it seems like it should work but maybe um I don't know it's kind of weird oh we have more than one log oh you know what maybe we're looking at an old one no not that one so I'm getting kind of confused here I'm going to delete these here just so we have one okay I'm going to take all these bunny prize again maybe we had that one failed and we were just refreshing on it constantly give a refresh here we'll look into here what what is going on like what if we change this to be like we only have two logs in here so there's not a whole lot that could go wrong here um so but why does it not show anything clear all time you have to maybe you have to do like maybe it's outside of the reach that we're in absolute what if I go all the way back to here like all all I can think of is like maybe the dates messed up up and it can't see it apply let's try this how about this uh maybe I have to choose something that's actually today what's the day is the May 30th May 30th here to here nothing so this is confusing right like all right let me figure it out okay you know what um comparing our old one it seems like we uh this is still in seconds right so this might be the wrong format even though it's saying successful it's probably rejecting it because the time format's not right this is just kind of like a blind way of me trying to figure it out uh because this one here will do this and then it will do this and we still haven't fixed our old time here which is uh which sucks okay and so we'll go here um and then we'll put the time stamp in here but I'm surprised it doesn't like throw an uh throw an error or something because I again I really think that's our problem but we'll go ahead and try this again so I'm going to go ahead and type in x exit exit here whoops exit I'm going to go over to cloudwatch I'm going to go back a layer here I'm going to get rid of these two streams we'll delete them we'll go back over to our code here we'll type in clear and I'm going to try this again okay um I'm going to type in q log events and so now it's showing zero well that's no good for our time stamp that's not going to work so there's clearly a problem here so I'm going to go binding pry here and we we toi it twice we don't need to do it twice oh you know what it's not a sign like this that's the problem okay so I'm going to go ahead and type exit here clear we'll try this again and we'll go ahead and type in see I don't want to I don't have to debug this right now but I guess we could just write time stamp here and see what we have time stamp looks better it looks longer it looks like what it's supposed to be I don't think we need the binding pry down here right now so I'm going to just take that out and we'll type in clear oops clear and I'll go ahead and write that and so supposedly we've written our logs which is here or sorry over here over here this is what we're looking for and it's showing stuff in there so now it must be writing the data so there we go so that's record that we want excellent so now we can go back and regenerate out our proper log so I'm going to go delete this file first delete this and we'll go to our rake file here and I'll change this back to app we don't need it to be app 2 and I'm going to go ahead and clear this out and I'm going to generate this out again I'm going to go back over to here and I'm just going to delete this log group here okay and we'll type in clear and we want to rake our logs and then from here we'll give this a refresh and we'll go into here and so I don't know why we have two in here because we hadn't I don't think I ran it twice it doesn't want to delete it's kind of acting weird that's okay and we'll go to here and so now we have it in adjacent structure and the whole point of this was to show you how you could filter based on these let's go down here and take a look so if we go into this property equality operator string okay so that's what it's showing us um let looks like uh we I know I must have documented this in another area so um it must be covered in the the lecture cont CU remember doing this but this I thought it's a special object is the dollar sign but we can try it here I don't think that's what we're supposed to do but we'll try it here anyway and I'll go back over to here and if we have dollar sign time stamp or maybe like in the message if we were to drop this down and say IP equals okay I'm just guessing I'm not sure why it's doing weird things as I write 171 uh 250. 72 that and hit enter will that filter it no okay let me read this for a second okay all right so I just carefully read through this and it just says dollar sign the thing that you want equals and then whatever so we did do that we go back over to here I guess the question is did it actually interpret it as Json that's the first question the way we know is if we expand it it does this right if it expands like that then we know that it understands what we want I don't know if we need to put curlies here but I'm going just go ahead and say IP um grab this here say IP equals this does that work no does this work no does this work there we go okay so that's the format that it wants okay so if we wanted to get something like method put we could do like method here and then put okay and so now we're getting the filtration that we want so that's what I wanted to show for you so we're basically done here the only thing that I didn't look up was like can we filter a CLI uh commands uh like for the logs this way I also want to fix our old script because I don't want to leave you with a broken script so if we go here is there like a get describe filter log events List log events from a specific log group and then maybe you can filter them so they don't have an example here but it has filter expression so I'm going to ask jgpt for this so um AWS you really got to fill out your examples filter log event um it CLI example with a filter expression I don't think it's that hard to do but I just don't feel like writing it out all by hand here and grabbing all the stuff there we go okay great so what we'll do is grab this here and I'm going to go down here uh filter event data and so here we could say method we'll say put and then this log group is I guess it go across all the streams right because we're not specifying a particular stream in there maybe we can because I could have swore there was that option right stream names so let's not trust chat GPT to get this right but we'll do this for the uh start time we can probably figure it out by going to our log here and in our time stamps I kind of wish that we had done that uh the beginning part here here ooh we don't have what those time stamps look like that's kind of frustrating um but we can go over to here right no we can't because we have to take this time stab and convert this over so what I'll have to do is make a tiny tiny tiny tiny little script here like super tiny and this will just be we'll go into here and we'll just grab um these two lines okay so I'm going to just grab this here and I'm just going to say I don't even want to make a script for this I just want to I mean I do I just don't want to make a file for the script so go ahead and say copy and we'll say paste here I'll say task range do and then we'll just say file read or is it like read lines say read lines I think we can do that read lines and then we'll provide this here log file path this will read all the lines and then I want to say line first lines last require Json here and then we'll go down here and then we'll say json.parse so be Json first Json last and then we'll bring this here for a second this will be Json first and we'll say 2 I * 1000 so will be uh time stamp first and then we'll grab this this will be the last and then hopefully that just works it'll say puts time stamp first just want to do match this a bit here this is start time so let's say start time just so I don't get them mixed up end time and then hopefully that just works first go that'd be really nice so I'll go ahead and do task uh or sorry um bundle exact rake range and so we now we have those here so I'll just copy these here and go ahead and paste this in of course you will not have the same times as me so you have to run this for it to work okay can't just take my code and and hope that it's going to work perfectly and so we'll go here and then I'll just put this around here I just noticed this is called Rd not MD we'll fix that right now there we go this one doesn't need that on the end and we'll go ahead and copy this and we'll try to see if this works and there we go so we're getting uh filter data back it's not easy to work with because it's in Json format but at least we're seeing it we could uh probably filter this out so if we did and we said uh let's say query events Square braces and then we'll just say message and then um I mean this would be a little bit nicer so we'll go ahead and copy this like this like that uh so this query must not be correct oh it's lowercase events okay lower case events like that okay so we'll just try that again clear hit enter okay and so not the nicest looking thing I I'm not sure if text would look better so say text on this output text then we'll just paste that again copy paste it's not letting me paste I'll go to another tab I guess yeah so I guess I mean it is Json so that kind of makes sense but it's uh not the not the nicest display um but anyway uh I just want to go back and fix this script really quickly because uh we might use this again in the future or people that are encountering it I want to actually not run into problems so what this is supposed to do it's supposed to parse out like it's going through each line and it's supposed to then um extract the line from it which I'm really surprised that it didn't do that now we did use chat GPT to generate this out so let's see if we can find it and so in here it was using the dollar sign one because this regular expression is selecting it so what it's doing here see the square braces if we go over to here uh the this time is encapsulated in the Square so it's saying grab everything between the square and that's the date and then we just put dollar sign one here so that might just be a really easy fix but um this might not work if it's not in the correct order so I'm going to go back into uh the logs basic here well I refreshed my screen or something so it took me or I'm in another tab that's why so I'll just go this tab does not work so we'll just ignore it so we'll go back into cloudwatch Cloud watch this is up we'll give this a refresh here it's acting really funny when things act funny just give them a refresh and so we'll wait for this to open here there we go and we'll go into basic here or logs basic and I'm going to go ahead and run this rake file so this will be bundle exec rake log um to generate out the log so go ahead and delete this well this one's not the problem right the dates are fine here it's it's just this put logs so actually that's all we have to test we don't have to regenerate anything out so um I'm just going to go to my rake file this one's called uh let me just make sure this is basic again I'll do bundle exec because I'm expecting to say like it has to be in the correct order yeah so I think that yeah we have that issue so we'll just have to go back to this one here our other one and then just grab our um it's in the generation that's why we wanted to regenerate this out so this uh this time stamp method is no good so we'll go back to this one here and we'll go to here and we'll just copy this one here copy and then we'll go back to this one here that time stamp here okay so that should fix our issue here all right and um so we'll have to regenerate this one out and it's I'm missing one piece of code I we forgot to bring over this line here okay so that line is now generated and so the idea is that we need to get those logs uh up there so we'll go ahead and do our bundle ex R rake log and so now we have less of an issue we'll go over to this here and we click into here no no that's not it this one here this one and boom okay so now we actually have real working EX examples that are proper sometimes that happens you know you guys just got to watch out for stuff here I won't always remember to tell you to delete everything so just be thorough and make sure you not causing any spending your account um and they weren't even setting the expire on those we'd have to adjust our scripts to do that just say uh update uh or this will just be more Cloud watch log examples I didn't expect this video to be so long but you know this just the work that that we put in here but you know if we need to do more examples in cloudwatch this is going to be great I'll see you in the next one okay ciao all right so we're on to Cloud metric and this represents a Time ordered set of data points and it's a variable that is monitored over time so cloudwatch comes with many predefined metrics that are generally namespaced by adus service so uh specifically for an ec2 instance we already have some metrics like CPU utilization disk read Ops disk write Ops disk read bytes disk write bytes Network in network out Network packets in um and there could be even more but let's just take a look here and network packets out but let's just take a look at where this would go so if you were to look in your metrics you could choose based on the category and so that's where we went to ec2 we would choose Network in there and then we could see that information over time for network in on a particular resource so there you [Music] go hey this is Andrew Brown this video we're going to take a look at metric filters so metric filters allows us to create a filter um on top of our cloudwatch log and we saw that button earlier but we just never pressed it I don't even think I I I I called out to it so what we'll do is we'll open up our ad examples repo um so I'm going to open this up in GitHub as I normally do again get get pod sorry get pod use whatever you want code spaces whatever but understand that you'll have to configure your environment I'm going to get logged in here into AWS which apparently I'm not logged in so just give me a moment all right so I'm logged in here and let's make our way over to cloudwatch uh and so we'll have to generate out some logs um we we're lucky that we fixed our basic script from before uh if you're for that lab there um so we're going to go and do CD into uh cloudwatch um cloudwatch where is it cloudwatch logs basic here and we need to bring that data there so if we want the latest data we should generate out every single time so I'm just going to remove the old web log file here and we're going to run our python script that will generate out our elf logs so we'll go down below to our read me here I'm going to go ahead and copy this I'm going to go ahead and paste this in say allow and hit enter so now I have an upto-date uh web server logs and we'll go to our rake file that uh is the one that is responsible for pushing it that one looks fine so let's go ahead and uh run that so we'll say bundle exec we'll do bundle install first because um we'll have to do that for Ruby and then we'll do bundle bundle exact uh rake log and so that's going to run that task and so now if we make our way over to our logs in our log group we should see somewhere in here that so I'll type in for SL examples and um we'll go into this one and they have a button here that says I think we have to click into it and we don't see yet but if we were to filter something out here so we learned the other day that no it did just work when we did login but let's wanted to filter login and home I think it was like percentage login pipe home percentage enter and so now it's filtering for those two things and so we could create a metric filter and we'll just call it uh home or you know we could just do login to make it a bit easier we'll just do login here and notice that some are delete some are put some are get so maybe what we' want to do is just make sure it's consistent and oh we'll be fine we'll be for all of them it's totally fine we'll hit create metric because I was thinking our our data is random so it doesn't logically make any sense right there wouldn't be a delete put and get on login but that's fine so it just be login attempts as our filter filter name and then below we have metric name to identify this thing I'm going to see if I can name it the same thing metric value that is published to the metric name when the filter match occur so valid metric value are floating points numbers are stuff so what what do we want to here we'll have about one for each one that's there it we'll get a value one the default value will be zero um and then this is the unit and so we're going to go down with count because it's just counting stuff actually that's we just want regular count right so that should be sufficient oh we need a namespace and so our namespace here we'll call this um cloudwatch app okay we'll go ahead and create that and so now we have ourselves a metric okay so if we go back to all metrics here right I imagine that it would show up somewhere in here because that's what a metric is so where would our metric show up let me go find it okay all right so I still can't find it in the metrics I it could be that it might take time for data to come in and then for it of us to detect it but I was expecting when we go to metrics here that uh there's a lot of ways that we can look at our metrics but if we were to um this says cloudwatch metrics Insight so that's the data source here but I was expecting that if we will go here we should see a nam space uh for our own or be able to search and say login because that's what we called it and find it here so it maybe it doesn't have data yet there's many reasons as to why this could be happening um but at the very least what we can do is try to set up an alarm because if we go to um our metric here all right if we go over to here um we do have the filter here and we can apply an alarm so let's go ahead and create an alarm because maybe we can get it to trigger um and so that's what I want to find out so right now we'll leave it as static and we want a number here so we'll say when it goes over 20 right and we want to sum it up within a f minute period so I'm going to go ahead and hit next [Music] and alarm state so the metric or expression is outside the threshold then it's an alarm yep okay and we can cause a notification I don't really care I just want to see that it gets triggered so I got to ignore all these other options down below here oh it's making a select one here uh I thought we could just skip this here create a new topic sure fine um I don't know why I thought we like if we had another action we could just oh you know what maybe we can just remove this one there we go okay next yeah sorry I'm just used to doing it pro programmatically and I guess we could have done it programmatically but uh whatever so we'll call this login attempts exceed uh I can't remember we said 20 we'll say next and so here we have it we'll go ahead and create this alarm and so we should be able to monitor our alarms here it is right it says insufficient data which is totally fine but let's go ahead and import more data so what I'm going to do I'm going to modify our script because I want to have way more data than we have here and I'm going to change our script so that instead of 100 we can specify I think there's like 100 in here somewhere in here it says how much it's generating out oh sorry it's in our python file here so in here there's the number 100 yeah and I want to be able to say how much I want to generate out right so I'm going to go to the top here and I'm going just say uh number of entries and say 500 right and then I go down below here and we'll say five oh well number of entries while we're here I'm going to go update my other one for my events because it seems like a good idea to do that I would like to bring in bring this in as a parameter um I would really pre prefer that so we'll just look up um what is it called Nars python so I don't necessarily remember thank you free Camp fre Camp's always very helpful for this stuff and so that's probably something we're looking for here so go to here and this is going to be number of entries I don't know if we can do this in Python I'm going to try it anyway that's like a python uh or that's a ruby syntax going to say um python um or operator single line assignment variable it's a ruby thing I just I have a feeling that you can probably do it uh let's see here assign if blank if value not assigned python yeah I'm GNA ask chat GPT I just wanted to know sorry just I'm a programmer I like to think so I was like uh using Ruby I can do this hello [Music] equals uh bar like uh string value or hello world how can I do something similar in Python that's the great thing about like if you know Ruby really well you can translate it over to other languages okay so we're just using an or command all right so pretty close again I'm not sure if that's true they know I just faked it and didn't actually think about the language but let's try here or yep okay there we go um and so I'm going to set this back to a more realistic number like 100 and so the idea is that we'll want to supply that when we call this so what I'll do is go over to our readme file here and I'm going to adjust this so that it has uh this I'm going to say I'll call this like n entries so we can say like how many we want and then the idea is we'll just say you know like 200 or whatever I'm going to put 10 here like a l number here just for the moment and I'm going to go ahead and try this out so I'm going to go ahead and delete this delete permit copy this and paste this into enter oh we have to require it that's fine require or import OS we'll try this again [Music] and it still has a problem here and entries and entries I want to make sure that it's working so I'm going to go ahead and say just print this here I just want to make sure that it is pulling in that environment variable okay try that again and I'm not sure if it's working at all so print again hello world try this again here no it's a syntax here so it's just not going to run at all so we'll go down to here first it says number of entries I mean that should be fine right number of entries number of entries what's the problem string cannot be interpreted as an integer okay so this here is probably because it's thinking that's what it is so uh string to integer python because I don't know what it is int okay great so we'll go over to here and we'll wrap this in an INT I mean there's still issues that things could crop up here but this should be sufficient enough there we go and we will try this again and so I'm hoping that it does 10 I'm not sure what happen if this return zero because if this was blank it might still uh trigger so this code might still suck but I don't care so I have 10 here now my question is if I change this will it just overwrite the file every time I don't have to delete it yeah it does if I go ahead and try this again we'll do one okay that's good so what I'm thinking here now is that I want to um keep generating out data so we'll go back over to here this one is fine yeah I don't think we're going to be worried about that Jason one we'll leave it alone so what I'm going to do here is I'm going to go ahead and uh we'll do 100 and then after that what I want to do is I want to do um bundle exact rake log so we'll generate 100 logs and then we'll send them so I'm going to hit enter UND do that great I'm going to wait a few seconds here just hanging out hit do that again okay I'm G to wait a little bit do that again I'll wait a little bit and I'll do that again okay so what I'm trying to do is just populate enough data okay and we're doing that because we need enough data so that we trigger that but we might still have to wait a little bit so I'm going to go back over to here I'm going to go to our alarms oh it says we're alarms did it work this is Target tracking table low events uh this is for a Dynamo DB table do I have a Dynamo DB table I forgot to delete sometimes when you're alarms you might see this stuff and you go here no so sometimes Dynamo DB doesn't delete the alarms and it's really annoying so I'll just get rid of these because I don't want to pay for these but what I'm looking for we'll go back to all alarms is that we're waiting for this to become sufficient so we'll have to wait here for a bit so I'm going to wait here and we'll be back in a bit okay all right so it's been uh easily over five minutes I believe I'm just going to check my time here it's been so it's 21:14 and this is 49 so about 15 minutes so it should have had enough time um for something to happen here and so what I'm think is happening here is that it's just going to take time for this to propagate because it's not showing up anywhere right and oh now it's here okay so clearly there's a bit of a delay right but we'll click into here and so now we have our metrics with no Dimensions all right and we can click into this and I don't uh I mean at least there's data we know that it's it's been detected but I'm not seeing anything in the graph and so this is the last three hours and I'm thinking that it just takes time for data to show up so I think what we should do is we should go back over to here and we should give this a bit more data so I'm going to go ahead and hit enter and keep doing this I'm just going to confirm that we are sending data over to it so we'll go back over to our logs here and I'm in systems manager I'm not sure why I must have opened it by accident and we'll go back over to our logs and we'll go to examples here and we'll open this and so I believe that we're creating more I'm going to go ahead and try this again okay I'm refresh this one two three four five six seven eight nine does it tell us how many we have selected no uh but we'll just keep doing this a little bit more because I think what's going to happen is that it is going to eventually work but I think there's just a delay at least with these user metrics because people are saying online you might have to wait days um and I suppose I could come back here uh to this video days later and see what happens but uh right now I'm just going to run this for a bit okay and I'm going to go back over to our metri and notice we still don't have anything in here uh one thing we can do to test this again it's just because I can't tell uh until the data comes in and so it's like am I doing it wrong or is the or is my data wrong and that's always that's always a concern but what we can try to do here is if we go over to I believe milter uh metric filter and we click into it we can do a test pattern here so I can go ahead and grab my log data here so we just grab there's a lot of login so I'm sure we're triggering this uh and then I'll just go ahead and paste this in as such and we test the pattern it shows that it's counting events so it clearly is is getting numbers and so that is not our issue here so I think we're just going to have to patiently wait and I have no idea how long it's going to take I personally am probably going to wait hours and check uh a few hours from now and I'm going to just go ahead and do other labs but we're going to assume that we know how to create a metric filter that we can use in a metrics if I don't make another video then maybe that just means that I never was able to see the data but I will try to come back here and then Stitch the videos together so that we can uh see the result okay so I'll see you uh if this works okay all right so it's been a few days uh and I want to take a look at our uh metric filters because I wanted to see if uh it will show up somewhere if we go back to our alarms let take a look here uh and we go to all of our alarms arms and right now I'm in North Virginia I think we were in C Center one last time so let's take a look here it's been multiple day so I'm a little bit confused as to where I was last and so it still says insufficient data so I'm not exactly sure what's going on with this uh this is three hours ago so we could go one week okay there we go so we were getting information and so it looks like uh this does actually work did we ever get close to our alarm no we did not but at least it did track that information which was really important let's go over to our metrics and let's go into here uh metric with no dimensions and we will checkbox this and we'll go back to one week and so we have those records so for whatever reason it takes time for that to show up when is a realistic time to uh see that information I don't really know um but at the very least uh we can see that information so I go here doesn't show it with the bars probably be better if we kind of uh zoomed in on that stuff a bit better but anyway obviously there was a uh data coming in and so we'll consider that as uh successful okay so um we can keep the alarm around or delete it I'll probably go ahead and delete the alarms because alarms do cost money if we keep in the round so we'll go ahead I'll just go and delete this okay and if you want you can clean up those log groups I don't really care in particular so I'm just going to keep them around but if you want you can delete the log groups and uh um or make sure that you set your data to expire so just remember that wherever that was I can't remember was basic app or not did we set it up in this one again it's been like a multiple days so I don't remember if we go to metric filter so we have it here yeah I guess I'll just clean it up might as well do that let's delete this one out um and I guess I can go ahead and delete these logs as well there we go so I guess I am cleaning up but I'll see you in the next one okay ciao [Music] all right so with cloudwatch uh for different Services it will emit data uh to Cloud watch and the availability of the data is going to vary based on the service okay and so just looking at an example here with ec2 and other services so for ec2 um if you spin up an ec2 instance it's going to uh it's going to uh make the data available to you every 5 minutes and if you put detailed monitor on it's going to be 1 minute but for all the other services uh it's going to be generally 1 minute but sometimes it can be three or five but I really want to point this out because it's really really important on the exams that you understand that ec2 is 5 minutes uh and then with detailed monitor it's one and all their services are one by default so it's just kind of one of those little um weird gchas so I just really wanted to point that out to you so we keep on talking about the cloudwatch agent but how do we actually install it onto our ec2 instances so to install it we can just use adus simples systems manager also just now called systems manager or abbreviated to SSM and in there they have a service called run command uh and this what you can do is you press a a button that says run command and then they have all these packages and the one you're looking for is called adus configure adus package so what does this package do it installs or uninstalls a distribut a distributor package so you can install the latest version default version etc etc packages provided by adus such as the cloud watch agent and some other things are there so this is the one that you use it's confusing because if you go into um systems manager run command there's actually one named cloudwatch agent but that's actually just for restarting uh the agent but this one's the one that is used for installing the actual agent okay and so what you'll do is you'll choose it you'll have some parameters so you'll have to put install and then in the name you'll have to put Amazon cloudwatch agent because this package could be used to install a variety of different AWS packages and you'll want to use the latest one and then you need to just choose what ec2 instance you want to install it on so you can manually choose it or if it has a tag or if it's in a resource Group uh and one other thing I have to point out is that you must attach the cloudwatch agent Service uh server Ro um as an IMR uh to your ec2 instance profile so that when the agent's running it can actually go and uh report back to cloudwatch okay so there you go so now let's take a look at cloudwatch agent and the host level metrics so uh some metrics you might think you are tracking by default for your E2 instance are not and requires you to install the cloud watch agent okay uh and this caught me off guard because when I spended up an E2 server uh and then I was running out of storage space I thought I would have that data but I did not have that available to me and I'll show you what I mean so with host level metrics and this is what you get by default these are uh if you don't install the agent at all so um cloudwatch uh cloudwatch metric you're going to get is CPU usage Network usage disk usage it it'll do status checks okay so the underlying hypervisor status and the underlying ec2 instance status so you know you see those two checks for an ec2 instance that's what we're talking about but agent level metrics this is when you actually install the cloudwatch agent you're now going to get memory utilization disk swap utilization disk space utilization page file utilization uh and log collection okay um and so the cloudwatch agent is able to collect a various amount of long so if you had applications logs and uh other things that you installed on your server you need the cloudwatch agent to do it but I just really want to emphasize uh on the agent levels the memory utilization and dis space so like if you're wondering if you're running out of memory or you're running out of disc space you need the cloudwatch agent okay so there you go so now let's take a look at custom metrics and high resolution metrics so you can publish your own custom metric using the a CLI or SDK so you do ads cloudwatch put hyphen metric hyphen data uh you provide the the metric name the name space the unit the value and some Dimensions which are basically variables you're passing along if you're doing the CIS Ops the devops you definitely want to know how to do this um for high resolution metrics uh you can only switch to high resolution metrics when you are publishing a custom metric uh so the standard resolution is 1 minute but if you want it to report even more frequently so under a minute uh down to a second you can use use high resolution with high resolution you can track in intervals of 1 second uh 5 Seconds 10 seconds 30 seconds or multiples of 60 seconds okay so there you go so we just said that the cloudwatch agent can collect logs and let's look at that in a bit more detail here so the cloudwatch agent can send logs running ec2 instance to a cloudwatch log group uh and to send logs what you're going to have to do is configure the agent um to include the logs that you want to send over the cloudwatch agent service needs to be restarted the agent's configuration file is located here so if you're looking for where it is uh that's where it's going to be located and what you'll do is you're going to go ahead and edit it so you can specify the location of the log file uh and the log group that you want it to be sent to and this is how you restart it so that's how you do log collection you you'll probably want to do that on your server [Music] hey this is Andre Brown in this video I want to take a look at installing the cloudwatch agent so the cloudwatch agent is is what is utilized in order to install uh or uh to collect logs on a server and it's the preferable way to do it so obviously we've been writing our own script um in previous labs to send data but that's not really what You' want to do if you have a web server that is outputting data you would rather send those logs uh there so um I guess that's what we're going to do here so I'm thinking that we probably have to build a very simple app that does logging and we'll want to deploy it so I'm trying to think of something that I know how to deploy that is very easy for me to do um and um I want to use rails and the reason I want to use Ruby on Rails is because it has a very distinct way of logging and that's going to be very useful for us so it will take a little bit of time not maybe not that much time but it will take us a little bit time to get set up here I'm going to make a new one folder here called agent and um we're going to start building something out here so we want to have an app so I'm just thinking about this so I want to build a rails app I don't want to build a big deployment pipeline I just want to build the app get it on the server and then um have the agent installed and then uh hit end points to the server okay so what we'll do is we'll CD into our agent directory here all right uh go up a layer here agent agent logs agent there we go and I'm going to generate a new rails app so again you have to have Ruby installed for this and if you're on uh um uh G pod or code spaces this makes it a lot easier if you're on your local machine you're going to have a hard time following I'm just going to be honest with you um but uh you can also just grab the code from here so if you can't code it here and you don't want to code it you could just grab the code and put it on the server but it shouldn't take us too long to to build an app here so to use Ruby we need to have rails installed so I'm going to make a new file here called readme.md and we're going to keep make the simplest app ever so I'm going to go ahead and just say uh install rails so we'll have to do gem install rails and there's a couple things I want to do I want to do rails API only um SQL light like like SQL light database or no database can we do it with no database here we'll do SQL light database we'll do SQL light database no we'll do no no database sorry no database again really picky here right um so here yeah we can skip active record so that's the first thing I want to do and I I want this to be API only so I'm just going to say rails new configuration Flags okay we're just going to tear out a lot of stuff in here so that we don't have to worry about configuration so there should be a bunch of options here I'm just trying to find them so like hyphen hyphen skip rails new flag command line here we go and so in this one it's going to tell us so if we go rails new this is what we're looking for and rail shouldn't scare you because it's the it's uh most Frameworks are based off of it but we have skip get and actually we do want to skip G in this case because we're just going to do this locally okay so I don't want to get uh a get repo here uh skip Docker file I mean I don't care if it creates a Docker file if it does that's kind of cool but um I I want to I don't want to install it in a container on the server because I want uh to it to leverage the cloudwatch agent which is on the virtual machine I want to do API only so there should be something for API in here API and so I'm just looking for it here skip I don't want jbuilder I don't want hot wire there's like a rail there's like an API only mode let me see rails API only here yeah it's weird that they don't show that flag in here when that's such an important flag and so that's going to leave out a bunch of the stuff like jbuilder and stuff like that because that's usually the pain uh the pain the butt Part to configure and so I think we just need those few Flags so we don't have a database which is fine we don't have git which is fine and we don't have a front end we just want to have it API and so that's should generate set out app we need to name our app um I'm trying to think of an okay app name to to do because I'm just trying to think of like the nature of this app and this one will be for I don't know uh Shogun sorry I been watching the TV show Shogun so I don't know I don't know what else to name this here okay so good show by the way it's a h FX Max show really good Nine episodes on four fourway in there if you got Disney plus you can watch it on there you got to change your parental controls it's not a show for kids but if you want a really good show I like it anyway so that will be what we do here and hopefully this works okay so that's going to build out a rail app and so we don't have to worry about a front and looks like there's a lot of stuff in here we do have a Docker file but we don't plan to use it but it's cool that they give us one off the bat um so I didn't know that they had that that's uh used to have to make those uh back in the day but that's really cool that they have that and I want to see what it's installing so even though we we're not using a database it's installing SQL light uh it has boot snap in here so the only thing I don't want in here is this one here because we're not going to use active record okay so I'm going to go ahead and CD into that directory so we'll say Shogun and it's trying to change the default Ruby version which is fineable to bundle install here okay and this one's very particular about what it wants I'm actually going to comment out the Ruby version because if we have the not exact version of Ruby it's going to ask us to install it and so I just want to get a version that might work on um ec2 and so ec2 Amazon Linux uh we'll probably have a Ruby version that is easy to install it's probably a version three and so as long as it's three compatible that should be sufficient okay so I'm doing a bundle install here I'm not sure why it's taking so darn long um because it seems like it did it again but I'm going to do bundle install again so that it knows in the um the gem lock file that we're not locking to a specific version of Ruby okay and so as long as we're on version three we should be fine so Puma is the server rails is this thing over here we have uh uh TZ info data boot snap and debug and this is fine um and so what I'm going to do is I'm going to go ahead and uh test our application so we have our app but we need to create an endpoint so if we go into controllers here we have this application controller and we could create under control other controllers here but I feel like we could probably achieve most of the things we want in here so I'm just trying to think about this for a second because I don't need anything complicated I'm just going to go ahead and say um I'm going to go ahead and let's create some end points so I'm trying to think I G to go back like Shogun characters I don't know I called it Shogun so I'm going to go ahead and ask gbt I am uh creating an example rails app I want to create end uh end points uh in the application controller based on uh uh character names from the TV show Shogun the fs FX FX show please give me a uh file okay let's see if it can do that for me because if we call them it will show that we made an htb call to that and so that should be sufficient for it so I'm going to give it a moment to generate out because I've asked something really bizarre and it's getting a bit confused I don't need that I just want the that is not what I told it to do okay you shut up shut up shut up shut up okay give me a list of characters from the TV show shun FX just a plain bull plain list okay it totally did not want it okay there we go so now we have some names I got to go back over to here I'm going to paste in our names here and I'm going to go down here and change these all to so these are now functions I'm going to lowercase all this because they have to be gu to go lowercase in Vim underscore underscore underscore underscore underscore there we go and so we have these end points all right so that is good and and I'm going to split this here and there's a routes file here and this is so you know where the routes are so we'll go config routes. RB here it is and I'm going to make some routes so I'm going to copy these again this deaf here is doing nothing and this indentation is wrong again we won't take very long to build this app we're almost done basically and I'm going to go ahead and just take out uh this because I need them all back as a single line so I should have copied them earlier but I didn't and the idea is that these are going to be um uh commands these are now end points that we can hit and what I want to do here is just say I want to point this to our application and then it'll be the name of it this just like the Shand how it works so I'm going to go ahead and copy this like this okay and uh this will be get this will be put this will be post this will be delete this will be well we won't do uh post there we'll just say post we'll say get get put post get get post I'm just giving it some variety here okay and then over here we have something we could have a response I suppose and I'm just going to make the response uh so we'll say render Json and I'm just going to give back its name so just say name this will be John Blackthorne okay this will be Lord toranaga really good show by the way really good show and this will be Maro this one will be father Alvito this one will be lady fujo this one will be Amy I actually don't know which character Amy is we have aido ishido ishido sorry ishido for those who know how to speak Japanese I apologize we'll have yabo bonaro and and Kiku okay so these are all responding back with Json okay and so I want [Music] to test this app here so this is enough for this app to work okay and to run it we'll do a rails you should do bundle EXA in front of everything so say bundle EXA rails s and that should start up the application I'm going to go back to our read me here our not the one in the rails app but the one that is um this one here okay so here it's like start app so we'll just CD into Shogun we'll say ra bundle exec rails s all right and so the idea is that if we hit those end points I'll make a new tab here this will uh call endpoints so this runs on Local Host so if we do curl Local Host uh 3000 for SL uh Maro uh something should happen here we got an error here saying SQL light adapter is missing a gem so it's complaining about that but I don't want to have any kind of adapter so I'm going to go here because I'm trying to do this without any dependencies so we don't have a really hard time here and I'm going to go into our database. yaml file this is what it's complaining about because this is configured for this so I'm going to see if I can just comment this out I'm not sure if you're allowed to do that but I'm going to try this anyway and I'm just going to say no database because we don't even have active records so there's no database connection anyway so I'm doing contrl C to kill the server I got to go back to this tab contrl C we're hitting up okay I'm going to go back here and hopefully this doesn't complain and it still complains Puma caught this error database is not configured development error so I'm going to ask this how can I configure this I don't want a database how can I exclude in rails CU then we have to figure out how to install es light it's not that hard to do but um just remove it maybe we just need to remove it okay so let's go ahead and see if we can get just get rid of this here delete it and I'll try this again I didn't think that that would happen but it's not the normal way to run rails with database but we just want again afford the outputs okay so yeah you're full of baloney okay so remove or comment out the database we did that modify the application RB and Skip active record it shouldn't be loading active record so if we go down to uh active record here or sorry application.rb which uh is under our config I believe and