Build Apache Nifi using Serverless Cloud Infrastructure | DataHour by Vinod Boga

Key Takeaways

I'll just give you a quick background about myself I think since already we have already seen that in the registration page but I just wanted to quickly run through about my bio uh I'm vinod Boga I'm currently working as a lead data engineer at Collinson group I have completed my bachelor of Engineering in computers from Mumbai University from shine Anchorage engineering College overall I have 15 years of experience in the field of data predominantly I got to work on various facets like data warehouse data Mart data Lake Big Data Technologies and Cloud infrastructure building during this journey I man I got an opportunity to work on various tools and Technologies right from python SQL to the Big Data suit of things in all the Apache nifi High Atlas in all tricks while this technical skills are available with me I was able to work on these domains like Telecom dth Insurance loyalty and fmcg Airline and hospitality uh in today's session um we'll be focusing more on a bit of practical based uh I'll try to give as much information as possible which are more important as well but uh since we only have one hour of time uh I'll be sharing a lot of references documentations uh with you guys at the end of the session where you guys can go through those documentation to have a bit more in-depth knowledge but the session that we're gonna have it today will be focusing more on the important points on li-fi as a data engineer there are three major aspects that we need to focus first is where you need you should be able to build infrastructure for your application or a software that you want to work on secondly when you one should once you build your infrastructure and deploy that you should be able to provide your production support and troubleshoot all your issues which are ongoing and thirdly all the development work that can happen on that software you should be capable enough to do that so these are the three major areas that as a data engineer will normally Focus uh otherwise as a developer you will be focusing more on the third side of things which is only the development of the workflows but as a data engineer we'll be focusing on all these three facets right from your infrastructure build production support and troubleshooting and your design of the application as well okay so that would be the motive for me to share the knowledge on these three aspects and as we go along uh I'll show you uh the application Live how we can build how we can use all the necessary components of it and you know we will see all this is in this in today's session actually on the agenda side of things I'll try to give you a short overview on Apache 95 then where does it fit into the entire data ecosystem uh its architecture and major components the co-components of nifi uh the features of nifi and a quick run through of the application and its component how we can use where does it reside and so on and so forth uh we'll also do the installation of Apache 95 on our local and see how it can be done and an architecture diagram of Apache 95 from an industry standpoint and if time permits I will also show you some demo of building it in AWS cloud and then I'm gonna take up come some of the Q a sessions of your site as well so I will take up that uh to start off with on the overview of Apache 95 the nifi stands for Niagara files software which was first built by uh us uh from National Security Agency and it was then handed over to the Apache Foundation where Apache 95 was later on came into the hot in Works distribution setup distribution setup has been taken over by Cloudera hence I think it's kind of a no more uh open so I would say a free version but since we have the docker files available on Apache 95 we should be able to build that on a local and at the same time we can use it from an industry standpoint as well so what we're going to see today is using the docker image of Apache nifi and we're going to build that on our local and that's how we're gonna look into that now there are two approaches on nifi side you can install that on the Standalone machine or you can build your knifi as a cluster as well but what we're going to focus today would be more on your Standalone machine how we can deploy that how we can run our application and so on okay uh to start off with uh knifi is an open source software and it has given multiple names to it people call it as data logistic platform we can call it as a data orchestration platform we can call it as a data ingestion platform you can name it as an ETL tool as well but the major aspect of having nifi is to have the automation movements of data between its the separate systems when I say this separate system means systems which are not in common you can have multiple systems which are in different shape in size you can connect to those systems and then bring that data back into the destination of your own choice so that's the reason Apache 95 has its more Advantage compared to other ETL tools or a orchestration tools which we have now it is very easy to use powerful and it's a web-based user interface application where there is no coding required at all at the moment a lot of people will feel that you know do we need any coding requirement but to work on nifi it's a GUI based and we can drag and drop and create our flow so that is where the ease of use on Wi-Fi comes into the picture but at the same time there are some integrities of information that we need to look into and you know how can we build how can we mitigate certain issues while it can come okay so that we're gonna look into it it's a data source agnostic which means you can connect to any Source system which can generate your data and data of your interest you can plug into those systems and bring the data into the destination of your own choice you can push into high Edge base H2 database and so on and so forth uh one of the biggest advantage of knights is gives you the real-time scenario to update your workflows normally what happens is when you develop a pipeline in say ETL tool or any of the ETL processes whenever you deploy and when you start your applications uh you can only see some errors and it stops in between and all of a sudden you need to go back and look at that application and then start it all over again but with nifi it gives you a real-time feel you can modify things on the go you can stop your pipeline in between Rectify certain things and then start your process from there on that is where Apache 95 comes in very handy in terms of updating your workflows in a real time we're gonna see that as well as part of our practice today it is very highly configurable and modifiable as I said because of its real-time nature you can configure more decide on the fly at runtime as well one of the major aspect of nifi is the flow file now four five flow file sits at the heart of nifi it's as good as say when you play a game of football without the ball without the football you can't play the game at the same time in knife fight without the flow file there is nothing inside the Apache 95 so everything that we're gonna see today in Apache 95 will be based around the concepts of flow file so a flow file is nothing but a packet of information which travels from your source to destination take an example that is the source you have one lakh records you break that into a chunk of smaller size of say 5000 records so that profile gets split into that 5000 records and Travis is from source to your destination and that is how the flow file comes into the consideration now if it doesn't take into consideration all the records at one group the best practice of working with nifi is to break your big chunk of information into smaller packets of flow file and then transfer from your source to destination so that is what you need to consider like the major aspect in UniFi is for file based and lot of in you will see this word coming across in lot of our practice today and as we go along as well this was a bit about nifi next is like to see that where does it fit into our data ecosystem majorly essentially Apache suit Apache knife it has its place in Big Data ecosystem where it can handle lot of uh volumes as well and as I said I mean you can name it as a data orchestration tool or an ETL tool or a data ingestion tool because it does all these kind of work as part of the knife file you can build your data pipelines using nifi the major difference between your data pipeline in etls are in data pipeline there may or may not have your transformation available but in ETL you have this ET which is transformation available so you can build using nifi the pipelines data pipelines the ETL process the orchestration section and the ingestion section as well so this is where I think you can take nifi and it can fit into all your requirements of transporting your data from one place to another which is like a logistical requirement and if you have any pre-processing requirement of your Source information you can do that as well in 95 so that's where it fit into the ecosystem variable on the data transportation from one layer to the other layer so next is where we're gonna look into the architecture of it it's it's a very simple architecture but it's it's very I would say uh robust enough to handle a lot of information uh since I said that nifi is a web-based application we would need a web server over here and nifi is written in Java so it runs on a Java virtual machine on top of the operating system and the web web server is required to handle that web-based interface and all its API commands and all so that's why the web layer sits at the top the second layer is your flow controller FC means flow controller which contains all your processors and extensions I'm gonna show you what all these processors and extensions are probably better to show you that as well I hope everyone can see my screen and this is a basically a GUI of nifi which looks like the small boxes that you see inside these are all processors that we call it as and extensions are nothing but all these extensions like processors the input Port the processor groups we will see what all these are but all these connections on tops and and you know these are called as extensions and these are processes and this is your flow controller which is also called as canvas in Wi-Fi and then since I said that it is based on Flow file concept the flow file management happens in these three repositories these repositories are up most important on li-fi and as a data engineer if you know the concepts of these represents is what it holds uh and what information that it provides then I think majorly you can troubleshoot any of your technical issues that you find on nifi and we're gonna see what these are on the flow file repository flow file is nothing but it contains the metadata of your current flows and actual content of the data where you transport your data from source to destination that actual content is stored in your content repository so imagine when you have an hdfs you have a driver and executor memory the driver contains all the metadata an Executor is you do all your work that is something similar concept you have it in nice eye as well where flow file contains all the metadata informations of the flow file which are currently in process and the actual content of the flow file which is data is stored into your content repository and the flow file metadata has a pointer to those content repository to understand that these flow file content is stored in this repository okay so that's where content repository comes into the picture by far of these three repositories the biggest size is flow content repository because of the actual results because it stores the content hence the size of this repository will be always higher the third repository is nothing but your Province repository because it holds all your data lineage of the flow file so the flow file when it moves from One processor to another processor and as it goes on uh now if I gives you that lineage information that how the flow file is processing through who is consuming it and how and at which stage what changes are happening to that profile that lineage information is stored in this province Repository and these are the five major core components of nifi one is your flow file processor processor group flow file controller and the connections probably to understand these Concepts I'm gonna move on to the uh application side of things just to say that you know whatever we just seen on the slide how it looks like on the application side of things and as I said these are all your processors which does the actual work when we go back to this canvas and this is your processor group processor group is nothing but where it encapsulate or groups all your processor into one Hood it's called as kind of you can see it as a project like my project is Excel to CSV I'm trying to convert an Excel file to a CSV and I'm going to create a processor group inside a processor groups I'm going to have all my processors processors are nothing but your actual uh piece which does all your back end work it's a black box for us at the front but these are all your um connections which basically does your work at the back end so these are processes and the processes groups then if you see these connections which are going from top to bottom the success message this is nothing but your connection where once your process flow files have been generated from Top it will then hand that processor to flow file to the below processor and likewise your flow file travels from top to the bottom till here okay that is where the flow file comes into the picture and we talked about connections as well and I'll just gonna start these processors just to show you how profile gets generated if you look at there is one flow file got generated and it got stuck here the reason it got stuck here because my flow file the processor here is in stock date stop State you see this red signal this is nothing but it's it symbolizes the state of the processor whether it is stopped whether it is on so red signal means it is stop that means all your flow file will come here and get accumulated here okay likewise the movement I start start my processor you will see that this symbol this is something known as Bridge your processor are in active State these kind of symbols active State and it has processed as well and I have to look at the bottom I have four flow files came into the picture now what I did in my flow file is I have split up 11 records of Excel file into three records each which means four files will get generated one file will contain three records and my fourth file will contains the two records because 11 files that is what this flow file comes in the picture now put file processor is nothing but it will put into a destination which I want currently I want that destination to be available in opt 95 MNT okay so this is just a flow of what we have designed how the flow file travels from top to the bottom and what this flow file is all about we will look into deeper of what these are but just to show you what processors are what a processor group is and what is the connections and you know and what uh how this flow file traverses from your source to the destination okay now there are a lot of parameters that we need to configure that we will see as well so now coming to the flow file repository and profile content repository and Province I'll show you where does it reside on our server so I'm gonna log into my Docker image where currently I am on my Docker terminal so inside my container this is my container for nifi and I'm gonna go back inside and open my terminal over here this is my root path for nifi and inside that root path you see this content repository database repository flow file repository and Province repository this is where the actual information resides and these repositories are helpful in flow management flow file management actually if you look at the size I'll do any size on edge the if you look at the size of your content repository is 20 KB okay which is by far the highest of all the others okay other repository like content repositories 20K your flow file repository is only 4K your Province repository is again 4K now the reason for having this content repository is 20K is because you have this flow file stuck here and the actual content of this flow file is been stored in your content repository okay now see if I delete this empty this content Repository so I deleted your flow files and now let's look at our size of our content repository it will take some time to get refreshed I think we'll come back to this and see whether it will delete it but ideally this 20 KB will fall back to 4 KB 4 KB again where the content repository will get deleted because we have deleted the flow files from here okay and since we have looked at the nifi architecture and the components a bit in detail we'll go on to the next side now this is a bit more about your flow file what does it actually stores how the management of the flow file happens inside and you know how that this information is very helpful from a troubleshooting perspective okay now ideally a flow file what it contains is your metadata and a Content claim okay that is what we're gonna look into this I'm gonna generate this again now again you have these four flow files you're going to look into these flow files this is where you can list your flow files okay and if you look at this I section click on this you see this this flow file contains your flow file ID your file name what is the size of the flow file and the content claim content claim is nothing but it contains the information about the content repository where exactly it has stored okay inside the content repository you will see a containers and inside the containers you will have this sections okay section one two three where exactly it is stored and an offset to that content reposit using this offset information the container information the section information flow file understands that you know these are this is where the content information is stored in your content repository basically and if you look at these you can also view the content of your data okay so the 11 records that we have created out of which I have split it into three records then these are the three records so this is where it is allowing your you to do the changes in live real time okay ideally in any of the the ATL tool these kind of information or flexibility is not available but if you're seeing what we are doing we are on the Fly changing your pipeline we are on the flight trying to delete the information you can even say like you know stop here and you know you can then configure currently I said three records and all of a sudden I can make it like you know uh fire records and when you run it again you will only have like you know around three flow files generated I'll delete this again it has only generated two profiles because file records in each okay so you got this passive flexibility of changing these things on the fly on the Fly real time and web-based okay so this is one of the biggest Advantage which knife I gives you otherwise if you look at your uh any other ETL tools you may have to do this configurations at a lot of places change certain properties redeploy into but on nifi you did it on real time you've got this flexibility and you can work on this okay so this is another advantage that we spoke about and that we have seen as well over here coming to the flow file thing you have seen what metadata it stores what the content claim is content claim contains all this information a container in the sanction in which what is identifier offset and side size and this content claim has a reference to your resource claim this is an abstracted layer which we cannot see but at the back end this content claim has a reference to Resource claim which in turn has in with access to these information through which it understands that you know what kind of content is to where the content is stored using this profile information so it's like a tracking of your content information via this profile that is how these information flows through okay since flow file are actively been processed by the system are held in jvm memory okay now as I said it's a Java based program it's runs on a virtual machine virtual Java virtual machines each and every flow file is captured inside your main memory Java Heap memory and it then processes efficiently now you can think about why it efficiently processes that many bigger information of data in a big data side of things because nifi captures all this flow file in its memory at a time in smaller batches okay you can set this threshold in your nifi application where how many flow file you want to process at one go so that it processes in batches and it gives that to another processor to process likewise this cycle goes on and it starts from source to the destination one by one one by one now given its nature of repository is a right ahead log what do you mean by write ahead log means it's always appends to your repository which means it never overwrites its previous state whatever state it has gone through from say for example it has come from patch file state to convert Excel to CSV processor then these information is written back to your file repository and when it comes from this information to the split record it will again write it back to your repository so it will keep a log of all this information and tomorrow for example if your nifi crashes out you should be able to recover from that point itself and run it ahead because of this right ahead mechanism or flow file now you will say where I can see all this lineage information we talked about Province repository which shows us the you need when you come to the top right corner click here you see this data Province here click on this data Province these are all the information that you can see from a data lineage perspective okay now let's click on this convert Excel to CSV processor and see what's happening here uh actually the lineage will be can you see this three settings button show lineage click on here and it will see that you know where that flow file has come now the red button indicates that this is the state of the end state of the flow file at this location at this location so this is the flow file at which red means this is the current location of that flow file it has traversed from this area to this area and when you look at these sections again it will also show you where exactly in your flow you have designed it so see it has come back to this section so this section of preprocessing of your flow file like you know forked cloned dropped all this information has been captured in your lineage so if somebody asks you whether your data has been processed correctly or whether it has reached your destination now if I gives you 100 delivery guarantee from your source to destination it cannot drop off any information in between that's the beauty of nifi and if somebody claims that you know I didn't get these many records which was there in source and destination these lineage information will help you to trace back that you know where it got stuck and how it got stuck okay now some other information which I wanted to show is the relationship and the connection okay now what you have seen is this is the success connection what do you mean by success connection means if your processor processes your profile successfully you put that flow file to the downstream processor that means that success then give this flow file to this processor now likewise you can configure all your relationship like what about if your processor fails what you want to do currently what I did is I have terminated that relationship which means I am not expecting any failure I don't want you to put any failure because these are not important ones but imagine if I uncheck this because currently it has stopped it I will stop this and I will come back here configure here relationship and I will remove this and apply now you see there is an invalid processor symbol it has come on here relationship failure is invalid because your relationship failure is not connected to any of the component okay that means I will either have to terminate my connections or I need to give that connection to some other processor these are the thumb rule as part of your connections okay now if I say I will come here and I'll connect and I will say failure and add see it has come back again to Red which means all your connections has to either get connected to other processor or you terminate that processor terminate that relationship basically now I don't want this failure to get connected here because there is no point so I'll delete this you can directly delete because you need to stop your starting source and destination and then the relationship can be deleted okay so this is how I delete it now again I talked about if knife I crashes how are you gonna recover it back so this is the location where nifi keeps it back up on your uh container on your Docker file when we CD into this location which is your root location of nifi and a config file which is Con file when we do LSI from hell you see this flow file dot xml.gz okay this is nothing but what you have developed online on the UI this imagine your nifi goes down when your main goes down all your design and the processor design data flow which you have designed will also go down or will finish off how you're going to recover back so what a general practice is you keep a backup of this flow file somewhere periodically at whatever interval you want or whenever you have developed some critical projects you keep a backup of it or you keep a backup of it on a regular basis of a weekly monthly or a daily depending upon the criticality now the moment you store this file and when you recover your nifi services back on your server on your Standalone machine or on cluster you put that file into this location again and you restart your nifi services you will look at these flow file back into your system okay so whatever work that you have done it will come back to your system again okay so no work will be lost all your hard work will be saved back again just keeping a backup of this small file okay so this is utmost important and very helpful when it comes to your so we have seen how we can recover back from our flow file because this is very important now imagine you want to share this uh template with somebody else okay now how are we gonna do so what we need to do is you need to just right click on it and say create template okay and what happens is next time somebody wants to design something very similar or wants to customize this they don't want to they don't need to have build all this from scratch all you need to do is come to here in the template section drag and drop here and then I have just created one template called as Excel to CSV you click on it and add see all the processes came into existence again okay so you're able to recreate it again with a very less effort all you need to do come back here and do the necessary configuration of your own and and done it okay so that's how you can reuse your work uh cool that's what we have looked into we have looked into what is your uh Province repository what does it give it gives you the lineage do you looked into your content repository where it actually stores your content and we have also looked into your flow file repository which actually stores all your current flow files in the process okay now let's move on to something more there is more to this profile but better I'll go ahead and do some practicals and then if time permits I'm then going to come back and show you how you can configure certain profile parameters where you can uh actually resolve some of these issues but one quick thing is when you right click this where you can do this list and empty you can also get this option of view configurations in the view configurations you can basically set the priority of your queue of how you can run it whether you want to do it fifo first in first out first in last out your priorities of the cues can be changed over here okay back pressure object threshold this means if there are a lot of flow file to process by 95 then it can go down as well you need to ensure that the flow file management is up to Mark and this is where you can do a back pressure object threshold of say default is 10 000 but you can increase to whatever configuration that you give it to your nifi server and basis on which it can handle that mini profile in conjunction during the live session or you can also give the sizes of that profile if it reaches 1 GB of size then you apply a back pressure now what is a back pressure means it what it does is in the memory it will keep maximum hundred ten thousand profile or 1GB and the rest of the flow file it will transfer back to your repository when I say transfer back that is where it will come back to this flow files you see this flow file flow file repository it will go into this repository under a swap folder you see this slap folder this sap folder contains all the back pressure flow files which is currently not been served by nifi what happens is now if I will first give priority these to this 10 000 process it and then it swaps another batch of 10 000 from this folder and brings into the memory that's how the memory utilization of the nifi is Optimum and it should be able to handle lot mini flow files as it processes checkpoint is nothing but as I said it keeps all the information of the flow file from where it has started to where it has gone and this is utmost important in case of failure where you can start again using this checkpoint information okay so this is all the major important information which I wanted to share from a repository perspective and the flow file perspective uh will probably move on to the next slide over here we have also covered the flow file section we have seen how we can recover a flow file ah sorry what the flow file contains we have seen the content of the flow file how you can manage the profile better the memory of it and everything okay uh coming to the features of the knife and we have seen a lot of features now that is what we're going to discuss here browser-based user interface yes it is browser-based and it gives us a seamless experience of design which can be easily designed control which you're able to control on the live you can monitor things as well sorry you can monitor using the data Province as well and you can feedback as well I mean in terms of feedback where whatever you see you can come back and rectify here this is based on the logs that has been stored on the nifi coming to the data Province tracking we have seen a complete lineage of information from the data Province section that is where I think the data Province where it where we were able to see the lineage of the information then the extensive configuration we have talked about now if I guarantees us guaranteed delivery that means no information from your source to the destination while you're processing next I will be lost it has a very low latency you have seen with a click of a button we are able to process all your files profiles from your source to destination in a very quick manner very high throughput and you can also have a prioritized queuing we have just seen you can also configure your flow file priority you can set first in first out newest flow file processing first oldest profile processing first that all you can configure in your connections by doing the view configuration runtime modification of flow file configuration we have done that we have shown you how we can delete the flow file on the file modify your processor and you know have that as possible data buffering and back pressure where it can control if if your application has been flooded with lot of low file at one go you can apply this back pressure control by setting up some threshold on your number of low files or the size of the flow file basis on which it will apply your back pressure and push the uh upcoming flow files or the pending flow file processor into the swap folder of the flow file repository so that is where features comes into the picture rapid development and iterative testing we have seen that we can easily develop we can easily reuse your components we can easily configure with the click of a button and you can also have component architecture for custom processes and services over here from a security standpoint we can't look much in details over here but I can talk about it since it's a web-based interface since your UI is https connection we can attach an SSL certificate to your DNS name that you have which ensures your data in transit is secure using that SSL certificate multi-tenant authorization you can currently what we have is at nifi you can also integrate with Apache suit which is Apache Ranger where you can give control to the user saying that they can use they can look into these processes and that processors where you are ensuring your authorization is into picture you can also have this integration with any of your authentication mechanism in your company could be ldap could be SSO all that is feasible in your nifi as well okay so these are some of the features we have also seen this feature in Practical and that's where I think an if I looks quite beautiful now coming to the docker installation guys uh this is what we will now look into how you're gonna you can build this on your local before you build this local some of the prerequisite that we gonna require would be uh Docker you need to have this Docker installation done on your laptop uh if you look at this symbol this is that both symbol of Docker once you install your Docker this Docker desktops comes onto your machine which helps you to control your Docker images you can SSH into your Docker containers and you know insert certain files of interest and all that stuff okay so the primary requisite is to have this Docker installed on your laptop which is again open source you can easily install on your laptop with the Google search you will get that download file and you can download this Docker now once you've downloaded this Docker what you next need to do is I'm gonna share all this commands as well with you guys so you can go back in and do that now what you need to First do is you pull your Docker image from your public Docker repository and this is that name of that Docker image which is Apache in iPhone latest okay so once you do this Docker pull Apache Network and it is what you're gonna see is you will have this image on your local I'm gonna do Docker images on my laptop if you see this Apache 95 I have done a git Docker pull and it has come on my laptop on my machine as a knife I Apache nifi there are certain tags which you can give the current tag is for latest this is your image ID which we gonna require as we go along once you have that Docker image on your local while where we have verified using Docker images you're gonna do this Docker build hyphen T Apache 95. latest it will build your application first and when you run this Docker run hyphen hyphen name now if I and Port is 8443 the reason of having Port 8443 is because of which https connection that we need which is needs to be secure now as part of the documentation when you look at this is what they recommend to use as 8443 for https if you want to use HTTP connection which you can go for Port 80 okay uh once you fire these three commands on your laptop you're gonna get this URL you're gonna next put this URL which is nothing but https localhost Port 83 and nifi login okay I'm gonna come back here okay but since I've already opened this one I'm gonna remove this and I'm gonna come back to this session I'm gonna log out of this section so you have successfully logged out and you have come back so I have added an authentication layer as well when I say authentication layer it comes as part of your Docker image authentication only once you do this you will be able how to get the username and password okay so when you do this Docker log my file graph you should be able to get this username and password you will have this application password somewhere stored in this section since it's say this username generated and password this is the username and password which you can take on your local apply this username and password and you should be able to authenticate yourself I'll use this username I'll use this password so you should be able to come back into your login so you have authenticated your application on your local as well there is one more references which I wanted to give you guys is Docker Hub knifi so this is the Apache 95 Docker URL you can follow along this uh there are all the necessary uh commands that has been written here that's what I have taken from here and showed you how it can be done and you know uh posts running you can also configure your custom uh username and password as well so you can follow along these documents and run that and that is where you can do this installation the prerequisite for having this installation is to have your Docker and you can then run all these commands on your docker just a pull build run and Logs with this you will be able to get into your log nifi in your local now one of the other very important thing is now we talked about having your Docker as yeah as data source agnostic but what do you mean by data source agnostic which means you should be able to have all your connectors here okay since Apache 95 gives you a base image not necessary that all your connections of your Source are available here for example I have typed Hive this was not available at the Apache 95 image but this is what I wanted to have as part of my testing and environment where I wanted to use this Hive so what we did is the major thing that you need to do is all this connector that you see all this processor it's it has certain softwares with it can you look at this lipfry Library whatever you see right now on the processor these are all the Nar files which you which enables you to look at those applications processes and this okay so what we need to do is we need to actually come back to the Maven website this is a Maven website where you will get all your processors Nar files okay say for example you are looking to connect to a mongodb I have typed you look at this so there is a mongodb in our file you click on this and you can click which version of nifi you have installed and you can go back here and pull up that Nar file and push that Nar file to your Docker image container then you'll be able to see those mongodb connectors in the processor okay now since I have I have been using I wanted to have nifi what we did is I went to an i5 and I have I have installed 1.19 Apache 95 version and from there you see this big file of lotner I have taken I have copied this file on my local and what I then did is fired these Docker copy so what I did is I have stored these Nar files in my local folder which is user windows and location and here and I have given my Docker image ID this image ID we have seen here Docker images see this is the ID where you will get your Docker image and you can get this Docker ID and you push to the location of your interest since I wanted to have these narrow files at the library folder I have done Docker copy from my local to the yeah container knife I container Docker container in my local okay the moment you do this okay these files will be available here using same commands I have fired on my uh local as well and it has come and pasted these files over here can you see this my f95 these were not available which we have manually copied from my local to your Docker container via those Docker commands Okay now what we then did is came back to the container restarted this container and it will then come back if your UI will start again you will login in again and when you come back to this processor and type Hive you will see all your high processors have come into picture and then you can pull and create your profile so this is what small design have created what I have did is this is a hive select you will what I'm trying to do is I have created this in the properties I have created a connection to my hive where I have to give all my details like this is the controller service where you will create all your jdbc connections and this is where you will configure all your properties my jdbc URL my username and password and a small validation query to validate my connection is correct along with my authentication details once you set your controller service here you call in your your you connect here and you provide a select statement so this is a query that I wanted to run okay big query where I have taken and what happens is I generate this I come can I create a connection and this query gets submitted on my Apache Hive okay and I was able to connect wired so what I did taken the base knifi images I wanted to connect that Wi-Fi to Hive I've taken the Nar files from the maven websites I have copied from my local to the docker container I have restarted my service and I have just came back and dropped my processor and I have typed and I have all my connectors here okay that is how you can um connect to any of your data source of Interest by pulling up the now for example mongodb see delete get it already has these connections so you don't have to bring those narrow files but if you are looking out for some other stuff ways you know Kafka it also has as well so wherever which is not possible not available on your processors you can pull this like this and you should be able to do so as a data engineer you ensure that your hurdles you have resolved by pulling all these Nar files into the system and your application is running you have able to connect your data source of Interest as well okay so this was a typical demo that we had on the local file system that we wanted to show I hope everyone it was quite simple enough you can follow along with all these documentations and uh you can try it out on your local and you can see how it goes uh one one other issue that you may get is while you do this Docker build this one you may get an error in case if you get an error what you need to do is come back to this container from this container you can do a stop and start okay one should do this it will come into your running State and then you can push this uh this command HTTP localhost 8843 onto your browser and it will start up all these things are available on your references documentation which I've given you you can just follow along those things okay now since we have already burnt out I'll take another five more minutes uh just to show you what industry architecture that we are trying to build in our company is we have did this POC first on Hive on my local first tested how I can build all my Nar files and everything so we are trying to achieve this kind of an architecture at our company where what we are trying to do is we're gonna take that Docker image from the docker Hub we're gonna build our Docker image and push it to the ECR registry via the cicd pipeline of GitHub the ACR registry is nothing but where you can register all your Docker images now in order to cicd pipeline to push my Docker file from my GitHub to your ACR registry I need to have some IM rules which I am need to consume and it needs certain permissions so that it can login to your HCR registry authenticate itself and then post this Docker image onto your ECR registry okay but on the contrary if you want to do via console what you can do is install this AWS CLI on your laptop and Via AWS CLI command as well you can push this to your ACR registry that is something which you can practice uh let's see if time permits I'm going to show you that but that is something which you can do from ewcli perspective as well and you're going to push this to your ACR registry as well now once you have this ECR registry which means the docker images which you are able to see in your local that is something very local for an AWS that means AWS cannot read directly from your Docker images what we have done is brought that Docker images from the public repository to the AWS local which is your ECR registry where you can manage all your Docker containers okay now from your ACR registry you're gonna create a AWS forget which is nothing but a server list compute Services where you can take consume this Docker images and run on your uh on your website as well okay so in that AWS far gate what you're gonna have is you're gonna have a ECS fire gate cluster and inside that ACS formulate cluster you can create a task definition and that task definition gonna have this containers on which your application will run okay uh the this information can be a bit overwhelming but uh this is always you can go back refer to this architecture and read more about this so you will get get a get a better understanding of all these Services which I'm trying to talk about and you'll be good after the container once it runs what happens is your Docker is running and then using that IP address you can then make that application up and running on your browser and since you cannot have that IP address which is not that idle state to have what we gonna do is we're going to create an elastic load balancer where users where and to that elastic load balancer we're gonna attach your Route 53 where we're going to have a DNS name custom DNS name for your application and to that dsness name the users will hit that DLS name okay now DNS name is nothing but whatever you are trying to hit just localhost is as good as one DNS name to is and we're going to have a custom DNS need so that everyone can use the DNS name on your browser and access the nifi application okay so one application which I'm gonna show you is what it's already running at my end on the AWS stuff is I'll gonna show you that quickly meanwhile it's up so this is already up and running currently I think it has been down but I think it was probably up it will be up and running on this IP addresses I'll just quickly take you through the ECR registry I'll try to quickly run through all these things on the AWS side of things so that you are aware so what we can do is on the ECS side of things you can create this repository okay once you have this repository inside a repository say for example I have been using this Hive test demo and you can register these um Docker images let's try and push one Docker image here so I have set my awcli to the account in which I'm I want to access and I've just already set up some of my code so this is an AWS CLI command where I'm gonna do a uthentication to my ECR Repository see you have registered and logged in successfully to ECR registry and I'll do a got Docker image and I'm gonna take this hello world and try to push into our ACR registry and I'm gonna name it as hello world okay so before you push you need to tag your uh image this is what the tagging is all about we are seeing this tag that is what we're going to tag there and we're going to take those tag over here under this URI and I'm gonna push this Docker image so it says your Docker image is already exist and it still goes ahead and updates if required and if you look at these see 10th February hello world and this is where you are able to push your Docker image on your ECR which means now you can play around with this Docker image in your AWS local and this Docker image you can then consume in your ECS container okay let's quickly go through that as well so what we're going to do is we need to create one cluster over here create cluster doesn't have much steps only networking only since we're going to use fargate you need to choose networking only option and then just do a create over here just give a custom name what you want and that's it this is how you will create your cluster and to this cluster you're going to add resources later on so this is one of the cluster which I created under here I'm going to create my task definition okay say for example I'm going to use my forget option I'll do a next I'm gonna just put some random name then you need a task rule that's what we have looked into our architecture this is what the task rule is looked about the small hat of I am role is nothing but your task rule you're going to select your task rows when you're doing for the first time it will automatically generate since I've already created it this rule has been created now this role is required basically whatever your ACS required access to like in S3 bucket or EFS file system or any service it communicates to it will have the necessary permissions in this task role the network mode should be default should be okay operating system we need a Linux box same task role you can have a say I have given eight vcpus and you should have a one is two one is to 8 ratio means one CPU should minimum have 8 GB of RAM so since I have chosen say eight CB CPUs I'll say 18 to 8 which is 64 GB memory okay that's how it should be enabled sorry 8 is to 16 it's what it's called about then you can add your containers in the container you can give your container name and here you have created your uh digit so you need to take your image URI which means you're going to consume this image in your ECS okay cluster and put this image over here and you can keep all the rest of the port mapping they said 8443 okay then we can keep all the health checks environmental as it is network settings storage as well defau