AI Engineer - The next big tech role!

Key Takeaways

that's why I say that the next big Tech role is going to be of an AI engineer hey everyone Hi hasagi the site with more than $1 billion in Revenue just from startups we've seen some early signs of success in Ai and now every tech company is trying to infuse their products their customer support Bots their departments with these generative AI capabilities now ai is at the same Junction where internet was back in the late '90s but what does it mean for us the programmers the data scientists the analysts I think this is probably the best time to start building with AI why now to spot the trend I looked at why combinator's portfolio of companies and for those of you who do not know why combinator Y combinator is a prestigious uh startup accelerator based in us and they've bagged many startups which have now become Tech Giants such as Airbnb Dropbox stripe and Reddit and here's what I got look at the number of companies building with AI from 2017 to 2023 there's been a significant rise in the number of companies building with AI since 2021 and then it shot up after the the launch of Chad GPD in 2022 now this gives us some confidence that there are going to be more and more companies building with AI in the near future which as a result will lead to increased demand for AI Engineers we've seen that there is enough demand for AI Engineers every company is trying to build something with AI and the AI development space also has evolved enough with open source large language model with readily available apis now anybody can get started very quickly with that even the community has grown enough to provide you the required support whenever you get stuck but when we say that you have to get started what does it mean there are so many questions to answer the first thing is you know what all is going on in that industry so I tried to create this air development canvas of sort which captures the different layers where most of the work is being done in AI as you can see there are three main development layers on the left hand side I have tried to capture the kind of people who work within these layers and on the right hand side you see the sub field the toss the kind of companies and the tools that are being used within those lay layers the first layer the application layer now this is where applications the real products are being built on top of our large language models on top of the eii generative AI capabilities and this is the most buzzing layer real use cases real value this is where the money is most of the companies are coming up with these applications and what does it entail the kind of work work you are basically developing interfaces some would be developing chat Bots some would be developing tools to create uh and draft articles using generative AI then you are developing rag pipelines which is retrieval augmented generation pipelines we'll talk about we'll understand all these use cases in the come upcoming videos then some people are building multi-agent workflows and then there is obviously monitoring and evaluation involved at every step of the development process be it model development or application development and now let's come to the second layer which is the model development layer this is where companies like open AI Google meta are developing large language models now the kind of people who are working here AI researchers applied scientists data scientists and the kind of work that they do entails everything from data set curation data engineering to model development to inference optimization such that you know they are able to generate those tokens at a very very quick speed and again model evaluation and model benchmarking is a huge and the most complex task which is where a lot of research work is also being put in the third layer is the infrastructure layer which underpins or provides support to both model development as well as the application layer this is where compute management you know uh companies which are providing uh GPU Services uh companies like corv companies like Lambda labs these companies they provide services such that you can now rent gpus and then train your large language models pre-train them and also provide storage Services storage management data management services to you and again the cloud services providers like uh AWS Google Cloud platform they all are working within this infrastructure layer such that your models perform better they are deployed and they are serving their inferences at a very quick speed and the kind of people who are working here Cloud Architects data Architects Hardware Engineers Cloud Engineers given the highest Traction in the application layer it has now led to an increased demand for a special kind of engineer people who know how to build on top of AI while there is no standard or specific term for these Engineers most companies are calling them AI Engineers or llm engineers and while I was recording this video I came across this amazingly detailed blog post by chip hu on the 900 most popular open- Source tools where she said with readily available models anyone can develop applications on top of them this layer has seen the most action in the last two years and is still rapidly evolving this layer is also known as AI engineering so what do these AI Engineers do how do we define the role of an AI engineer so AI Engineers are basically specialized programmers who have the skills to leverage AI Technologies to build comprehensive and form agnostic applications now when I say form agnostic basically you should be able to build wide variety of application starting from you know simple chat interfaces to complex full stack applications python packages libraries or even complete sdks software development kits now unlike AI researchers who delve deep into the foundations of algorithms and model training AI Engineers are required to only apply AI Technologies to build real applications but again the question arises don't you need to be an expert in AI in order to become nii engineer and the short answer is no let me explain why just like when you start swimming you don't learn about the physics of buoyancy right and similarly when you are required to build on top of AI you don't need to learn how you know Transformer architecture actually works you simply need to know how to leverage it obviously depending upon the use case you can if it requires deep expertise uh in AI in deep learning in machine learning then that understanding could be advantageous but again depends on the problem statement that you have let's also draw a line between AI researchers and AI Engineers what is the difference between these two profiles okay in order to do that I've created this diagram where I have plotted the engineering skills against the AI Reon research skills okay on the x-axis I have the engineering skills think like you know building something using the apis and on the y-axis I have the research skills think something like you know designing model architectures creating training models uh curating data sets Okay now ai researchers research scientists data scientists these are folks who have deep expertise in AI these are the people who are actually training those models okay so they're high on this Y axis whereas I have a different Persona you know core Engineers full stack Engineers backend Engineers front-end Engineers data Engineers who do not know much about AI but they are really good Engineers so they are towards the right high on the engineering access now in order to bridge the gap between the AI research skill versus the engineering skill we need this specialized Engineers who can build on top of those AI Technologies and this is where AI Engineers actually come into the picture these are the AI Engineers who are going to build those real applications you must be wondering if AI researchers are people who have good engineering skills they have deep expertise in AI then why don't companies hire them over AI engineers and that's a fair question the answer to this is scarcity we do not have enough AI researchers or llm researchers which in turn leads to increased cost and on top of that all the top llm researchers or AI researchers are already cornered by the Giants like meta Google open AI anthropic that's why I say that the next big Tech role is going going to be of an AI engineer we need a special class of professionals who can serve as the bridge between Cutting Edge research and building practical applications on top of it to ensure wider accessibility and implementation of these AI Technologies now you might ask me what about traditional ml problems will everything be solved using generative AI so the answer is no while traditional ml problems like recommender system fraud detection and anomaly detection will continue to get better we have a whole new range of AI applications to cater to and as the co-founder of hugging phase CLM said AI is the new paradigm to build all technology thus we need more and more AI Engineers I mean look at this generative AI Market map from seoa application layer is filled with us use cases and companies in almost every domain so to put this all together we have call outs from the industry leaders deep experts in AI like Andre karpati chip huan and Clem amongst many others then big incubators like y combinator VC firms Angels who have been investing in AI companies and they're going long in AI which proves that this is the next big Paradigm to build all technology then I talked about the gap between AI research and Engineering which can only be bridged with this new class of Professionals of AI Engineers lastly the fourth point which is the growing ecosystem for AI powered applications now with developer tools with Cloud platforms with open source repositories and projects that people are developing in this space and on top of that we have a growing Community to support you in this journey to become an AI engineer which is obviously always great to have so that is why I feel that this is the next big Tech role and I hope I was convincing enough and I could share my Discovery with all of you now in the next video which is dropping this Saturday I'm going to be sharing a comprehensive road map to become an AI engineer along with the learning resources to develop the required skills so consider subscribing to my channel click on the Bell icon so that you get notified when that video goes live up until then keep learning keep building