Elasticsearch Course for Beginners

Key Takeaways

in this beginnner elastic search course you'll discover how to harness the power of one of the world's most popular distributed search and analytics engine in this course you'll learn how elastic search enables Lightning Fast search capabilities across massive data sets while providing realtime analytics and visualization possibilities that can transform how your applications handle data imod sadik developed this course hi everyone my name is iMed saik I am an AI and data science engineer at AGC Europe I am extremely happy to share my first course on the fre Camp YouTube channel since I couldn't find an extensive course for beginners that talks about elastic search I created my own one to fed this C this course is for beginners so even if you know nothing about elastic search by the end of this course you will learn a lot of things and hopefully you will be good at El at using elastic search this course is divided into two parts the first part you will learn a lot of things about El Sixers I'll talk about indices I'll talk I will show you how to index documents what are the data types how to use analyzers to analyze text how to work with embedded models how to use semantic sech how to use pipelines to try transform your data and other things in this second part we are going to use all the knowledge that we have learned to work on a final project and as you can see in the background I have a dark background with some Stars moving from the bottom of the page to the top of the page this is the theme that we are going to stick with it's going to be about astronomy and in this second part we are going to use the Astron om picture of the daily data set I will show you how to use pipelines to transform this data because it contains HTML tags and then we will try to index the data after that you will learn how to use pagination in that website that we are going to build also we are going to provide the user the ability to switch between regular and semantic search and we will allow the user to use filters to filter documents by date in this second part we are going to use VJs and fast API to develop the full sack web application in the back end we are going to use Python but if you want to use other programming languages this is possible because elastic search basically is just a server that runs either locally on your machine or in the cloud and to communicate with elastic search you make HTTP request this course as you can see from this introduction contains a lot of stuff I hope that it is going to be useful and informative and please when you finish this course give me some feedback so that I can make better courses in the future that's it for the introduction now let's get started welcome everyone I am excited to start this series where we would explore the power of elastic search and learn how to build an amazing things with it in this introductory video I will give you a global view of what we will cover throughout the series let's dive in elastic search is a versatile search engine that offers flexibility in deployment you can install it locally for personal use or use it in the cloud for large scale applications for personal project running elastic search locally with Docker is a great option however if you are working with an organization handling vast amounts of data Cloud deployment might be the way to go for easier scalability and management now let's talk about getting your data into elastic search data is organized into indices or vector source so in this example I have three indices these can be slded either locally or in the cloud elastic search is incredibly versatile accepting various data types such as pure text numbers dates and even vectors if you want to use embeddings this flexibility makes elastic search a powerful tool for a wide range of applications okay so once your data is indexed the possibilities are endless you can build for example powerful search engines intelligent recommendation systems and rag systems retrieval augmented generation throughout this series we will work on practical examples to apply the techniques you will learn giving you hands-on experience with great applications of elastic search while elastic search supports multiple programming languages we will focus on using the python client in this series I hope this overview has pictural interest in elastic search and its capabilities in the upcoming videos we will delve deeper into each topic providing you with the Knowledge and Skills to harness the full power of classic search stay tuned and let's embark on this exciting journey together hello everyone in this video I will show you how to run elastic search locally and how to install all the libraries that you will need in order to interact with elastic search visit this website the link can be found in the description here as you can see in this page we have two options to run elastic search you can do that in the cloud or use Docker to install it locally in this series I will focus on using elastic search locally so click on this link and make sure to copy this command now open the terminal make sure that you have Docker installed and here paste in the following commands so as you can see here it did not find the image so it will install it so I'll wait for this to finish and I will come back awesome so the installation has finished so now if you want to to see the running containers you can run this command and as you can see um here it is running now here they said that to verify that the installation was done successfully or correctly you need to click on this link so let me do that and as you can see here you have some information let me zoom in so that you can see better so here you can see that this is the version that we have installed so this is running on Docker and yeah so if you see this you you now for search this means that everything is good awesome so now because as I said in the beginning of this series we will be using python to interact with elastic search so the first thing I always like to do is to create a new virtual environment so here because I have Anaconda installed on my PC so I will use it to do that to create a new environment you can run the following command cond create dasn and here um you specify the name of the environment and here I will be using python 3.1 click on yes to install all the necessary libraries and I just want to say if you don't not if you do not have araka install uh you can create a virtual environment with using the V module from python okay so here the only thing we need to install is elastic s so Pap install elastic search but here don't forget you need to First activate the virtual environment so let's do that quickly going to activate elastic search and now pick install elastic search awesome so you can see that this package did not take a lot of time to install and yeah if you have arrived at this point you installed everything you need so see you in the next video Welcome Back in this video I will talk about creating an index in elastic surge with the python client I will start by explaining what an index is how to create it and what are the settings you can configure while creating it an index is a collection of documents that share similar characteristics let's illustrate this with an example say that you have a lot of products and for each one you have a PDF documents in these PDF documents you can you can have for example the name of that products its price and the description Etc each document will would be converted into a Json object and stored in for example the products index an index in elastic search is comparable to a database in a relational database system but it is optimized for storing and searching large volumes of textual numeric and structured data in this image I show you how to create the index first we to connect to the cluster and then use this create method to create your index this is very simple to do now let's go to the code and show you how to do this okay so here is a notebook that I have created for you you can go to the description you will find this on GitHub there I'll try to put all the notebooks that I will be showing you in these videos so make sure to check that and download the notebooks if you need first you need to connect to elastic search if you try to run this it it might not work why because remember in the previous video which is the second video in this series I showed you that you need to install elastic search locally and we use Docker to do that so you need to first run the image so sud sudo Dockers T and here uh the container is called elastic search so now it is grunning and now if I rerun this cell uh let's give it some time because yeah it takes some time to start the cluster so yeah now you can see that we are connected to elastic search and here we see the same information as if we click on this link so you can see that we have the same info that that is showed in this um website window okay so here it says that we are running on doer and the version of elastic search that we are using is aot 15 which is the latest one great so now let's see how to create an index um so because I was testing uh the index might be in elastic sour so I just I will just delete it and then create the index so if I on this so here in this output you can see that it shows us that everything went well and we were able to create the index now let's go back to the slides when we want to create an index we have some settings that we can tweak for example here I will be talking about just the number of charts and the number of replicas what are they and what do they do well let's say that you have your product index if you specify that the number of charts should be equal to two then elastic search will try to split each document into two parts so here after the charting process you can see that the product index contains six parts instead of just three full documents but don't worry when you make a query to retrieve the data elastic search knows how to reconstruct the documents to find information you need to change the number of charts in the index it's simple just add this key to the settings dictionary and here I specifi three so this means that we will split documents into 2D charts now let's move to the number of replicas a replica is a copy of the data let's take our charted index because here I specified the number of replicas as one this means that we will duplicate this fragmented index just once why we do this well it's simple when you duplicate the data you improve system resilience let's say for example that the first index shuts down what will happen if you don't have if you did not duplicate the data then the whole system will fall down so in this case you can quickly switch to the duplicated index and use it also replicas enable parallel search and retrieval operations so it makes searching very quick again to change the number of replicas it's so simple you just add this key and here I specify two which means that I will duplicate the data twice let's go back to the code and let's continue so here I have um this section and here you can read about what are shards and what are replicas if you want but here you see that I can specify the number of shards to be three and the number of replicas to be two so again I will delete the previously created index and create this new one and again I get this response which means that everything was created successfully I hope that you now know how to create indices and hopefully understand what happens under the hood I hope that you found this video helpful if you did then please leave a like subscribe to the channel and wait for the upcoming videos see you next time welcome back in this video I will talk about how to insert documents into an index in elastic search I will start by explaining the format of the document that elastic search EXP ects after that I will show you how to insert one document or multiple documents finally I will talk briefly about mapping in the previous video I showed you how to create an index but how do you add documents to it say that we have a PDF file which consists of just text or let's say that that we have a pure text or even a website if you want to put that type of document into elastic search you cannot do that just like that so you need to convert your document into a Json format because this is the format that elastic search expects for example if you have a PDF file about a product you could extract the name the price the description Etc this process is necessary and should be done before indexing the documents let's say in this example that you have 100 documents and to say that you converted them into Bas on format and inserted them into this index that I am calling my index a quick reminder all those 100 documents must have the same fields and this is how we do that in the code you can see it is very simple you just call the index method and you are done when ins searching the data there is something that happens under the hood which we call mapping basically elastic search tries to figure out the data type for each one of the fields you have in your documents this process as I told you is called mapping and is done automatically by default but if you want you can force the mapping manually for example here the field created on should be considered as date and the text and title fields are considered to be text in a later video in this series I will talk about the different data type stat elastic search supports so don't worry about this for now now back to the code again as always make sure that elastic search is running locally before running the first cell and then if you see this message or this information printed to the console this means that the connection is established to elastic search and you are ready to go here as it as we saw in the previous video This is how we create a new index so here I delet it if it exists and after that I just create it and here I have created a dummy document that contains three Fields the title the text and the creaton field and we insert it by using the index method so here we specify that the index should be so here make sure that the index is the same as the one you have here so here I made a mistake instead of my documents it should be my index and here you're passing your documents so I'll just run this and as you can see here this method returns a response object which contains some information like the ID the the number of charts Etc so for example we can look at this result key and here it size created which means that we were able to insert the document in into the index here we have the number of charts you can see that the total number of charts is equal to two remember in the previous video charting means breaking the document into multiple smaller parts for example here this document was broken into two parts here if you want for example to retrieve your document by ID you can do that so this document was assigned this ID and here just to verify you can see that we inserted that document into this index that I called my index now to index multiple documents I have created again a tmy Json file that contains three documents and they have the same Fields as before this is simple you use the Json package to read that file and it gives you back a list of documents here I created two methods one that inserts a document to index and the other one prints the results of that operation so as you can see here I am inserting each document separately in a for Loop this is how you insert multiple files and here you can see that we have created for all of them which means that we were able to index the three documents and they were all split into two sharts finally we can show the mapping that was done automatically by elastic search with this get mapping method let me run this for example create a dog was considered as a date field and for text and title both of them was considered as a text field so for most for most of the time elastic search does a good job identifying the type of each field but if you want to do this manually for any reason you can do that so here but this only works when when creating the index so before inserting the documents you need to specify the mapping while creating the index so here as you can see you can either pass it inside this create method if you want so you can see that you have mappings or you can do that later if you did not add any documents okay so here I have this mapping object inside of properties you specify the data type for each shield for example created on is set to page text is set to text Etc and after that you use the put mapping to add the Mapp manually and here if I pr the mapping you can see that is the same and as I told you if you want to add this directly in the create method you can do that here I am deleting the index and while creating it I am passing the mapping instead of using Bo mapping but both of them works and all need to know is this should be done before insert inserting the documents and ideally when creating the index and for me personally I don't like this approach which is the manual um mapping because it is more involved but in some cases it might be needed we arrived at the end of this video I hope that it was helpful and that you learned something from it please don't forget to like this video leave a comment and subscribe to the channel see you next time hello everyone this is video number five in this series today I will talk about the field data types remember when indexing files in elastic sech we have a process that is called mapping which can be done either manually or automatically during the mapping phase elastic search tries to figure out the data type for each field in the document this example we have three fields and here are the data types I will start by showing you the common data types and in each slide I will be referring to the documentation to learn more to get the slid you can find the link in in the description you will be redirected to my GitHub repository there you can download the slides for free in this category we have the B data field which accepts a basic C4 encoded string this field is not searchable and is not stored because simply the basic C4 encoding is too expensive to store so what do you do if you want to retrieve data well you use the underscore Source key to retrieve the document and after that you can get the binary value and decode it here is an example of how to use this field let's say that you have an image first you encode it to get the binary representation and after that you Index this encoded string alongside the binary data field we have bonan data field which is very simple you sort true or false values we have numbers they can be long integer bite short and others we have dates and finally the keyword data type you would use use the keyword data type if you want to filter or sort the documents because as you can see the use case for the keyword data field is when you have IDs zip codes Etc now before I move to the other category I will stop and go to the to the notebook to show you how to use the data types that we have seen for now so here are going to use this notebook you can see that once again I am referring to the documentation the first step again is to connect to elastic search and here is how you use the binary data view here I am creating a new index specifically for this data type and here you can see that the image data will be sld in binary format okay so let me create this index and here I am just taking a an image basically I am taking this image and I am using the B 64 package to get the base 64 representation of the image and here let me run this C so here I'm just printing the first 100 characters but if I show you the length of this string you can see that it contains well over 270,000 characters which is very big so this is why we don't store this kind of data in elastic search and it makes it not not searchable because if you have 100 images I mean multiply this by 10 or even more it depends on the size of images and you will get a fair heavy um number of characters stored in your database so after that we construct our document and we use the index method that we have seen in the previous video to insert this document into this index okay so let me run this and as you can see the result of this operation is created which means that this document was inserted successfully Now quickly let's move to the other data types so here we have the keyword the floats uh this is basically a number here we have the date the date data type and finally the Boolean here um as an example I I created here a book so here this is the reference of a book so this is why you are making this a keyword because we want to for example after that to filter or sort books by reference the price should be a float number and so here is the date and here is our bulling value and we can insert this document to this index and one more time we have created which means that this operation went successfully awesome so now let's go back to the slides to see the other categories now let's move to the objects types here we are going to deal with Json data the first data type in this category is the object data type to demonstrate the use of this data type I have here an example consider this Json object when indexing it it will be stored like this you can see that instead of having nested objects we will use the dot notation to represent each value for this nested CHR so after that if you want to let's say get the age you will use manager. age and then specify the value this is one way of indexing a Json object but we have other data types in this category like the flattened object you will use this one if you are dealing with deeply nested Json objects otherwise stick with the object data type but the problem with the flattened object is that you lose the hierarchical structure I will show you what this means later in any example we also have the Ned data type this one is used when you have an array of Jason objects and the great thing is that this type preserves the the relationship between Fields now here is the example that I was talking about say that you have this Json object you can see that the user contains an array of objects while indexing it it becomes like this if you use the flattened data type you can see that we lost the relationship between Fields before we had Alice we had Alice wise in the same object but now if we try to get this person's first and last name we won't be able to do that because they are not aligned in this array the problem though will be solved if you use the nested object data field so in the nested data type you would have Al and here you will have white and this will preserve the relationship between your Fields now before I move to the other category let's go back to the Cod to see how to use these data types so in the same notebook I have the object types here this is how we use the object data type you can see that we have basically a Json file with two keys first name and last name and both of them are type of text we will see this in the slides don't worry about it okay so we created the index and here so this is my document so an author contains the first name and last name so when running this so it says created great so we we indexed the values inside this index and in a later video I will show you how to search or get documents so don't don't worry about this for this video for the flatten version so it's simple so it's so you take the same other documents and here just specify flattened and elastic search would do the job to convert the document into a DOT Pro into a DOT notation okay so let's create index we have the same author basically we have the same document but internally it will not be sa the same way if we have used data type or flattened or uh even nested data type okay so let's move to the last data type which is nested again so here the only thing you do so is to change this from flatten to nested but for nested you need to make sure that you had an array of objects okay so let's create the index and let's index our documents so here remember you have a set of documents in um in your uh documents and here basically you can see that we have these objects and the N data type will make sure to preserve the relationship between the fields so uh John Smith will be preserved instead of being thrown out randomly okay so this is what I have for you for the object data type now let's go back to the slides now we are going to talk about the text sech types the the first data type in this category is text we can use this data type to store a string of text but this is different than using the keyword data type because the text data type is optimized for search for example you can take the body of an email or the description of a product and store it but when you do that so let me show you this example let's say that this is the description of a product stored in this document and if you specify this field to be stored as a text data type elastic search will try to analyze the document and optimize it for search we we also have the completion data type search as type and the annotated text you can see that I am going quickly on some data types but don't forget you can all always refer to the documentation just click on this link and there you can read more about each data type separately so this is what I have for you uh for this category let's go back to the code and let's see how to use them okay so three text search types this one is going to be easy so here I am creating a text index let's say that we have uh a set of documents that have the email body you will saw that as a text okay so let's create create the index cool and here is our document it has just one field the in body let's insert it and great it works for the completion you will use here you need to use this suggest basically for each document you'll have a set of inputs so if you are typing Mars it's going to give you if you are typing for example na it will give you Mars because this is the CH it completion to what you were typing and here you basically basically you have a set of documents and based on what you are typing elastic search will try to analyze the text that you have written and the inut stat you gave it in each document and after that it will give you the closest match so let's create the index so here we have our documents and basically you use the index method to add them uh like this and here we have Creator which means that everything went well finally in this video I talk about the spatial data types we have the goop Point data type you you use this to store the for example latitude and longitude of a point we have the Joo shape this is used to store complex shapes like this here you will have a list of points as the field and we have have other options such as the points and the shape data types this one was quick but let me show you how to use that in in code okay so we add out at the last section let's expand this so let's start with J points again here you have the location it is set as GE point and as I told you for GE point you specify the latitude and longitude we can you can do you can do it like this so you you put the values inside an array and you specify that this is a point because elastic search will determine that okay so this is Aton which means that this should be the latitude and this should be the longitude and when inserting this it gives us created for G shape it's the same so you have the location so this is just an arbitrary name but what matters is this so when you specify GEOS shape you need to pass a list of points for example here we you have this is the first point this is the second point and it's going to create a line because one to draw a line you need at least two points and for a polygon you will have a set of points and when you work with complex shapes you need to make sure that the shape is closed so that this works okay so we we have the first document which is just a line but here we have a polygon and we can insert both of them like this and here I'm showing you how to use points again so it's it's similar to um to Z points but uh you just specify that here to be points and yeah this is how you you work with spatial data types that was a lot I hope that this video helped you know about the diversity of data types that elastic surge supports I am once again link to the documentation because it is the best place to learn about field data types thank you for watching please like this video share it with your friends and subscribe if you like see you next time hello everyone in this video I will show you how to delete a document from an index in elastic search let's demonstrate how this works by showing you an example say that we have an index called my index inside this index we added three documents each document is assigned a unique ID the delete operation expects two arguments the index where the document that you want to delete exists and the ID of that document if we give the delete operation my index and the ID number one then this operation will successfully delete that document from the index on the other hand if we provide the delete operation with an ID that does not exist it will throw an exception this is all you need to know about the delete operation but if you want to read more check out the docs from here now let's see how to delete documents in Python make sure to open this notebook you can see that once again I am pointing to the documentation click here to read this if you want first of all we need to connect to elastic search make sure that the docker container is running after that here I am creating an index as we already do here I am using that dummy data just to look some documents and insert them to this index here I have only three documents and here are the IDS okay so I made sure that I saw the ID for each documents because I will be using it later okay so this is just a preparation now let's see how to Del documents so the first example shows you how to delete documents successfully here you can see that we use the delete method and we give it two arguments the index as well as the ID here I am taking the first document or the ID for the first document and I am using it to delete it so if I run this cell and the cell after it you can see that here it says that the result is deleted which means that we were able to delete this document with no problems the second example shows that if you specify an ID that doesn't exist in the index it will throw an exception so here I am using the triy except block to catch this exception and as you can see here it says that the document was not found we arrived at the end of this video I hope that it was helpful please don't forget to leave a like and subscribe to the channel if you like see you next time hello everyone and welcome back to the channel in this video I will show you how to get a document from an index in elastic search let's demonstrate how this works by showing you an example say that we have an index called my index inside this index we added three documents this is the same as the previous video where we talked about delete arguments or the delete method and you are going to see that it is very similar to it remember that each document has a unique ID now the Cat Operation expects two arguments the index where that document is stored and the ID of the document if I give these two arguments which are my index and ID equal to one this operation will get me the first documents on the other hand if I specify an ID that doesn't exist in in the index for example four again the operation will return an error saying that document was not found that's all you need to know about the G operation but if you want to die deeper I highly suggest you to to click on this link to read the documentation there you will find everything you need to know about the get operation now let's see how to do this in in a notebook that I have prepared make sure to open the seventh notebook in this series again if you want to read the documentation please click on this link but if you are ready to start then make sure that the container is running so that you can connect to elastic search after that here I am doing the same preparation that we did in the previous video I am just taking that Dy Json file and on in I have inserting three documents into the index and as you can see here are keeping track of the IPS so that we can use them in the Cat Operation okay again we have two examples one that shows how we can retri documents successfully and the other one shows you that if you want to retrieve a document with an with an ID that does not exist you will get an exception so here I am taking the first ID and here if I look at the response you can see that here I am getting the document so if you want to get the document back just use the source key and this will will get you the Json that you have inserted but if we try to give an ID that doesn't exist again we get a not found error exception we arrived at the end of this video I hope that this was helpful please before you leave if you want leave a like on this video And subscribe to the channel this will help me a lot see you in the next video hello everyone and welcome back to this video in this one I will show you how to count documents that we have inserted into an index again I keep talking about this my index example but bear with me you will understand what I will show you in this video let's say that we have inserted some documents to this index but we forgot how many documents we have in the index to be able to count the documents we use the count method this method receives one parameter which is the index okay so when we fit the index to this method it gives us the number of documents we can also provide this method another parameter which is called the query this parameter is used to match certain criteria for example if you want to search for a specific word in all the documents this method will return you that number of documents that has that specific word that's all I had to show you about the count documents in this video if you want to read more please make sure to click on this link now let's go to the code to show you how to use this functionality make sure to open the eighth notebook here as you can see first you need to connect to Classic search after that we are creating the my index index here again we are using that damid data Json file to insert three documents and as you can see to count the documents we can use the count me method here we are providing the index and in the response object we can access the count value and if are in the cell it says that the number of documents in the index is three which is correct and here is an example that shows you how to use the query here you can see that we have some specific things related to date because here I am searching for a specific document that has this value so let me show you what I'm talking about if we look at Tomy data JSO file you can see that I have a field which is called created on this is going to be mapped into a data field and when searching or when making a query we have some specific terms that we can use for data fields like range greater than or equal and less than or equal don't worry about this I will talk about this in details in the search API in one of the upcoming videos so here basically I am searching for a document which will which has the value 2024 924 okay and here it returns me one which is correct if I go back to the D data adjon file let's put it aside you can see that we had just one document with this value now if I change this and put 24 again here and let me run everything okay so here it gives you zero because it needs times to index the files but if I run it one more time you can see that now it says that we we found two documents with this value in the created on field we arrived at the end of this video I hope that it was helpful please before leaving this video make sure to leave a like And subscribe to the channel if you like see you in the next video hello everyone and welcome back to the channel in this video I will talk about the exists API this API is very useful you can use it to check if an index exists in elastic search or if a document exists inside an index so let's show how this works with an example let's say that we have three documents we are going to insert them in this my index here if you want to check if the index exists you need to use this so you take clients then you take indices and you use the ex exist method inside that module this one checks it takes in an index and it checks if it exists or not but to check if a document exists inside the index you just use this you take the client and use the exist method inside the client and but but here you need to specify two things the index and the ID of the document okay so this is what I said this method checks if a document exists in index Okay so this is how you use the exist API and one more time please check the documentation to learn more about the exists API now let's go back to vs code to see how to implement this okay so open the ninth notebook in this series if you are wondering where to get this notebooks you can find the link the link to the repository in the video description so please go there and clone the repo or download it to get access to the notebooks first of all we are doing some processing we are taking some documents from a Json file that you will find in the repository and we are inserting everything into this my index okay here we have documents IDs because we need that in order to check if a document exist or not so let me run the first cells and as you can see here are the documents IDs great so that that was just a preparation now let's see how to use the exists API first of all to check if the index exists you use as I said use the client then indices inside indices we have the exist method this one only takes in the index name okay so here I'm giving it my index and he returns it returns to me true which is good because we created that index before but if I gave it something else let's say my index two you can see that here it return is false because we don't have this index in elastic sech okay now to check if a documents exists we can just use the client because inside this we have another exists method but this one takes in two arguments the index as well as the ID so here we are checking if this document exists inside my index and the response should be true but if I do this and provide a an ID that doesn't exist it returns to me false we arrived at the end of this video I hope that it was helpful please before leaving leave a like on this video And subscribe to the channel if you want that will help me immensely see you in the next video hi everyone in this video I will walk you through how to update a document in elastic search the update API in elastic search works whether the document already exists in the index or or if you want to insert a new one let's start with the first scenario imagine we have an index containing two documents the update method requires two main parameters the index name and the document's ID internally the update method performs three operations it retrieves the document it modifies it and finally it reexes the updated document back to the index so so how do we perform an update let me show you an example say we have a document with two Fields book ID and book name we first insert this document into the index this operation assigns the document a unique ID next we use the update method with my index and ID equal to one as inputs to retrieve the documents this is the first step in the updates process process to make changes we use a parameter called Script where we specify the field we want to modify for example let's change book ID to two once we pass this script parameter to the update method we can modify the document this is the second step in the process finally the updated document is saved back into the index completing the third step of the pipeline this is a simple example but you can do much more for example you can add or remove Fields insert items into fields and more we will explore additional examples in the notebook now what happens if you try to update a document that doesn't exist in the index for instance we still have the same two documents in the index but what if I try to update a document with an ID that does not exist you can see that here I am trying to update the document which has ID equal to 4 but this document does not exist in this case the update method can create a new document with the specified ID all you need to do is provide additional parameters such as the Json content and set doc as upstairs to through if you do these things the update document will work like a create method or a index method and it will add this document to the index that's a quick overview of how the update API works I have covered the two main use cases but if you want to explore further be sure to check the official documentation now let's jump into the notebook to see the update API in action open the 10 notebook in this repository you can see that here you can click on docs to read about the update API in details first of all we need to connect to elastic search here I am using again I'm creating a new index and I'm inserting three documents which comes from this dumy data Json file if you don't have these notebooks please go to the description click on the link that will take you to GitHub and download or clone the repository there you will find everything okay so that was a preparation now let's start showing you how to use the update API so as I said we have two use cases so here uh if the documents exist in the index what we can do we can take for example a document and update an existing field first of all let me just open the dummy data file and put it here you can see that here by default the title is simple title one so here what I am doing I am taking the first documents which is this one and here as I told you in the slides you need to use the scripts parameter and inside it here you specify the the new value for the title so here parameters allows you to change the field dynamically which means that you can reuse these values inside the script but you can just come here and type in the new value if you want so here you can see that context. Source will get me the document and after that when I can take the document and access the title this is what I am doing after that and here I am passing the new value which is new title so let me run this cell and as you can see here if I take response and try to use the get method which I showed you in a previous video you can see that here the new value for title is new title instead of sample Title One okay so this is the first use case but you can add for example new fields that doesn't exist in the documents so here again I am taking the first document and here I am adding a new field which I called New Field and I'm giving it this dummy value let me run this again let's use the get method and as you can see so we have created on we have our new field and we have text and title which already exists in the documents you can add a new field by using this script parameter or by using doc both of them do the same thing so let me adjust find this and as you and as you can see we have a new field which I called new value to and you can either use doc or scripts they they will work together so I showed you how to modify a field how to add a field now let's see how to remove a field it's simple you just use context. source and here you have a method which which is called remove and inside it you specify the field that you want to remove let me run this and as you can see so here we removed New Field and as you can see in this Source documents we have we don't have that field okay so that was the first use case the second one is where you don't have the documents in the index so here you can see that ID equal one does not exist if I take the documents IDs list you can see that here are the IDS for the documents that are already in the index okay so here as I told you in the slides if you want to if you want the update method to create this document you need to specify doc as upsert equal to true and you need to provide it with the Json data so here we have both both of them and if I do this you can see that the response says created but if I go back to the first use case you can see that the updates method gives us updated because we already had that document in the index so it will just updated but in the second case because that document was not in the index it says created and let's just verify by using the count method and sure enough it shows us four which means that we already have the three documents that are inside tomida the Json and we added the fourth documents thanks to the object B method we arrived at the end of this video I hope that it was helpful please before leaving leave a like on this video subscribe to the channel if you like see you next time hi everyone in this video I will walk you through how to use the b kpi in elastic search let me show you an example of how the B kpi Works normally if you have an index and want to add a new document you would perform the index operation here I have two documents and I will perform the index operation for each one of them you might also update or delete documents along the way each of these operations requires a separate API call which means the more operations you perform the more API calls are made this can become inefficient when dealing with a large number of operations that's where the bulk API comes in instead of making multiple individual requests you can bundle all operations into a single API call significantly improving processing speed and reducing overhead let's take a look at the Syntax for the bulk API when using it you alternate between specifying the action to be performed and providing the corresponding Source okay so as you can see this is a process where it alternate between the action and The Source the available actions include indexing creating updating and deleting documents however not all actions require a source only the updates index and create actions need one now that we have covered the theory let's look at a practical example of using the bulk API in Python first you will use the client to access the bulk method inside this method you will pass a list of operations where we alternate between actions and sources the first action we will perform is index which requires a source so we provide it right after the action next is the delete action which does not need a source so we move on then we use the create action followed by its source finally we have the update action along with its corresponding source as you can see the syntax is straightforward just alternates between actions and sources remembering that some actions require a source While others do for more details you can click this link to access the official bulk API documentation now let's go to the notebook to see how to apply the bulk API make sure to open the 11th notebook in this series let me remind you if you don't have access to these notebooks please go to the video description there you will find a link that will take you to GitHub and feel free to download the GitHub reposit there you will find everything first please make sure that elastic search is running to make sure that everything is working just run this cell and here it should say connected to clastic search after that here I will show you two sections the first one I'll try to perform some actions without using the B API and in the second part I'll show you how to transform these actions by using the B API okay so let's start first of all we have this dummy data Json file inside it we have three documents so we will be performing the index operation three times after that I will update two documents I will update the first one by modifying the title field to have this new value and I will modify the second document by adding a new field to it finally I'll delete the third document in total we have performed six operations and each one of those operations require a apepi call this approach is slow and inefficient especially if we are dealing with very large number of documents so this is how to do this without the bulk API but let's see how to transform everything with this API I'll make sure to recreate the index and here as you can see so first of all we have the first action which is index you can see that here I am using my index and here I am specifying an ID after that I am taking basically if I go to doc d data you can see that I took the first document and I specified it as the source because remember the index method requires a source I do the same thing for the other documents just make sure that the ID is unique so here I will try to index the second document that's why I gave it ID equal to two and here is the source finally I am indexing the third document in the third action after that we will update two documents first I will update the first document by changing the title to a new title and for the second document I will add a new field with this value finally here this should say action six I will just delete the third document if I run this you can see that we have a key which is called errors if this value is false it means that the bulk API was able to perform all those actions with no problems but if you get true this means that some or all the