How to Automate Filling Out Forms with AI Structured Outputs | OpenAI + Claude + Gemini
In this video I go over some experiments I've been doing with integrating LLMs with structured outputs to improve copy pasting for filling out forms. The core idea initially was to use AI to automate filling out forms in the browser by having an LLM process unstructured text and extract the required input forms, and then paste them properly in any online form needed, I had some issues with a general approach but got some cool things working! In this video I go over the successes and failures of these experiments! Hope you like it :) Cheers!
📚 Chapters:
00:00 - Introduction to Automating Fo…
Watch on YouTube ↗
(saves to browser)
Chapters (21)
Introduction to Automating Form Filling with AI
1:00
Using Pydantic Objects for Data Structuring
2:00
Extracting Information with Functions and Endpoints
3:00
Integration with HTML and API Endpoints
4:00
Conceptualizing Form Filling with Screenshots
5:00
Experimenting with Google AI Studio and Spatial Understanding
6:00
Locating Input Points with Gemini 2.0
7:00
Automating Screen Input with Python and PyAutoGUI
8:00
Challenges and Future Prospects for Form Automation
9:00
Using Multimodal AI for Data Mapping
10:00
Combining Raw Text with Form Structures
11:00
Generating and Using Pydantic Objects with Claude
12:00
Dynamic Code Loading for Form Automation
13:00
Integrating APIs for Enhanced Automation
14:00
Handling Screen Location Issues and CSV Export
15:00
Using Click-based App for Locational Reference
16:00
Data Management with Pandas and CSV Files
17:00
Troubleshooting Screen and Input Matching
18:00
Finalizing Inputs for Accurate Form Filling
19:00
Evaluating Experiment Outcomes and Future Improvements
20:00
Conclusion and Invitation to Collaborate on GitHub Projects
Playlist
Uploads from Automata Learning Lab · Automata Learning Lab · 0 of 60
← Previous
Next →
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
A Quick Tutorial on NLP Basics
Automata Learning Lab
Automating your Digital Morning Routine with Python
Automata Learning Lab
Exploring Problem Solving with Python and Jupyter Notebook #1
Automata Learning Lab
Summarize Papers with Python and GPT-3
Automata Learning Lab
Automating Anki Flashcards with Python
Automata Learning Lab
An Experiment Tracking Tutorial with Mlflow and Keras
Automata Learning Lab
Automating Google Forms Submissions with Python
Automata Learning Lab
Productivity Tracking With Python and the Notion API
Automata Learning Lab
When your Machine Learning Model Fails Do This ;p
Automata Learning Lab
Machine Learning Tip#1 Practical Deep Learning Course
Automata Learning Lab
A Practical Summary of Matplotlib
Automata Learning Lab
Machine Learning Tips: Deep Learning Monitor
Automata Learning Lab
Machine Learning Tips#5 MLOPs specialization in Coursera #machinelearning
Automata Learning Lab
Automatically Changing Desktop Wallpaper with Python and the Nasa Image API
Automata Learning Lab
Building an Image Classifier to Filter Out Unused Images From Your Photo Album with Machine Learning
Automata Learning Lab
Automating VS Code Snippets with Python
Automata Learning Lab
How to Set Up a Machine Learning Environment with Conda and Pip-Tools
Automata Learning Lab
9 Google Search Tips for Machine Learning
Automata Learning Lab
Thinking Tools
Automata Learning Lab
Goal Planning for the IPad Pro
Automata Learning Lab
Automating Car Search with Python and Data Science
Automata Learning Lab
Generating Images from Text with Stable Diffusion and Hugging Face
Automata Learning Lab
A Practical Introduction to Data Science using the Spaceship Titanic Dataset from Kaggle
Automata Learning Lab
Quick Introduction to Python Syntax
Automata Learning Lab
Jiu Jitsu App with Python and Streamlit
Automata Learning Lab
Arbitrary Arguments in Python
Automata Learning Lab
2 Apps for Coding In The Ipad Pro
Automata Learning Lab
From Tensorflow to Pytorch?
Automata Learning Lab
Automate Lecture Notes with Python
Automata Learning Lab
Building an Audio Transcription App with OpenAI Whisper and Streamlit
Automata Learning Lab
Productivity Tracking with Python Short Summary
Automata Learning Lab
Summarize Papers with GPT3
Automata Learning Lab
Automating Expense Reports with Python
Automata Learning Lab
ChatGPT, Angry Pandas and AI Code
Automata Learning Lab
7 Strategies To Learn Anything Using ChatGPT
Automata Learning Lab
Building a Thought Summarization App with Whisper and GPT3
Automata Learning Lab
Automating your Morning Routine
Automata Learning Lab
Visualize a Neural Net Learning Polynomial Functions
Automata Learning Lab
Automating Notion with Python
Automata Learning Lab
Pose Tracking for Jiu Jitsu #automation #machinelearning #jiujitsu
Automata Learning Lab
Pose Tracking for Jiu Jitsu - Update #jiujitsu #machinelearning
Automata Learning Lab
Update to my Pose Tracking for Jiu Jitsu Project #machinelearning #jiujitsu #ai #deeplearning
Automata Learning Lab
ChatGPT API Released by OpenAI
Automata Learning Lab
ChatGPT API Response Format #machinelearning #ai #datascience
Automata Learning Lab
Beyond Stable Diffusion with Composer | Automata Learning Lab Paper Series #1
Automata Learning Lab
Beyond Diffusion Models with Composer #machinelearning #ai
Automata Learning Lab
Machine Learning for Jiu Jitsu
Automata Learning Lab
Finished my Project of Applying Pose Tracking to Jiu Jitsu #machinelearning #jiujitsu #ai
Automata Learning Lab
Prompt Engineering Basics #machinelearning #gpt4 #chatgpt
Automata Learning Lab
Visual ChatGPT: Integrating Images with ChatGPT Paper Series#2
Automata Learning Lab
Visual ChatGPT #machinelearning #ai #artificialintelligence
Automata Learning Lab
ViperGPT: Visual Reasoning with Code #machinelearning #ai
Automata Learning Lab
LERF - Language Embeddings + NERF for Querying 3D Spaces #machinelearning #ai
Automata Learning Lab
Summarize Papers with Python and ChatGPT
Automata Learning Lab
Large Language Models can use Tools Now! #artificialintelligence #machinelearning #ai
Automata Learning Lab
Sparks of AGI in GPT4? #machinelearning #ai #agi #artificialintelligence
Automata Learning Lab
Toolformer: LLMs can use Tools! #chatgpt #llms #gpt4 #gpt3 #artificialintelligence
Automata Learning Lab
Talking to Your Notes with LangChain #artificialintelligence #llms #gpt4 #chatgpt
Automata Learning Lab
MusicLM - From Text to Music #machinelearning #artificialintelligence
Automata Learning Lab
How to Talk to a PDF using LangChain and ChatGPT
Automata Learning Lab
DeepCamp AI