The Truth About AI Coding Metrics ๐
Key Takeaways
Factory AI discusses the limitations of traditional metrics such as code churn, number of commits, and lines of code in evaluating the effectiveness of AI tools for developers, highlighting the importance of developer sentiment instead.
Full Transcript
But like is code churn up bad? Yes. Because what it tends to be is that in a in very high quality code bases, you'll see 3% 4% code churn when they're at scale, right? This is like millions of lines of code. In poor code bases or poorly maintained code bases or early stage companies that are just changing a lot at once, you'll see numbers like 10 or 20%. Now, if you're Atlassian and you have 10% code churn, that's a huge huge problem because that means that you're just wasting so much time. If you're an early stage startup, code churn is less important. This is why we don't really like report that to every team, just enterprises. Any other like measurements are popular that I mean, you know, this is nice that I'm hearing about coturn, but like what else are enterprise VPs of CTLs? What what do they look at for for the enterprise? is I think the biggest thing cuz there's so many tricks and different dances you can do to like justify ROI number of commits like number of commit like lines of code metrics are usually popular and at the end of the day like what we we initially went really hard in all the metric stuff what we found is that often times if they liked it they wouldn't care and if they didn't like it they wouldn't care and so in the at the reality like at the end of the day no one really cares about the metrics what people really care about is like developer sentiment
Original Description
How do you measure if AI is actually helping developers? Factory AI reveals why traditional metrics fail and what really matters when evaluating AI tools. #coding #softwaredevelopment #AI
Watch on YouTube โ
(saves to browser)
Sign in to unlock AI tutor explanation ยท โก30
Playlist
Uploads from Latent Space ยท Latent Space ยท 0 of 60
โ Previous
Next โ
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Ep 18: Petaflops to the People โ with George Hotz of tinycorp
Latent Space
FlashAttention-2: Making Transformers 800% faster AND exact
Latent Space
RWKV: Reinventing RNNs for the Transformer Era
Latent Space
Generating your AI Media Empire - with Youssef Rizk of Wondercraft.ai
Latent Space
RAG is a hack - with Jerry Liu of LlamaIndex
Latent Space
The End of Finetuning โ with Jeremy Howard of Fast.ai
Latent Space
Why AI Agents Don't Work (yet) - with Kanjun Qiu of Imbue
Latent Space
Powering your Copilot for Data - with Artem Keydunov from Cube.dev
Latent Space
Beating GPT-4 with Open Source Models - with Michael Royzen of Phind
Latent Space
The State of Silicon and the GPU Poors - with Dylan Patel of SemiAnalysis
Latent Space
The "Normsky" architecture for AI coding agents โ with Beyang Liu + Steve Yegge of SourceGraph
Latent Space
The AI-First Graphics Editor - with Suhail Doshi of Playground AI
Latent Space
The Accidental AI Canvas - with Steve Ruiz of tldraw
Latent Space
The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert
Latent Space
The Four Wars of the AI Stack - Dec 2023 Recap
Latent Space
The State of AI in production โ with David Hsu of Retool
Latent Space
Building an open AI company - with Ce and Vipul of Together AI
Latent Space
Truly Serverless Infra for AI Engineers - with Erik Bernhardsson of Modal
Latent Space
A Brief History of the Open Source AI Hacker - with Ben Firshman of Replicate
Latent Space
Open Source AI is AI we can Trust โ with Soumith Chintala of Meta AI
Latent Space
Making Transformers Sing - with Mikey Shulman of Suno
Latent Space
A Comprehensive Overview of Large Language Models - Latent Space Paper Club
Latent Space
Why Google failed to make GPT-3 -- with David Luan of Adept
Latent Space
Personal AI Meetup - Bee, BasedHardware, LangChain LangFriend, Deepgram EmilyAI
Latent Space
Supervise the Process of AI Research โ with Jungwon Byun and Andreas Stuhlmรผller of Elicit
Latent Space
Breaking down the OG GPT Paper by Alec Radford
Latent Space
High Agency Pydantic over VC Backed Frameworks โ with Jason Liu of Instructor
Latent Space
This World Does Not Exist โ Joscha Bach, Karan Malhotra, Rob Haisfield (WorldSim, WebSim, Liquid AI)
Latent Space
LLM Asia Paper Club Survey Round
Latent Space
How to train a Million Context LLM โ with Mark Huang of Gradient.ai
Latent Space
How AI is Eating Finance - with Mike Conover of Brightwave
Latent Space
How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit)
Latent Space
State of the Art: Training 70B LLMs on 10,000 H100 clusters
Latent Space
The 10,000x Yolo Researcher Metagame โ with Yi Tay of Reka
Latent Space
Training Llama 2, 3 & 4: The Path to Open Source AGI โ with Thomas Scialom of Meta AI
Latent Space
[LLM Paper Club] Llama 3.1 Paper: The Llama Family of Models
Latent Space
Synthetic data + tool use for LLM improvements ๐ฆ
Latent Space
RLHF vs SFT to break out of local maxima ๐
Latent Space
The Winds of AI Winter (Q2 Four Wars of the AI Stack Recap)
Latent Space
Segment Anything 2: Memory + Vision = Object Permanence โ with Nikhila Ravi and Joseph Nelson
Latent Space
Answer.ai & AI Magic with Jeremy Howard
Latent Space
Is finetuning GPT4o worth it?
Latent Space
Personal benchmarks vs HumanEval - with Nicholas Carlini of DeepMind
Latent Space
Building AGI with OpenAI's Structured Outputs API
Latent Space
Q* for model distillation ๐
Latent Space
Finetuning LoRAs on BILLIONS of tokens ๐ค
Latent Space
Cursor UX team is CRACKED ๐ป
Latent Space
Choosing the BEST OpenAI model ๐
Latent Space
How will OpenAI voice mode change API design?
Latent Space
STEALING OpenAI models data ๐ฅท
Latent Space
[Paper Club] ๐ On Reasoning: Q-STaR and Friends!
Latent Space
[Paper Club] Writing in the Margins: Chunked Prefill KV Caching for Long Context Retrieval
Latent Space
The Ultimate Guide to Prompting - with Sander Schulhoff from LearnPrompting.org
Latent Space
llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE
Latent Space
Prompt Engineer is NOT a job ๐
Latent Space
Prompt Mining LLMs for better prompts โ๏ธ
Latent Space
The six pillars of few-shot prompting ๐ง
Latent Space
Language Agents: From Reasoning to Acting โ with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
Latent Space
[Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)
Latent Space
Can you separate intelligence and knowledge?
Latent Space
More on: AI-Assisted Code Review
View skill โRelated Reads
๐ฐ
๐ฐ
๐ฐ
๐ฐ
AI chip startup Etched defies skeptics, hits $10.3B valuation from big-name investors
TechCrunch AI
Agentic Engineering with GitHub Copilot: A Practical Guide
Dev.to AI
Same coding task, three routes: one model's own test failed
Dev.to ยท Zephyre
7 Best Claude Code Alternatives for CLI Agentic Coding
KDnuggets
๐
Tutor Explanation
DeepCamp AI