t-SNE Explained (Visualize High-Dimensional Data Easily) | CodeVisium #MachineLearning #AI

About this lesson

t-SNE (t-Distributed Stochastic Neighbor Embedding) is one of the most powerful techniques to visualize high-dimensional data in 2D or 3D. It is widely used for: Visualizing embeddings Understanding clusters NLP and image data Model debugging 🧠 1️⃣ What problem does t-SNE solve? Real-world data often has: 100+ dimensions Complex relationships Hidden clusters We cannot visualize this directly. t-SNE reduces data into: 👉 2D or 3D while preserving structure 📉 2️⃣ Why PCA is not enough? PCA: Preserves global variance Linear transformation Misses complex patterns t-SNE: Non-linear Focuses on local relationships Better for cluster visualization Example: PCA → overlapping clusters t-SNE → clearly separated clusters 🧩 3️⃣ How t-SNE preserves local structure? t-SNE: Converts distances into probabilities Ensures nearby points stay close Pushes dissimilar points apart Key idea: 👉 Preserve neighborhood relationships, not exact distances This is why clusters become visible. ⚙️ 4️⃣ What is perplexity in t-SNE? Perplexity controls: 👉 Number of neighbors considered Typical values: 5 → very local clusters 30 → balanced view 50 → more global structure Low perplexity: Tight clusters High perplexity: More spread-out data 🧑‍💻 5️⃣ Python implementation of t-SNE from sklearn.manifold import TSNE from sklearn.datasets import load_digits import matplotlib.pyplot as plt # Load dataset data = load_digits() X = data.data y = data.target # Apply t-SNE tsne = TSNE(n_components=2, perplexity=30) X_tsne = tsne.fit_transform(X) # Plot plt.scatter(X_tsne[:,0], X_tsne[:,1], c=y) plt.colorbar() plt.title("t-SNE Visualization") plt.show() 🔥 Real-world example (Embeddings) from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") texts = ["AI is amazing", "I love machine learning", "Football is fun"] embeddings = model.encode(texts) X_tsne = TSNE(n_components=2).fit_transform(embeddings) This is how people visualize:

Original Description