Show HN: I scraped 200M Shopify products to build a search engine

Hi HN! In December I launched an MVP for Agora here: https://news.ycombinator.com/item?id=38635695 After posting, we got thousands of users and hundreds of comments with valuable feedback from the community. I spent a couple sleepless nights frantically pacing around my room trying to keep the product live and, relatively, performant. After getting some sleep, I got back to work to make the product better. A few updates: 1. We've grown from 25 million to 200 million products on Shopify and WooCommerce. The team at WooCommerce reached out after the HN launch to help us figure out how to index their stores. Similar to Shopify, we found that there’s a public file available for all stores that use Wordpress and WooCommerce at [Base URL]/wp-json/wc/v1/products. For example, the file for Good Works Tractors is available here: https://www.goodworkstractors.com/wp-json/wc/store/v1/produc... So I bought a list of 3.5 mil