We started on AWS like everyone else. Three years in, our monthly bill hit $7,500 and our CFO was giving me that look<\/em> in every meeting. We moved everything to OVH<\/a>, spent four months optimizing with help from GenSpark AI<\/a>, and now we’re paying $2,180\/month for better performance.<\/p>\n\n\n\n Here’s what actually happened.<\/p>\n\n\n\n Picture this: It’s 2AM, I’m getting Slack alerts that queries are timing out, and our main collection just hit 4.8 billion documents. Our carefully-tuned indexes? Useless. Our query times went from “pretty good” to “are you even trying?” in about two weeks.<\/p>\n\n\n\n That’s when we knew we had to do something drastic. The AWS bills were bad enough, but watching our p95 query times climb to 8 seconds? That was the real pain.<\/p>\n\n\n\n Look, AWS is great. Their managed services are top-notch. But when you’re running dedicated MongoDB<\/a> instances and you know what you’re doing, the cost difference is just… brutal.<\/p>\n\n\n\n Check out these numbers:<\/p>\n\n\n\n That’s not a typo. Same hardware – 32 cores, 256GB RAM per node, NVMe storage. The catch? OVH<\/a> doesn’t hold your hand as much. But that’s fine because we were already managing everything manually anyway.<\/p>\n\n\n\n The real kicker? OVH’s vRack private network<\/a> between servers is completely free. AWS was charging us $459\/month just for replication traffic between nodes. That’s $5,500 a year on network transfers alone. For data that never even leaves the datacenter!<\/p>\n\n\n\n When you hit billions of documents, sharding<\/a> isn’t optional. But man, can you screw it up.<\/p>\n\n\n\n Our first attempt:<\/p>\n\n\n\n Seemed logical, right? Every query filters by user_id, so let’s shard on that. Except we have power users who generate 10x more data than normal users. Within a week, some shards were at 900GB while others were chilling at 120GB.<\/p>\n\n\n\n Queries on the hot shards were dying. The MongoDB balancer<\/a> was moving chunks around constantly, making everything worse. It was a mess.<\/p>\n\n\n\n I spent three days reading documentation, blog posts, and MongoDB’s forums. Then I just asked GenSpark<\/a>:<\/p>\n\n\n\n “I have 6B MongoDB documents with user_id and timestamp. Queries filter by user_id and date range. My shards are super unbalanced. What do I do?”<\/p>\n<\/blockquote>\n\n\n\n It came back with a compound sharding key<\/a> strategy I hadn’t even considered:<\/p>\n\n\n\n Hashing the user_id<\/a> distributes power users evenly across shards. The timestamp as secondary key helps with our range queries.<\/p>\n\n\n\n Resharding 6 billion documents took 72 hours of nail-biting, but the results were immediate:<\/p>\n\n\n\n We had 47 indexes across our collections. I knew some were probably useless, but which ones? Going through MongoDB<\/a> logs manually would’ve taken weeks.<\/p>\n\n\n\n So I dumped our index stats<\/a> and slow query logs into a file and asked GenSpark<\/a> to analyze it. Twenty minutes later, it told me:<\/p>\n\n\n\n I’ll be honest – I felt pretty dumb. Some of these were obvious in hindsight. But when you’re managing dozens of indexes across multiple collections, obvious things stop being obvious.<\/p>\n\n\n\n We dropped the useless indexes and reordered the problematic ones. Results:<\/p>\n\n\n\n The whole process took maybe four hours of actual work. Doing this manually would’ve been a multi-week project involving spreadsheets, meetings, and probably a few arguments.<\/p>\n\n\n\n MongoDB’s WiredTiger storage engine<\/a> cache is supposed to be magical. By default it uses 50% of your RAM. We have 256GB per server, so that’s 128GB of cache. Should be plenty, right?<\/p>\n\n\n\n Wrong.<\/p>\n\n\n\n Our cache hit ratio<\/a> was stuck at 72%. That means 28% of reads were going to disk. With billions of documents and NVMe storage, it wasn’t terrible<\/em>, but it wasn’t great either. Queries were averaging 145ms when we knew they could be faster.<\/p>\n\n\n\n I mentioned this to GenSpark<\/a> while troubleshooting something else, and it suggested bumping the cache to 80% of RAM. I was skeptical – doesn’t the OS need memory? But the logic made sense: we’re not running anything else heavy on these boxes, and Linux is pretty good at managing tight memory.<\/p>\n\n\n\n Changed one config line:<\/p>\n\n\n\n Restarted the nodes one by one (because downtime is scary), and watched the metrics:<\/p>\n\n\n\n Sometimes the simple fixes are the best ones. The MongoDB documentation<\/a> recommends 50% but notes you can go higher if your workload benefits from it. Ours definitely did.<\/p>\n\n\n\n This one almost took us down completely. We have 200 application servers connecting to MongoDB. Each one had a connection pool<\/a> of 100 connections because… honestly? That’s what some blog post recommended three years ago and nobody questioned it.<\/p>\n\n\n\n Do the math: 200 servers \u00d7 100 connections = 20,000 connections trying to hit MongoDB.<\/p>\n\n\n\n MongoDB started refusing connections around 15,000. Things got weird. Queries would randomly fail. Connections would hang. Our on-call person (me) was having a bad week.<\/p>\n\n\n\n GenSpark<\/a> suggested dropping the pool size way down:<\/p>\n\n\n\n And bumping MongoDB’s connection limit<\/a>:<\/p>\n\n\n\n Now we’re at about 5,000 total connections across all apps. MongoDB’s CPU usage dropped 40%. Connection errors disappeared. Query latency improved 35%.<\/p>\n\n\n\n Turns out you don’t need a massive connection pool per instance. You just need enough to handle your concurrent queries. Who knew? (Everyone who actually read the documentation properly, probably.)<\/p>\n\n\n\n After dealing with all this, I realized we were monitoring way too much noise. Most metrics don’t matter until they’re already broken. We rebuilt our primary dashboard to show just five things:<\/p>\n\n\n\n Everything else is in a secondary dashboard that we check during incidents. But this five-metric view? It tells us 95% of what we need to know at a glance.<\/p>\n\n\n\n GenSpark<\/a> helped us design this after I fed it six months of incident logs and asked “which metrics actually predicted our outages?” Turns out most of them didn’t. These five did.<\/p>\n\n\n\n Let me be real about the GenSpark<\/a> thing. It didn’t write our code. It didn’t magically fix our database. But here’s what it did:<\/p>\n\n\n\n Index optimization:<\/strong> Would’ve taken me two weeks of analyzing logs, testing changes, measuring results. With GenSpark analyzing patterns? Four hours.<\/p>\n\n\n\n Schema redesign:<\/strong> We were restructuring our product catalog. Normally this is weeks of research, testing different approaches, measuring performance. GenSpark gave us three solid approaches with pros\/cons in minutes. We tested the best one, it worked, done in three days.<\/p>\n\n\n\n Query optimization:<\/strong> Our analytics queries were slow. I’d spend a day staring at EXPLAIN output<\/a> trying to figure out why. Now I paste the EXPLAIN into GenSpark, it tells me “you’re doing a collection scan on 2M documents, add this index,” and I’m done in an hour.<\/p>\n\n\n\n Connection tuning:<\/strong> This would’ve been pure trial and error. Test a pool size, monitor for a day, try another. GenSpark gave us a sensible starting point based on our query patterns and we only had to tweak it once.<\/p>\n\n\n\n Total time saved on this project? About 7-8 weeks of work compressed into a week and a half. That’s not hype – that’s me being able to ship this whole migration in four months instead of six.<\/p>\n\n\n\n The key is knowing what to ask and when. GenSpark isn’t magic. But it’s like having a senior database engineer available 24\/7 to sanity-check your ideas and point out things you missed.<\/p>\n\n\n\n Aggregation queries that lie:<\/strong> MongoDB will happily try to aggregate<\/a> billions of documents in memory, fail silently when it hits the memory limit<\/a>, and return incomplete results. Always use The balancer is chaos:<\/strong> When resharding, the balancer<\/a> moves chunks between shards automatically. Sounds great! Except it hammers your cluster and makes everything slow. Schedule balancing windows<\/a> or you’ll get surprise performance hits at random times.<\/p>\n\n\n\n Backups at this scale are weird:<\/strong> Our full backup is 14.5TB. Restoring from backup takes 8 hours. We test this quarterly because the one time you don’t test is the one time you’ll need it and discover it’s broken.<\/p>\n\n\n\n OVH support<\/a> is… different:<\/strong> They’re helpful, but you need to know what you’re doing. If you’re used to AWS holding your hand, OVH will make you Google some stuff. That’s the tradeoff for paying a third of the price.<\/p>\n\n\n\n If I could go back and redo this whole thing?<\/p>\n\n\n\n Just to close this out with some actual data:<\/p>\n\n\n\n Running 6 billion documents at Avluz.com<\/a> taught us that scaling isn’t about perfect architecture or having infinite budget. It’s about making smart tradeoffs, knowing when to ask for help (even from AI<\/a>), and being willing to spend a weekend resharding your database when you have to.<\/p>\n\n\n\n Also, monitor your cache hit ratio. Seriously.<\/p>\n","protected":false},"excerpt":{"rendered":" So here’s the thing about running a database at scale – nobody tells you about the weird stuff until you’re already knee-deep in it. At Avluz.com, we crossed 6 billion…<\/p>\n","protected":false},"author":1,"featured_media":277,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_yoast_wpseo_title":"Scaling MongoDB to 6 Billion Documents: Our Proven Tips","_yoast_wpseo_metadesc":"Discover how we managed 6 billion MongoDB documents, optimized costs, and scaled efficiently. Read our journey and learn actionable strategies today!","_yoast_wpseo_canonical":"","footnotes":""},"categories":[128],"tags":[],"class_list":["post-270","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"yoast_head":"\nThe “Oh Crap” Moment<\/h2>\n\n\n\n
Why We Ditched AWS for OVH<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
![\"\"](\"https:\/\/avluz.com\/blog\/wp-content\/uploads\/2025\/12\/image-1.png\")
<\/figure>\n\n\n\n\n
Our Sharding Disaster (And Recovery)<\/h2>\n\n\n\n
<\/code><\/pre>\n\n\n\nsh.shardCollection(\"avluz.events\", { user_id: 1 })\n<\/code><\/pre>\n\n\n\n<\/figure>\n\n\n\n
![\"\"](\"https:\/\/avluz.com\/blog\/wp-content\/uploads\/2025\/12\/image-2.png\")
<\/figure>\n\n\n\n\n
<\/code><\/pre>\n\n\n\nsh.shardCollection(\"avluz.events\", { \n user_id: \"hashed\", \n timestamp: 1 \n})\n<\/code><\/pre>\n\n\n\n\n
The Index Audit From Hell<\/h2>\n\n\n\n
\n
<\/figure>\n\n\n\n
![\"\"](\"https:\/\/avluz.com\/blog\/wp-content\/uploads\/2025\/12\/image-3.png\")
<\/figure>\n\n\n\n\n
That Time We Melted Our Cache<\/h2>\n\n\n\n
<\/code><\/pre>\n\n\n\nstorage:\n wiredTiger:\n engineConfig:\n cacheSizeGB: 205\n<\/code><\/pre>\n\n\n\n\n
Connection Pool Shenanigans<\/h2>\n\n\n\n
<\/code><\/pre>\n\n\n\nconst client = new MongoClient(uri, {\n maxPoolSize: 25, \/\/ was 100\n minPoolSize: 5,\n maxIdleTimeMS: 30000\n})\n<\/code><\/pre>\n\n\n\n<\/code><\/pre>\n\n\n\nnet:\n maxIncomingConnections: 5000\n<\/code><\/pre>\n\n\n\nThe Dashboard That Actually Matters<\/h2>\n\n\n\n
<\/figure>\n\n\n\n
![\"\"](\"https:\/\/avluz.com\/blog\/wp-content\/uploads\/2025\/12\/image-4-1024x572.png\")
<\/figure>\n\n\n\n\n
How AI Actually Saved Us Weeks<\/h2>\n\n\n\n
The Weird Stuff Nobody Warns You About<\/h2>\n\n\n\n
{ allowDiskUse: true }<\/code> on big aggregations. Always.<\/p>\n\n\n\nWhat We’d Do Differently<\/h2>\n\n\n\n
\n
The Numbers That Matter<\/h2>\n\n\n\n
\n