Verdict: Scaling model serving from tens of thousands to millions of models is not solved by buying more GPU or vertical-sharding your database. It is solved by separating metadata from binary artifacts, pre-computing search tokens at insert time, routing reads across specialized replica set members, and running two layers of Kubernetes autoscaling (pods + nodes). The team behind the Hugging Face Hub proved this in production — growing from 20,000 models to 3 million in a few years while keeping search latency low and cost contained. The patterns are reproducible by any team running an ML model registry or inference platform.
- The single biggest scaling lever: separate metadata (database) from model artifacts (object storage). Each scales independently.
- Pre-compute search tokens at INSERT time, never at query time — this is the single most impactful search optimization at scale.
- Route reads by workload: strong-consistency to primary, aggregations to secondaries, heavy analytics to a hidden replica.
- Two-layer autoscaling: HPA/KEDA for pods + a node autoscaler (like Cast AI) for capacity. One layer alone is not enough.
- Migrating from HPA to KEDA lets you scale on actual request rate, not just CPU/memory.
- Pricing/limits scale fast — last checked 2026-07-31.
What Does It Take to Serve 3 Million ML Models at Once?
Serving 3 million models is not an inference problem first — it is a metadata and search problem first. The Hugging Face Hub hosts 3 million public models, nearly 1 million datasets, and serves 14 million users, with approximately 30% of the Fortune 500 integrating the platform into their AI workflows (MongoDB case study). That scale demands an architecture where model discovery stays fast even as the catalogue grows 150x in a few years.
The key insight: at 20,000 models, any query is fast — even without an index. At 3 million, the same naive approach collapses. The architecture that works separates concerns aggressively, pre-computes expensive operations at write time, and scales each layer independently.
Why Does Separating Metadata from Model Artifacts Matter?
Separating metadata from binary model files is the foundational decision that makes everything else possible. In the Hugging Face architecture, MongoDB Atlas stores all metadata — users, repositories, model entries, dataset descriptions, billing data, access controls — while the actual model weights, tokenizer files, and configuration files live in cloud object storage like AWS S3 (MongoDB case study).
This means:
| Layer | Stores | Scales By |
|---|---|---|
| Metadata DB (MongoDB Atlas) | Model names, descriptions, tags, user accounts, org data, access rules | Query volume, document count |
| Object storage (AWS S3) | Model weights, tokenizer files, config files | Storage capacity, download throughput |
| Compute (Kubernetes) | API pods, search services, inference endpoints | Request rate, CPU/memory |
Each layer optimizes for its own workload. You can scale object storage for download throughput without touching the metadata database. You can scale the database for search query volume without adding compute capacity. And you can scale compute for request rate without re-architecting storage.
This separation also means your database documents stay small — they hold pointers to S3 objects, not the objects themselves. Small documents mean faster reads, better cache utilization, and more documents per MongoDB page.
For any team building an ML model registry: if you are storing model weights inside your primary database, stop. Move them to S3 (or equivalent) and keep only metadata in the database. This single change removes the most common scaling bottleneck.
How Do You Keep Search Fast at Scale?
Search is where most platforms break. The Hugging Face team's evolution from working search to scalable search illustrates the three decisions that matter:
1. Pre-compute search tokens at INSERT time, not query time
When a user publishes a model like meta-llama/Llama-3.1-8B, the system splits this long name into shorter tokens — meta, llama, 3.1, 8b, meta-llama, llama-3.1 — and stores them as a pre-computed array in the MongoDB document at insert time. This is called a searchTokens array.
Why this matters: At query time, the search engine does a lookup against pre-computed tokens — no tokenization, no parsing, no regex. The expensive work happens once, at insert. The query is cheap.
2. Use a denormalized read-optimized collection
The search does not hit the primary write collection. It hits a separate, denormalized copy optimized for reads and listings. This means search traffic never competes with write traffic for resources.
3. Migrate from regex queries to Lucene-based full-text search
The original search used MongoDB's regex operator to find models matching a query string. Regex works fine at 20,000 models but degrades badly at scale — regex does not use indexes efficiently and latency grows with document count.
The solution: MongoDB Atlas Search, which runs Apache Lucene as a separate process alongside MongoDB. You query it through MongoDB's unified API using the $search aggregation operator. Lucene provides autocomplete and full-text capabilities that scale to millions of documents without the latency penalty of regex (MongoDB Atlas Search docs).
// Before (regex — does not scale):
db.models.find({ searchTokens: { $regex: query } })
// After (Atlas Search — Lucene-backed):
db.models.aggregate([{
$search: {
index: "model_search",
autocomplete: { query: query, path: "searchTokens" }
}
}])
The results are then sorted by a trending score — downloads and likes from the last seven days, recalculated every five minutes. This keeps the most relevant model at the top of search results even as the catalogue grows.
The lesson for builders: If your search latency is growing with your dataset size, you are doing tokenization or matching at query time. Move it to insert time and switch from regex to a Lucene-backed full-text search engine.
How Does a 7-Node MongoDB Replica Set Route Workloads?
At 14 million users, the database cannot be a single primary. Hugging Face runs a 7-member MongoDB replica set with specialized routing (MongoDB case study):
| Node Type | Count | Role | What Runs Here |
|---|---|---|---|
| Electable primary candidates | 3 | Handle writes + strong-consistency reads | Inserts, updates, deletes; queries that need the latest data |
| Read-only secondaries | 3 | Serve heavy read traffic | Searches, listings, cached aggregations, change streams |
| Hidden analytical node | 1 | Invisible to the driver, no production traffic | Heavy reporting queries, ad-hoc analysis, experimental queries |
The pattern is simple: the primary should focus on what only the primary can do — writes and strong-consistency reads. Everything else gets pushed to other machines.
What goes where?
- Queries that do not need the latest data run on secondaries. Only queries requiring strong consistency stay on the primary.
- Complex aggregations (scanning large datasets, sorting, grouping, transforming) run on secondaries — they are heavy and should never block the primary.
- Change streams (reacting to data changes in real time for cache validation, syncing to data warehouses like AWS Redshift, or event-driven workflows) run on secondaries.
- Ad-hoc and reporting queries run on the hidden analytical node, completely isolated from production traffic.
All secondaries continuously tail the primary's oplog, keeping the cluster synchronized. The hidden node replicates too, but the MongoDB driver never routes queries to it — you connect directly only for analytical workloads.
The lesson: If your primary is doing reads that do not need strong consistency, you are leaving capacity on the table. Route them to secondaries. At any scale above a few thousand concurrent users, this is the difference between a responsive system and one that falls behind during traffic spikes.
When Do You Need Database Sharding?
A 7-node replica set keeps a full copy of the data on every node. That works to a point — but at 14 million users and 3 million models with continued growth, a single replica set will not be enough. The next step is sharding: splitting the dataset across multiple replica sets (shards), each holding only a portion of the data (MongoDB sharding docs).
| Dimension | Replica Set | Sharded Cluster |
|---|---|---|
| Data per node | Full dataset on every node | Only a shard (subset) of data per node |
| Scales | Reads (distributed across secondaries) | Reads, writes, CPU, memory, storage — all horizontally |
| Adding capacity | Increase node size | Add more shards; balancer redistributes |
| Complexity | Lower | Higher — shard key selection is critical |
Sharding lets you scale CPU, memory, storage, reads, and writes by adding more shards. The MongoDB balancer automatically redistributes data across shards. The critical decision is the shard key: if chosen poorly, queries may hit every shard (scatter query), negating the performance benefit. If chosen well, most queries route to a single shard.
Sharding is not something to start with. It is the architecture you migrate to when replica sets hit their ceiling. The entry cost is real (shard key selection, balancer tuning, migration), but the ceiling is much higher — you can keep adding shards indefinitely.
How Does Two-Layer Kubernetes Autoscaling Work?
The Hugging Face Hub runs on Kubernetes with two independent layers of autoscaling:
Layer 1: Pod autoscaling (HPA → KEDA)
The Horizontal Pod Autoscaler (HPA) scales the number of API pods based on CPU and memory thresholds. The Hub deployment scales from 10 to 500 pods depending on traffic — new pods start automatically during spikes and scale back down when traffic drops (Kubernetes autoscaling docs).
However, HPA has a limitation: it only sees CPU and memory. A pod can have low CPU utilization but a high request queue — HPA will not scale because the CPU threshold is not hit. The team is migrating to KEDA (Kubernetes Event-Driven Autoscaling), a CNCF-graduated project that scales based on real application metrics:
- Request per second
- Event loop utilization
- Queue depth
- Any custom metric or event source
KEDA works alongside HPA — it does not replace it. KEDA creates and manages HPA objects on your behalf, feeding them external metrics that HPA alone cannot see. It can also scale to zero, saving cost when there is no traffic (KEDA docs).
Layer 2: Node autoscaling (Cast AI)
When HPA/KEDA wants to add pods but the Kubernetes cluster has no free nodes to schedule them on, the second layer kicks in. Cast AI adds new nodes automatically when pods are pending due to capacity constraints. Cast AI is a commercial Kubernetes node autoscaler that handles node provisioning, bin-packing, and spot-instance orchestration — teams have reported 30–50% compute cost reductions through automated rightsizing (Cast AI blog).
| Layer | Scales | Trigger | Tools |
|---|---|---|---|
| Pod layer | Number of API pod replicas | CPU/memory (HPA) or request rate, queue depth (KEDA) | HPA, KEDA |
| Node layer | Number of Kubernetes nodes | Pending pods with no capacity | Cast AI, Cluster Autoscaler, Karpenter |
The two layers work together: pods scale first (fast, seconds), then nodes scale when capacity runs out (slower, minutes). Without both layers, either pods cannot scale (no node capacity) or nodes are wasted (pods do not fill them).
What Can You Apply to Your Own ML Platform Right Now?
You do not need to serve 3 million models to benefit from these patterns. Here is what any team building an ML model registry, inference platform, or AI model catalogue can apply:
- Separate metadata from model storage now. If model weights are in your database, move them to S3. This is the highest-leverage change and the easiest to get right early.
- Pre-compute search tokens at insert time. If you are tokenizing model names or descriptions at query time, you are paying that cost on every search. Move it to insert time and store the tokens in an array.
- Migrate from regex to Lucene-backed search. If your search uses regex or simple string matching and latency is growing with your catalogue, switch to a Lucene-based search engine (MongoDB Atlas Search, Elasticsearch, OpenSearch). Lucene scales to millions of documents with autocomplete and relevance scoring.
- Read-rout your replica set. Do not send all reads to the primary. Route search traffic and non-time-sensitive reads to secondaries. Route heavy analytics to a hidden replica. Keep the primary focused on writes.
- Set up two-layer autoscaling. Use HPA (or KEDA for event-driven scaling) for pods, and a node autoscaler (Cluster Autoscaler, Karpenter, or Cast AI) for capacity. Neither layer alone is sufficient.
- Plan for sharding before you need it. When your replica set hits 70–80% capacity (CPU, storage, or connection count), start the shard key evaluation process. Do not wait until you are at 100% — migration under load is risky.
How Does This Compare to Other ML Serving Scale Stories?
Hugging Face is not the only team that has hit this scale, but their architecture is among the most well-documented. The patterns hold across platforms:
| Pattern | Hugging Face Hub | Databricks Model Serving | Generic Enterprise ML |
|---|---|---|---|
| Metadata/artifact separation | MongoDB Atlas + S3 | Internal metadata store + cloud storage | Database + S3/GCS |
| Search at scale | Lucene (Atlas Search) | Internal search/routing | Elasticsearch/OpenSearch |
| Replica routing | 7-node specialized replica set | Distributed servingfleet | Multi-AZ replica sets |
| Pod autoscaling | HPA → KEDA | Custom auto-scaling | HPA with custom metrics |
| Node autoscaling | Cast AI | Managed cluster autoscaler | Cluster Autoscaler/Karpenter |
| Horizontal DB scaling | Sharding (in progress) | Sharded serving clusters | Sharding at high scale |
The shared thread: every team at this scale uses metadata/artifact separation, a Lucene-grade search engine, read-routed replicas, and two-layer autoscaling. The specific tools vary (Cast AI vs Karpenter, Atlas Search vs Elasticsearch), but the architecture patterns are identical.
What This Means for You
If you are building or scaling an ML platform — a model registry, an inference service, an AI application marketplace — these architecture patterns are your roadmap:
- Early stage (under 10K models): Focus on the metadata/artifact separation. Everything else can be naive — regex search, single-primary reads, basic autoscaling will work.
- Growth stage (10K–500K models): Pre-compute search tokens, migrate from regex to Lucene-grade search, route reads across secondaries, set up HPA + a node autoscaler.
- Scale stage (500K+ models): Plan for sharding, migrate from HPA to KEDA for event-driven scaling, add a hidden analytical node, and monitor P99 latency aggressively (not just P50).
The teams that scale successfully do not buy their way out with more infrastructure — they separate concerns, pre-compute expensive operations, and route workloads to the nodes best suited for them. That architecture holds at 20K models and at 3 million.
FAQ
Q: What is the biggest scaling mistake teams make with ML model serving? A: Storing model weights (binary artifacts) inside the primary database instead of object storage. This means every database read, backup, and replication cycle deals with gigabyte-sized documents. Moving artifacts to S3 while keeping only metadata in the database is the single highest-leverage architectural change you can make.
Q: Why pre-compute search tokens at insert time instead of query time? A: Tokenization at query time means the cost grows linearly with query volume — every search re-parses model names. Pre-computing tokens at insert time means the cost is paid once, and every query is a fast lookup against a pre-built token array. At 3 million models with 14 million users, this is the difference between instant search and multi-second latency.
Q: What is the difference between HPA and KEDA for Kubernetes autoscaling? A: HPA (Horizontal Pod Autoscaler) scales pod replicas based on CPU and memory thresholds only. KEDA (Kubernetes Event-Driven Autoscaling) is a CNCF-graduated project that extends HPA by feeding it external metrics — request rate, queue depth, Kafka consumer lag, or any custom metric. KEDA also supports scaling to zero, which HPA does not. KEDA creates and manages the HPA objects on your behalf.
Q: When should I consider MongoDB sharding instead of just scaling a replica set? A: Start evaluating sharding when your replica set approaches 70–80% of CPU, storage, or connection capacity. Sharding splits data across multiple replica sets (shards), each holding a subset of the data. The critical decision is the shard key — a poor choice causes scatter queries that hit every shard. Plan the shard key before you need to migrate; migrating under load is risky.
Q: What is a hidden MongoDB replica set member and why use one? A: A hidden node is a replica set member that replicates data from the primary but is invisible to the MongoDB driver — no application traffic is routed to it. You connect to it directly for heavy analytical queries, reporting, and ad-hoc analysis. This isolates expensive workloads from production traffic entirely, so a complex aggregation never impacts user-facing query latency.
Q: Can I apply these patterns on a smaller scale (under 100 models)? A: Yes — the metadata/artifact separation is valuable at any scale. If you have 100 models, separating metadata from S3 storage keeps your database fast and your backups small. You likely do not need Lucene-grade search, specialized read routing, or sharding until you cross 10K+ models and have measurable search latency growth.
Discussion
0 comments