A while ago, the Taalas team etched Llama 3 8B (4-bit) directly into silicon and got a single card to output 16,000 tokens per second.
Much of the industry discussion circles around the same doubt: can a hard-wired design like this actually hold a SOTA-class model? If you want an 80B+ workhorse model, do you need to chain a dozen cards? And if you need multi-card interconnect anyway, how much advantage is left over a regular high-memory GPU?
From the experience of building agent systems, this debate misses another, very large slice of real demand: high-frequency inference on very fast small models (4B–8B), at the edge and in the orchestration layer of systems.
1. Auto Router: hundreds of millions of low-latency routing decisions a day
Start with the most familiar product experience. ChatGPT’s default mode (Auto) decides in the background whether a request goes to Instruct, Thinking, or Pro.
After a user sends a prompt, ideally they see the response start within 0.1 seconds. To hit that latency, the Auto Router can’t rely on a slow large model, and it can’t rely only on keywords or shallow semantic matching either. The backend usually needs a purpose-trained routing model, or a 4B / 8B small model, to judge how hard the task is.
A rough order-of-magnitude estimate:
- Say an active user sends 100 requests a day;
- The platform has 1 million daily active users;
- That’s 100 million routing inferences a day.
This kind of inference has a distinct profile: the input is usually the full context, the output is often just a few tokens (a class label or a route), it is extremely latency-sensitive, and the model weights are relatively fixed.
2. The intermediate summarization layer in search and tool calls
Now look at the pipeline of AI search products like Perplexity. It retrieves, reads dozens of pages, and keeps streaming progress within a few seconds. That doesn’t come from making a flagship model swallow 20 full web pages of HTML word by word.
In a real agent search pipeline:
- The main model produces the search intent or query and calls the Web Search tool;
- Deciding which pages to fetch next, and how to extract the most relevant passages, shouldn’t be handed to the expensive, slow main model. Passing raw pages in and out burns a lot of tokens and drags out time-to-first-token;
- A better architecture lets a fast small model handle filtering, extraction from fetched results, and intermediate summaries — feeding live progress to the frontend, then handing a high signal-to-noise summary back to the main model.
Every concurrent search request fans out into many small-model calls. That concurrency demand is enormous too.
3. Large-scale data cleaning and automated trial and error
In AI R&D, data quality often sets the ceiling, and huge corpora need to be cleaned and labeled again and again.
Large-scale cleaning used to rely mainly on rules and traditional classifiers, because the throughput and cost of general-purpose LLMs couldn’t support cleaning everything — they were mostly used on small curated sets. If there were dedicated small-model hardware that is extremely cheap and runs at tens of thousands of tokens per second, pipelines that used to be unaffordable could become routine infrastructure:
- Semantic filtering, rewriting, and quality scoring over large raw corpora;
- Cheap, high-frequency trial and error in synthetic data, eval grading, and agent trajectory filtering.
Takeaway
Most of the conversation around Taalas focuses on one path — “can it fit an 80B+ flagship model?” — and overlooks that inside real AI products and agent systems, fast, fixed small models in the 4B–8B range are already a huge market of hard demand: request routing, web extraction and summarization, data cleaning, and online guardrails.
These steps don’t necessarily need the latest flagship reasoning, but they badly need low latency, high throughput, and low unit cost. That may also be why hardware teams like this can raise serious money while the reasons rarely get spelled out in public coverage: you only really feel this small-model inference bottleneck once you’ve built a complete agent system yourself.