Hamming: The Guardian of AI - Policing LLMs for Reliable Outputs↕️

In partnership with

This is The Startup Breakdown, the only newsletter looking into the new Y Combinator companies that you need to know. By joining this growing community of hundreds of founders, investors, and other startup aficionados (think i spelled that right?), you're getting a firsthand look at the companies that may be the next Airbnb, Coinbase, or Reddit.

Hamming: The AI Evaluation Platform Keeping LLMs in Check

BACKGROUND:

We have our first member of the summer cohort!

There are still a couple of weeks of interviews and invitations, but YC has gone ahead and wired the $125K SAFE to Hamming, an LLM to police other LLMs. About as meta as it gets.

• Side note, thinking about creating a resource further explaining terminology, the YC structure, how to think about the math, etc. Would you be interested? Anyone who replies will get it for sure, even if I don’t end up making it an official subscriber resource

If you’ve used ChatGPT, Claude, Gemini, or any other AI tool, you’ve probably seen the “our model might hallucinate or provide incorrect information. always fact check” disclaimers.

If you’ve used these tools enough, you’ve probably even experienced this hallucination yourself, like the time that Google’s Gemini told me my friend that I should drink 2 quarts of my his own urine to quickly pass kidney stones.

As more companies build their own LLMs, whether to compete with OpenAI or to use internally with proprietary data, the importance of AI evaluation and monitoring these outputs will only become more and more important.

Hamming is building the playground for AI evaluation, allowing developers and engineers to experiment with prompts, models, and architecture in a fraction of the time and cost.

NUMBERS:

Traction:

• Customers from industries like legal, medical, financial services, and more

• Names include Fora, Inkly, and Intuitive Systems

Market Size:

• TAM: $100 billion, growing at a nearly 40% CAGR

• SAM: $20 billion

• SOM: $100 million

Competition:

• Hugging Face

• Vellum AI, Scale AI, Mosaic ML, Comet ML, Arthur AI

Team:

• Marius Buleandra, CTO: Engineering Manager at Anduril, founding engineer at Spell (acquired by Reddit)

• Sumanyu Sharma, CEO: Head of Data at Citizen, Senior Staff Data Scientist at Tesla

Risks:

• LLMs to grade LLMs: There are genuine cost and time benefits to using AI to police AI, but though it might further reduce the already small error rate, their own LLM is still susceptible to the same hallucinations that they’re attempting to prevent for others, and no matter how successful their model is, they’ll always face the obstacle of associations

• Evaluation model: Though there are existing benchmarks and tests meant to evaluate models, there have been critiques of the methods as they’re often arbitrary and don’t necessarily test the true “intelligence” of models

• Regulation: The government has been increasingly sceptical of AI and has already explored ways to monitor and evaluate companies’ LLMs themselves, and there’s a chance Uncle Sam decides to get his greasy paws all over a company like Hamming that is doing this work

What I like:

• Cross-role: Hamming's tools are beneficial to just about every position in the firm, not just engineers, making the user experience great and the appeal to enterprises even more attractive as they have so many potential beneficiaries of a contract for AI evaluation services

• Industry-agnostic: Similarly, the company already counts industries across the spectrum as customers, making the market for their services even greater

• Massive market: The market is huge, and should the cost and time benefits continue to hold (or even become more prominent) and the error rate is proven to be lower, this company could genuinely count every business in the country (and beyond) as a customer

Opportunities:

• Become the standard evaluation benchmark: As mentioned, there is no universal AI evaluation system, as each of the evaluation services and research teams has its own benchmarks and criteria. Even if Hamming just develops the state of the art grader, they will stay around. Perhaps launch a free version to get the name out there?"

• Consulting: Arguably the quickest way to expand their potential customer pool is through helping more and more companies to learn to implement AI and the various use cases they can find for it. Becoming a consultant for these companies, and publishing case studies for inspiration, is another area for growth, both directly and indirectly

• Partnerships with LLM providers: Though it’s likely to take a bit more traction and proof of product, shaking hands with Sam Altman or Sundar Pichai to agree on letting these companies integrate Hamming’s evaluation tools into the companies’ LLMs would be a massive stamp of approval and one of those oversized checks that you get when you win a raffle

Hamming has the opportunity to be one of the most important infrastructure providers in the entire AI space. If it can continue to be cheaper, faster, and most importantly, more reliable, than human grading, their only ceiling is the number of businesses using AI, and that number should only continue to skyrocket.

You heard it here first, folks. Hamming is here to stay.

Links of the week:

Learn how to make AI work for you.

AI breakthroughs happen every day. But where do you learn to actually apply the tech to your work? Join The Rundown — the world’s largest AI newsletter read by over 600,000 early adopters staying ahead of the curve.

1. The Rundown’s expert research team spends all day learning what’s new in AI

2. They send you daily emails on impactful AI tools and how to apply it

3. You learn how to become 2x more productive by leveraging AI

Subscribe with one click.

How am I doing?

I love hearing from readers, and I’m always looking for feedback. How am I doing with The Startup Breakdown? Is there anything you’d like to see more or less of? Which aspects of the newsletter do you like most?

Cheers to another day,