Building an AI research lab

Source: https://www.youtube.com/watch?v=MGouk8W51v0&list=PLaqC3GACblSs&index=5

AI generated summary of the linked source. Posted anonymously. Transcripts are published only for public links, never for uploads, documents or pasted text.

Your library

The AI link is plain markdown. Paste it into any assistant and it can read the whole public entry.

Summary

Gabe, co-founder of Harvey (legal AI), explains how an application layer company competes with frontier AI labs by leveraging the broader ecosystem instead of building everything in-house. The playbook: build domain-specific benchmarks using synthetic data generated by experts, post-train open-source models with Neo lab partners, then serve them through rigorous production infrastructure with continuous evaluation.

Core ideas

One context card plus 9 ideas

The budget gap is real: Frontier labs have more money, talent, compute, and data. Application layer companies win by using the frontier ecosystem rather than trying to out-build the labs directly.
Benchmarks come first: Harvey built three data sets this year (Legal Agent Bench, a contracting negotiation set, and a large diligence data set with 80 million token data rooms) because you can't train or serve models without a way to measure them.
Synthetic data solves the privileged data problem: Since Harvey can't train on client legal data, domain experts (lawyers) use coding-model-style workflows to generate realistic synthetic data rooms, planting known issues into a rubric first, then generating contracts around them so outputs can be checked against ground truth.
Open sourcing benchmarks builds trust and quality: Publishing data sets, even controversially, surfaces bugs through community pull requests and gets labs to benchmark against them, similar to how ImageNet or MNIST worked in earlier ML eras.
Open-source base models are now good enough to post-train: Models like Kimi 3, GLM 5.2, NeMo-Megatron, and others have gotten strong enough that post-training on a narrow domain can reach frontier-level performance for that task.
Working with multiple Neo labs multiplies learning: Harvey partners with several post-training providers (Fireworks, Base10, Ngram, Trajectory, Applied Compute) because each has different bets, infrastructure, and expertise, and Harvey has more research ideas than internal bandwidth.
Production infrastructure must exist before post-training: Harvey operates in 60 countries with multiple product surfaces and model preferences, so they built a full evaluation pipeline (generic benchmarks, human side-by-sides, critical user journey tests, AB testing, uptime and cost tracking) that applies whether a model is post-trained or not.
Start with simple open-source swaps and routing: Before deep post-training, find low-risk spots (like citation generation) to swap in cheaper open-source models, then move to query-level routing between open and closed models.
The real end game is continual learning per client: Harvey's goal isn't one best legal model, but a system where every law firm's AI improves from their own client work while still protecting privileged data.

Quotes

“This talk is going to be our high-level playbook for doing this.”
Gabe

“If you don't have a good benchmark, you can't train models, and if you can't train models, you don't need to serve them in production.”
Gabe

“A big problem we've solved over the past four years is we operate in 60 countries.”
Gabe

“And then most importantly, we had Elon retweet it.”
Gabe

“But if we win on this budget with this team we'll have changed the game.”
Gabe

“We think in the future every company is going to need to become an AI company and figure out some version of this playbook.”
Gabe

Resources

Links to the tools, products, repos or reading this entry mentions. Found by a web search of what the entry names, so you can go and use them.

Anyone can run this once. After that the links are saved on this entry and shown to everyone.

Remix this knowledge

Turn this entry into something you can post. Pick a format and get a fresh, original rewrite.

Remixes are AI generated, original wording, and free to reuse.

Talk it through

Open a round table on this entry and argue with other people about what it means.

Start a round table

Drop your own

Paste any video, podcast or audio link and get the transcript, the core ideas and the quotes back in one pass.

Try Drop