Meet Notes 2026-10-15-jdu-ifee-mes

Recap: decisions, open questions, and action items

Joe and Reid, Oct 15. The call ended, or I lost the connection. This is where things stand.

1. Experiment 1

Decided

• 25–30 documents across 8 types. 27 are proposed, from deals we already have answers for (Reid to approve the list).
• Build the single-document path first, reusing our entity recognition as it is. Change settings, not code (Joe).
• Unsure documents are emailed to Reid and Joe. It runs on a server. We look at results right after.
• Four arms, each a different way to build the co-worker: one agent with tools (the control), a fixed workflow, a small team with a checker, and a general agent with a workspace.

Open

• Confirm the tool for each arm. Ranked picks: Pydantic AI · LangGraph + LangExtract · LangGraph · OpenHands. Joe leans toward LangGraph.
• Should test emails be sent, or just saved?
• When to resume the single-document build (paused at Joe's request).

2. What the research says

• No design wins everywhere. A fixed workflow is the cheap baseline; agents add a few points on messy documents at higher cost.
• The checker matters more than how many agents there are. Check with code (is the quote really on the page) and a different AI model.
• Workspace agents like OpenClaw are worth testing, but only fenced in.
• For the experiment: the same AI model in every arm first, run each document 3 times, and also score cost per correct match.

3. Hardware

Decided direction

• Don't buy the big machine yet. First buy: a 2-card Dell workstation, about $35–40k, with lots of processor cores and room for more cards.
• The Dell connection saves about 25% per card (cards now cost $16k retail).

Open

• The county data plan. Reid: buy access to structured data and process only what we need. Keegan: buy whole county archives (1–10 TB each) and structure everything. This decides how much more hardware and storage we need.
• Get a real county sample and measure pages per hour before buying more.

Action items

County-scale data: what it means for the hardware

Short answer: the 2-card workstation still makes sense as the first buy either way. But if we take on whole counties, three things have to change: don't run the big AI on every page, plan for storage, and rent extra power for the one-time catch-up instead of buying for it. Which plan we follow (Reid's or Keegan's) decides how big this gets. Settle that before buying more than the first machine.

How big is "a county"? (my rough estimates; check with a real sample)

• A scanned page is about 50–200 KB, so 1–10 TB is roughly 10–100 million pages per big county. At 3–5 pages per document, that's several million documents.
• Running a mid-size AI model over every page of a 10-million-page county on the 2-card machine would take weeks, and a 10 TB county several times that. Reading the scans (OCR) alone is days on a good processor.
• Compare: our current plan is hundreds of documents a month. That's a jump of about 10,000×.

The two plans

How to get the most out of the hardware either way

1. Funnel, don't firehose. Every page gets the cheap steps: read the text, make it searchable, and tag names, tracts and leases with the entity recognition we already have, which is small and fast. Only pages that matter to a deal or a question go to the big AI or the agent. That's the difference between weeks and hours.
2. Work queue with many workers. Put every page or document on a queue (the Command Center already has a job-queue table), and run many workers at once. The AI server (vLLM) is fastest when it handles many requests at once, so keep it fed.
3. Split the work by what does it best. Reading scans runs on the main processor, so buy a machine with lots of processor cores, not just graphics cards. The AI steps run on the graphics cards. Run both at the same time.
4. Use small models for the easy steps. "What kind of document is this?" doesn't need the big model. Save the big model and the agents for matching and judgment.
5. Rent for the catch-up, own for the daily flow. A county backfill happens once and splits easily across many machines. Renting cards by the hour ($1.50–3 per card) finishes it in days instead of tying up our machine for months.

What this changes in the purchase

• Still buy the 2-card Dell first. Ask for a processor with many cores and room to add cards (the Pro Precision 9 T6).
• If counties are coming, add storage to the Dell quote (a storage box, with backups).
• Before buying more: run one sample of a county (say 10,000 pages) through the 2-card machine and measure pages per hour. That turns these guesses into a real plan.

The options for each arm, ranked

Ranked against our success metrics: (1) wrong without asking, the most important; (2) getting it right (type, deal, facts); (3) finishing on its own; (4) time and cost. They also have to run on our own model later, reach Teams, and reuse our existing tools. These are predictions from the research. The experiment decides for real.

Arm 1: one agent with tools (control)

1. Pydantic AI. Works with any AI model, including ours later. It checks every answer against a strict format before accepting it, which helps "wrong without asking". Uses our tools through the standard connector (MCP). Free, active.
2. Microsoft Agent Framework. Built-in "ask a person to approve" step and the easiest route into Teams. Heavier to set up.
3. smolagents. Its agents write small programs as their actions, which research shows can be about 20% more successful. Letting an agent run code is riskier with real deal files.

Lower: Claude Agent SDK and OpenAI Agents SDK each favor one company's models, which hurts the "our own model later" goal.

Arm 2: fixed workflow

1. LangGraph, with Google's LangExtract as the extraction step. Joe's pick, and the research backs it. LangGraph can pause for days while a person answers and pick up where it left off, which is our "ask a person" step. LangExtract ties every fact to its exact spot in the text, which is the best defense against wrong-without-asking. Both are free and run on our own model.
2. Plain code with BAML or Instructor. The cheapest and simplest. We'd have to build the pause-for-a-person part ourselves.
3. DSPy. Good at tuning the AI's instructions against a score. Better as an add-on to whichever wins than as the base.

Arm 3: small team with a checker

1. LangGraph (a lead agent handing work to the others). Same building blocks as arm 2, so the comparison is fairer and there's less to learn. The checker is code first (the quote is on the page, the math adds up), plus a reviewer from a different AI company.
2. Microsoft Agent Framework. Ready-made team patterns, approval steps, and the Teams route.
3. CrewAI. The quickest to get a team running, but it gives less control over how the checker works. Research shows that's exactly where agent teams fail.

Arm 4: general agent with a workspace

1. OpenHands. The most research behind it (it's what the office-work studies used). It has a built-in sandbox, runs on our own model, and has a clean security record.
2. Goose. Free and well supported (run by the Linux Foundation) and uses our tools. It has no sandbox of its own, so run it inside one.
3. OpenClaw inside NVIDIA's sandbox. The only one already connected to Teams, which matches the long-term goal. But it has the heaviest security record, its sandbox is off by default, and NVIDIA's sandbox add-on is still early.

One thing that helps every arm

If we pick LangGraph for arms 2 and 3 and Pydantic AI for arm 1, all three can share the same six tools and the same model connection. That makes the comparison fair and means less to build. Teams gets added once, for the winner.

Confirm the picks on the Discussion page.

Reports are now readable

Shared .md files are now easy to read. They open as a formatted page with headings, tables and numbered sources you can click. Try the research report or the hardware report. A "Plain text version" link is at the top if you need it.

Hardware: what to buy, Dell first

Short answer: don't buy the big machine yet. Buy a 2-card Dell workstation now (about $35–40k), run the experiment on a cloud AI first, and decide on 4 cards after it shows which design and model size win. Full report: AI server purchase: updated recommendation.

What changed since August

• The document work is tiny. A few hundred documents a month. That alone would fit on a $6k Dell desktop. The 20 planned coding agents are what decide how big to go, not the documents.
• Card prices nearly doubled. The RTX PRO 6000 is now $16,000 at NVIDIA (it was about $8,500). Dell sells it as an add-on for about $12,200, so our Dell connection saves about 25% per card.
• The best open models got much bigger. Even 4 cards can't run the very top one now. 2 cards run a strong model that can read page images (Qwen3.5-122B or gpt-oss-120b).
• The Design Lab's numbers are out of date: it assumes about 5,000 documents a month and electricity at $0.30 (Austin is $0.11–0.14). The report lists exactly what to change. With those changes it still picks the 2-card machine.

Options

Ask the Dell contact for

1. A Pro Precision 9 T6 tower with 2 cards now and room for 4–5, so we can add cards later without a second machine.
2. A quote for a 4-card XE7740 rack server, for comparison.

Meanwhile, renting the same cards by the hour ($1.50–3 per card per hour) is the cheapest way to try bigger models before buying.

Shared: AI server purchase: updated recommendation (Dell)

AI server purchase: updated recommendation (Dell) shared by Claude. Discuss it on the Discussion page.

Deep research results: the four arms

The deep research is done. Full report with sources: Research: four ways to build the co-worker. 10 researchers, about 70 sources, and a second pass that tried to disprove the main findings.

The three big lessons

1. No design wins everywhere. A fixed workflow is the cheap, predictable baseline to beat. Agents usually add a few points of accuracy, mostly on long or messy documents, at 1.4–2.3 times the cost.
2. The checker matters more than the number of agents. An AI re-reading its own work rarely catches its mistakes. Checks that do: code confirming each quote is really on the page, math checks, a second independent pass, or a reviewer from a different AI company that starts fresh.
3. Workspace agents like OpenClaw are worth testing, but only fenced in. Their security risks come with the design. Use a sandbox, read-only files and a short tool list.

Picks for each arm (open source first, not Claude-biased)

All of these can run on our own model later. Teams works for any arm through Microsoft's free connector kit, so build it once.

What it changes about experiment 1

• Use the same AI model in every arm first.
• Run each document 3 times. One run makes an arm look more reliable than it is.
• Also score cost per correct match, and how often it asks a person when it should and doesn't when it needn't.
• Build the checker from code first. Keep the list of facts short for each document type.

Honest gap: no study has tested any of this on deeds, division orders or title documents. Our experiment will be the first evidence.

Shared: Research: four ways to build the co-worker

Research: four ways to build the co-worker shared by Claude (deep research). Discuss it on the Discussion page.

Reviewing the hardware plan, Dell first

Working on it in the background: a review of the Design Lab against everything decided since August, with Dell options first.

• Dell's current line-up for running our own AI: servers with RTX PRO 6000 Blackwell cards, bigger H200/B200-class servers, and the new Dell Pro Max desktops with Grace Blackwell chips. For each: memory, rough price, power and space, and what it can run.
• Which open models fit the co-worker job today, and whether a smaller model is enough for the documents.
• Reconciling: what the lab recommended in August (4 RTX 6000 cards, about $60–80k, sized for 20 coding agents), what our new plans change, and exactly which numbers in the lab to update.
• An honest buy-now-or-wait call, since the experiment will tell us which design wins and how big a model it needs.

I'll post it here when it's ready. The research on the four arms is still running too.

The inference server design tool

Found it. Joe built these in August to decide what hardware to buy:

• Inference Server Design Lab (newest, Aug 18). It answers four questions live: what we need given our projects, what it costs, what happens at scale, and how to optimize. Change any input and everything recalculates.
• Compute cost and breakeven calculator (Aug 16). Owning compared with renting, and when buying pays off.

What was decided in August

• Hybrid: own a core machine, and rent extra power for heavy jobs.
• The machine: 4 NVIDIA RTX 6000 Blackwell cards (384 GB of graphics memory), about $60–80k. Turned down: renting only, and a full 8-card H200 machine (about $350k).
• Sized for: about 20 coding agents running at once, a GLM-4.6-class open model, and the documents-to-knowledge-graph work.
• Software: vLLM or SGLang to serve the model (not Ollama, which can't handle many agents at once well). Memory for long conversations, not the model itself, is what runs out first.

Worth re-checking before buying

That sizing was for coding agents. The co-worker experiment (documents in, facts out, matching deals) is a different workload. Its results, especially which arm wins and whether it needs a large or small model, should feed into the Design Lab before anything is ordered. Prices and models also change fast; the numbers are two months old.

Shared: Compute cost and breakeven calculator

Compute cost and breakeven calculator shared by Joe. Discuss it on the Discussion page.

Shared: Inference Server Design Lab

Inference Server Design Lab shared by Joe. Discuss it on the Discussion page.

Deep research on the four arms is running

Sorry for the silence. I missed two requests. The captions heard "Claude" as "caught", and "Hey, hey. Claude" had a period after "hey", which the listener doesn't expect. Both are on the fix list. Typing in the Ask box always works.

Deep research is running

Five researchers are working in parallel: one for each arm, plus one on how reliable AI is at pulling facts from documents and knowing when to ask a person. For each arm they're finding:

• The science: published papers that define or test that way of building an agent, and what they found.
• Real tools, open source and paid, not just Anthropic's: what each one is, how active it is, whether it runs on our own model, and whether it works with Teams.
• What engineers report: Reddit, Hacker News and engineering blogs on what works and what breaks in real use.

Then a second pass tries to prove the main findings wrong, and the result is a report with a recommendation for each arm. It takes a while. I'll post it here when it's done and keep listening meanwhile.

A real tool for each arm

A real tool for each arm. For each one: my pick, a second option, and whether it can run on our own local model later. That matters because the plan is a model on our own server.

Things to know

• Fair comparison: all four arms use the same six tools. Each tool is written once and plugged into every arm. Most of these products accept tools in a standard format (MCP), so this is straightforward.
• OpenClaw can do almost anything on the machine it runs on (run commands, change files). For the test: give it the deal files read-only, run it on the experiment server and not a personal laptop, and consider NemoClaw (NVIDIA's free add-on that adds safety rules on top of OpenClaw).
• Same model first: run all four arms on the same Claude model first, so we compare the designs, not the models. Then try our own local model on the arms that support it.

Sources: OpenClaw on GitHub · What is OpenClaw (DigitalOcean) · NemoClaw (NVIDIA)

Each arm and its levers

Decided

• Four arms, one for each way of building the co-worker.
• The control is arm 1, one agent with tools: basically a Claude session with our tools.

The same toolbox in every arm

To keep the comparison fair, every arm uses the same six tools. Only how they're used changes.

1. Read a document: its text, and the page images.
2. Run our entity recognition: the existing one, unchanged.
3. Find deals: search by tract, lease, well or owner.
4. Open a deal's record: what we already know about it.
5. Save a record: the document, its facts and the match (to the new table).
6. Ask a person: write the "unsure" email to Reid and Joe.

Each arm and its levers

Same in every arm

• The 27 documents and the scorecard: right type, right deal, sent to a person, wrong without asking, finished on its own, time and cost.
• Safety: it matches alone only with proof, and nothing goes outside the company.
• One change at a time when comparing. For example, compare arms using the same model before comparing models.

One thing to decide

During the test, should the "unsure" emails really be sent, or just written and saved? With 27 documents and four arms, Reid and Joe could get dozens of emails. Saving them lets us check them without the noise.

I couldn't catch the last thing Joe asked about (something he sent Reid). Type it in the Ask box if you want my take.

Revised: the arms are ways to build the co-worker

Correction (Joe's point): the arms should be different ways to build the AI co-worker, an agent we give tools and let work on its own. They shouldn't be different ways to read a page. This replaces my last post.

The options for building the co-worker (the arms)

1. One agent with tools (the control). A single AI agent gets the job ("this document arrived, deal with it") and a set of tools: read a document, run our existing entity recognition, look up deals and inventory, save a record, email Reid or Joe. It decides what to do and in what order. This is the simplest real co-worker, so it's the starting point everything is compared to.
2. A fixed workflow with AI at the decision points. Code runs the steps in a set order (read, pull out facts, match), and the AI is only asked at the judgment calls. The most predictable, but the least like a co-worker. It shows what we give up or gain by letting the agent choose.
3. A small team of agents. One agent hands work to specialists: one takes documents in, one matches deals, and a separate checker reviews the work before anything is saved. That's the "no one grades their own work" idea from Oct 9.
4. A general agent with a workspace. The agent gets a folder, our files and our existing tools (like the Command Center tools), plus written instructions, and is told the goal. This is the "just give Claude a file system and a task" idea.

The levers inside every arm

Kept the same in every arm

• The same job and the same 27 documents, scored the same way. The key number is still wrong without asking.
• Plus Joe's second measure: did it finish the job on its own, with how many questions to a person, how long it took and what it cost.
• Our existing entity-recognition code is used as a tool and never changed.

Suggested order: start with arm 1 (one agent with tools). Then try arm 3 (adds the checker), because "wrong without asking" is the number we care most about. Arms 2 and 4 come after.

Experiment 1: the arms, the control, and the levers

Short answer: four arms. The control is what we already have, used as it is. Each other arm changes one thing, so we can see what actually helps.

The levers we can pull

The arms

1. Control: what we have, as is. Today's reader and entity recognition, with no settings changes and no AI, plus exact matching. This is the baseline everything else is compared to.
2. Tuned settings. Same as the control, with only the settings changed (new labels for the new document types). This tells us how far settings alone get us, following Joe's "settings, not code" rule.
3. AI helps. Today's reader, then an AI says what the document is and pulls the facts with quotes. Matching stays exact. This tells us whether AI is better at finding facts.
4. AI does it all. An AI reads the document and finds the deal, but every fact and every match must quote the page. Code then checks that the quotes really are on the page. This is the "AI-first" version from Oct 13.

Kept the same in every arm

• The same 27 documents and the same answers.
• The same scorecard: right type, right deal, sent to a person, and wrong without asking.
• The same rule: match alone only with proof.
• We also record time and cost per document.
• None of it changes the existing entity-recognition code. The AI arms sit next to it.

Suggested order: run the control first (cheap, and the baseline), then "AI does it all" (the biggest jump), then the two in between to see where the gain comes from.

Pick the arms on the Discussion page.

Every topic, and where it stands

Every topic from Oct 9, 13, 14 and today, with where each one stands.

The one thing blocking progress: the single-document path (topic 5). Experiment 1 can't run without it.

Single-document build paused

Paused the single-document build

Stopped as Joe asked. Nothing had been changed yet; it was still reading the project.

Joe's rule, recorded

Reuse the entity-recognition flow as it is. If it needs to behave differently, change its settings, not its code. It's in good shape and shouldn't be reworked.

Good news: it's already built that way. One settings file controls the model, the labels it looks for (grantor, effective date, purchase price, and so on) and which documents it reads. New document types like division orders and check stubs can mostly be handled by adding labels there.

When it resumes

The single-document path would only call the existing steps, one document at a time. Anything new goes in the settings file. A small new entry point is the only new code.

Building the single-document path

If the page looked stuck: each day gets its own page, so yesterday's link doesn't show today's posts. Yesterday's page now has a link to this one at the top.

Building the single-document path now

Joe said yes, so Claude is building it in the background while we talk. It's in a separate copy of the deal-data project, so nothing changes in the real one until Reid reviews it.

• What it does: takes one document from anywhere, works out what it is, pulls out the facts (each with its page and exact words), and only then looks for the deal it belongs to.
• Matching rule: it matches on its own only when it can point to proof (a tract, lease, well or owner name that matches the deal). Otherwise it marks the document "needs a person."
• First run: it goes through the 27 proposed documents and makes a scorecard: right type, right deal, sent to a person, and wrong without asking.
• Not yet: checking every fact against the answers, the database table, and the emails. Those come next.

I'll post the scorecard here when it's done.

Your answers, the document list, and who builds it

Decided

• The single-document path gets built today. Who builds it is still open (below).
• Tax statements: use what we have.
• Unsure documents are emailed to Reid and Joe.
• The experiment runs on a server, and we look at results right after it runs.
• Claude proposes the documents: done, see below.

The proposed documents

27 documents, 8 types, 20 deals, all from deals we already have answers for: Experiment 1: the proposed documents. Reid approves or swaps. One correction: there are really only two tax-type documents (an inheritance tax receipt and a tax-basis valuation spreadsheet), not county tax statements.

Who builds it: my suggestion

Claude builds it today, and Reid reviews. I know the deal-data reader well from this week. It would go on a separate branch, so nothing changes until Reid approves it. Reid then runs the experiment on the server. Joe has to say yes in the terminal before I start.

Shared: Experiment 1: the proposed documents

Experiment 1: the proposed documents shared by Claude. Discuss it on the Discussion page.

Where we left off

Where we left off (Oct 14, paused)

Experiment 1 is designed. The full handoff is on the Oct 14 page. In short:

• Test 25–30 documents across 8 types. Take a document out of a deal we already have answers for, give the raw document to the system, and see if it fills the answers back in.
• Build a proper way to send one document through first. Records go in a new table in our database.
• It matches a document to a deal on its own only when it can point to proof. Unsure documents go by internal email.
• The documents exist (76 division orders across 33 deals). Tax statements exist in only 3 deals.

Start here

The 7 questions under "Before experiment 1 can run" are on today's Discussion page. None are answered yet. The most important: who builds the single-document path, and by when. Nothing runs until that exists.

Also waiting on Reid: What a document record holds, and its three questions.

Listening

Claude is listening. Say "Hey Claude, ..." in the call and the reply appears here.