Experiment 1: four ways to build the co-worker

Choices the group still has to make before Round 1 can run. Updated 2026-10-26. The 10-26 call settled things around the project (rented cloud first, the old intake retired, the Ector text-reading run, one app) but none of the Round 1 blockers below. Excel as the answer sheet (items 5) is still Joe's line alone. Blocking Round 1: 1. Name the arms and the AI behind each (Joe and Reid, to-do T3). Joe's shortlist has five agents and may grow. Also: what is the one thing that changes between arms (see The four ways). 2. The agent setup from the two research write-ups (Joe, to-do T22, was due the weekend of 10-18). 3. The eight run details on the Discussion page (Joe and Reid; see The shape of one run). Detail 1 is narrower now: runs go on rented cloud machines (D16). Left: one machine shared by all arms or one per arm. 4. One shared interface or each agent's own tools (Joe and Reid; see The interface any AI plugs into). Narrower now: whatever is shared builds on the deal data project's intake, not the retired Command Center one (D17, unsure). 5. What the answer sheet is built from: Excel as it stands, recorded documents, or Excel with a disagreement list. Joe has said Excel three times, never with a yes from Reid or Keegan (D12 to D14). Needs the group's yes (see The answer sheet). 6. What fields a complete inventory record holds for a deal (question Q10). 7. Which memo is the eval criteria memo for Reid (Joe). Then what counts as a pass for each measure (to-do T5, no owner). 8. Joe's review of Reid's checker branch: merge or what to change (to-do T31). Four measures are still unscored there. 9. Joe's answers to Reid's three questions on the test documents and the plain-code scope (to-do T32), and Reid's answers to the three document-record questions (to-do T2). 10. Whether plain code is an extra arm or the pass line, and who runs it on the other seven document types. 11. What happens to the 27 test documents: keep, swap for whole deal folders, or use Reid's 198 checked documents. 12. The words of the written task every arm gets (see The written task), and the columns of the Round 1 report (see What a Round 1 report shows). Both are drafted and need the group to fix and freeze them. Not blocking Round 1 but still open: whether test unsure emails are sent or only saved (to-do T4), how a deal gets marked complete (to-do T28), the county data plan, how to search the thumb drive records (the Ector run may help), and the one-sentence goal for the whole series. The server purchase now waits on numbers from the rented cloud tests (D16), so the Dell contact (to-do T39) can wait. Also still unconfirmed: owners for to-dos T28, T29, T31, T32 and T34, which were never read back. Three of the 10-26 decisions (D15, D17, D19) were saved as unsure and need a spoken yes.

New topic from 2026-10-20. The goal from that call: set up the experiment so any agent just plugs in and runs. The done-when was: the environment, what the agent gets, and the one command that runs it, all written down. It was not reached. What was agreed in spirit: the setup is modular, so adding an agent is easy. Every agent gets the same written task and the same deal folder, and hands back a record in the same shape. Changed on 2026-10-26: the old intake inside the Command Center is retired, and the intake we build on is in the deal data project (D17, saved as unsure). The same call agreed one app does everything, with each part in its own code store behind one front door (D19, saved as unsure). For this part that means the shared tools, if the group picks them, should sit on the deal data project's intake and on the one app, not on the Command Center. Open, not discussed yet: does every AI reach our data and the name-finding tooling through one shared door, or does each bring its own tools? Option A: one shared interface. We build one set of tools once (read the folder, look up the inventory, ask a person, write the record) and every agent calls the same ones. Fair comparison, and the one-change rule is easy to keep. Costs the work of building the shared layer first. Option B: each agent uses its own tools. Faster to start, and shows what each agent does out of the box. But then two things change between arms (the agent and its tools), which breaks the one-change rule. Option C: shared interface for the things that touch our data and people (inventory, asking questions, writing the record), own tools for everything else (reading PDFs, searching). Keeps the comparison fair where it matters and lets each agent shine where it can. Suggestion for the group: whichever is picked, write down the one command that starts a run, so Reid can start any arm on a rented cloud machine without Joe's laptop.

New from the 2026-10-20 call (Joe and Reid, call dropped twice, so this is the part that was said, not a finished setup). Updated with the 2026-10-26 call. Three rules Joe gave, saved as suggested requirements R1 to R3: 1. Between any two arms exactly one thing changes. So if arm 2 beats arm 1, we know why. 2. A run is the whole job for a deal, not one step: intake, then the transfer, then the remaining transfer steps, including the parts that wait on other people (emails out, replies back). 3. Every run starts from one written task: here is the deal folder, here is what to do. The same written task goes to every arm. What the run looks like, as far as it was said: the agent works inside its own sandbox (a Docker box) with the name-finding tooling (NER) inside it. A person answers its questions. The setup is modular, so adding one more agent is cheap. Joe said the list of agents may get longer, never shorter. Eight details still to fill in. They sit on the Discussion page and were not answered on 10-20. For each, the options the group could pick from: 1. Where it runs. Narrowed on 10-26: tests run on rented cloud machines first, and the real numbers decide what hardware to buy later (D16). Still open: one rented machine with one sandbox per arm, or one rented machine per arm. Shared is cheaper; separate is cleaner and gives each arm its own clean cost and speed numbers, which is what D16 wants to buy from. 2. What is inside the sandbox: only the deal folder, or the deal folder plus the current inventory so the agent can match the deal to what we already own. 3. Tools: the NER tooling and the Command Center. Changed on 10-26: the intake inside the Command Center is the old first version and is retired; the intake we build on lives in the deal data project (D17, saved as unsure). So any intake tool an arm gets should come from the deal data project, not the Command Center. Still open: one shared door or each agent's own tools (see The interface any AI plugs into). 4. What it hands back: an inventory record in the agreed shape (see Everything is a record) with proof for every fact, plus a list of questions it could not answer. 5. Who answers its questions during a run: a named person for the whole round, or whoever is on the call. Suggestion: one named person per round, so arms get the same quality of answers. 6. What it may never do: send anything outside the building, change the inventory directly, or guess when unsure. These follow from the safety rules. 7. The test set: see The test set. 8. Plain code as an arm or the pass line: see The bar to beat. Suggestion for the group: answer these eight in one sitting next call. Once they are answered, Claude writes the setup down (the topic's next step).

Decided: focus on one region at a time, not many counties at once (decision D1). Keegan brought a thumb drive with the Ector and Martin County records on 2026-10-19. The Midland County records were expected Monday or Tuesday. How to search the records on the thumb drive is open with no owner (to-do T29). The old handwritten county indexes are the hard part, and Martin County is the first place to try turning them into a usable index (to-do T24). New on 2026-10-26: Reid goes ahead now with the text-reading (OCR) run on the Ector County records. Joe looked over the plan and Keegan is fine paying for it to see what it can do (D18). The Martin and Ector records are a test of how far we can get; they do not change the hardware choice for now (D15, saved as unsure). What this means for Experiment 1: once the Ector run is done, its output is the first real chance to answer to-do T29 and to see whether county records can join the test set. Until then they stay out of Round 1. Still open and not needed for Round 1: the county data plan. Reid's way is to buy access to structured data and process only what we need. Keegan's way is to buy whole county archives and structure everything. Reid was also trying a sample from a county data service (to-do T10). Hardware, changed on 2026-10-26: test on rented cloud machines first, and use the real numbers from that test to decide what to buy (D16, agreed by Joe and Reid). Any purchase has a long wait, so real numbers make a better baseline. The Dell contact Kathy was finding (to-do T39) is no longer on the critical path. Documents come in through the Fujitsu scanners into one SharePoint folder. Open: whether the free scanner software can split a stack into separate documents (to-do T13). For Round 1 this matters less, because a closed deal folder is already a folder, not a stack of pages. Reid is setting up the cloud server and deploying the one app (to-do T33). On 2026-10-26 the group agreed one app does everything: Reid's intake app joins the meeting notes and the inventory view behind one front door, each part kept in its own code store (D19, saved as unsure). Once it is up, anyone can run calls and the data views without Joe's laptop.

Round 1 needs a known right answer for every closed deal it tests. What the meetings have said about where those answers come from: The current answers were built from HubSpot and the Excel inventory for 40 to 50 deals, and the intake process has run on 48 deals so far (Reid, 10-19). Reid asked on 10-19 whether those answers are actually right. Keegan warned against trusting the Excel and HubSpot, because people typed them in by hand, and said to trust the recorded title documents (the PDFs of record) instead. Reid agreed. Reid also said the existing answers are right in direction, and that is enough to build a tool that knows what to look for. On 2026-10-21, testing the call alone, Joe said to use the Excel inventory as the source (decision D12, saved as unsure). Reid's earlier note already ruled Excel over HubSpot where both carry a field. On 2026-10-22 Joe said it twice more, on two practice calls with Reid, this time as the source for the title index (decisions D13 and D14). Both were sample lines to test the page, and the read-back got no spoken yes, so they stay unsure. Two things to settle from that wording: whether the title index and the inventory are the same thing here, and whether the group agrees in a real call. What this means for the experiment, as options: Option A: Excel answers now. Score Round 1 against the Excel inventory as it stands. Fast, and matches D12 to D14. Risk: an arm that reads the recorded document correctly gets marked wrong when the Excel has a typo. Option B: recorded documents as truth. Someone hand-checks the answer sheet against the PDFs of record before Round 1. Slower, but then a wrong mark means the arm was wrong. Fits what Keegan and Reid said. Option C: Excel now, with a disagreement list. Score against Excel, but every field where an arm disagrees with Excel and shows proof from a recorded document goes on a list for a person to rule on. Each ruling fixes the answer sheet. This also seeds the inventory with checked facts, which fits decision D8. Still open from before: what fields a complete inventory record for a deal holds (question Q10). Keegan added one on 10-19: the basis allocation, how the price paid splits across each property. Reid said that is already part of intake. Suggestion: the required-field list is the answer sheet's column list, so settle both at once. The same list also sets what the written task asks for (see The written task) and the columns of the report (see What a Round 1 report shows).

Reid's write-up of the two pre-experiment tests is the model for how every arm reports (to-do T21, done): what was done, why, how, and the results. This part proposes what the Round 1 report looks like when all arms have run, so the group can see whether it answers the question the experiment is asking. One table, one row per arm (plus a plain-code row if the group picks Option A in The bar to beat). Columns: 1. Fields right: matched the answer sheet. 2. Fields asked about: flagged for a person instead of answered. 3. Fields wrong without asking: the key number from How we measure. Lower is better and this column decides the winner. 4. Deals complete: every required field right or asked about, nothing wrong. 5. Hand-offs: how many times a person had to step in during the run (Reid's point, 10-19: fewer hand-offs to Kelsey, Tana or Kathryn is the aim). 6. Per document type: right rate for each of the 8 types, next to plain code's rate for that type, with pass or fail per type (decision D6). Under the table, for each arm, the four parts Reid's write-up uses: what was done, why, how, results. Plus one line saying the one thing that made this arm different from the arm before it (rule R1), so a reader can tie a score change to a cause. What the report cannot show yet: the checker scores only part of the measures today (four are not scored, and its keys cover deal and type only, from Reid's notes read back on 10-21). Until Joe reviews the checker branch (to-do T31) and the four measures are scored, columns 3 to 5 would be filled by hand or left blank. Open, as options: Where the report lives. Option A: a page in the one app, next to the Design, so anyone on a call can read it. Option B: a document Reid writes, like the pre-experiment write-up. Option A fits Joe's aim that anyone can see the data without Joe's laptop (see Region and data), but needs the app to show a table. Who signs off that a run counted. Option A: Reid, since the checker is his. Option B: the named person who answered questions during the round, since they saw the run. Option C: both, with Joe as tie-break. Suggestion for the group: agree the columns before Round 1 runs, because the answer sheet, the checker and the written task all have to produce these columns and it is cheaper to line them up now.

Rule R3 (Joe, 10-20): every run starts from one written task, and the same written task goes to every arm. Nobody has written that task yet. This is a first draft for the group to fix, built only from what the calls said. What the task should say, in plain words: 1. Here is a closed deal folder. Read everything in it. 2. Write one inventory record for this deal in the agreed shape (see Everything is a record). Fill every required field (see The answer sheet, question Q10). 3. For every fact you write, show the proof: which document, which page, the words you relied on, and why that is the answer. 4. If you are not sure about a field, do not guess. Mark it as a question and say what you would need to know. 5. Match this deal to what we already own where you can (see The shape of one run, detail 2: this only works if the inventory is inside the sandbox). 6. Hand back the record, the list of questions, and a short note of what you did. 7. Never send anything outside the building and never change the inventory directly (follows from Safety rules). Open bits, written as options: How much help the task gives. Option A: the task names the 8 document types and what each usually holds, so every arm starts with the same domain knowledge. Fairer, and closer to how a new hire is briefed. Option B: the task says only what the record must hold and lets each arm work out the documents. Shows which arm learns the work on its own, which is closer to what Joe wants from a co-worker. Who the arm asks. Option A: the task names the one person who answers for the round (see The shape of one run, detail 5). Option B: the arm writes its questions into the record and nobody answers during the run. Simpler, but then rule R2 (a run includes the parts that wait on people) is not tested in Round 1. Later rounds. Since a run is the whole job for a deal (R2), the later steps (operator and county emails, division orders, check and tax comparisons) get added to this same task as paragraphs 8, 9 and so on, rather than new tasks. Suggestion: write paragraph 8 (emails out) now as a placeholder so the arms know more is coming. Suggestion for the group: read this draft aloud next call, fix the words, and freeze it before any arm runs. Once frozen, changing it means a new round, not a new arm, or the one-change rule (R1) breaks.

The experiment is building Cletus, the AI co-worker. Cletus routes documents, runs title and handles tasks. We keep building until we are satisfied, instead of making separate one-off tools (decision D3, 2026-10-16). Deal intake comes first (decision D5, 2026-10-19). A closed deal goes in and a complete inventory record comes out. Keegan's reason: the inventory is the brain of the operation, and every later step depends on it. Reid's reason: the gate into our system is after close. Reid also noted paperwork arrives before close too (title packets), so scanning is not only after transfer. The transfer work runs in this order: deal intake, then operator and county emails, division orders, the monthly check comparison, and the tax statement comparison. Anything Cletus cannot do goes on a needs-a-person list. We are not comparing this against how the work is done by hand today. Round 1 tests deal intake end to end (decision D9). Since 2026-10-20 a run is defined as the whole job for a deal, so later rounds extend the same run rather than start a new one (see The shape of one run). Joe's picture of the end state (10-19): the AI co-workers get their own computers, we talk to them, and they work on a schedule like any employee. The aim is to lower what needs a person, not hand the team a new pile of work. Joe also said a possible goal for the whole series once: confirm with confidence that we get paid correctly and pay taxes correctly on every asset we own. The group has not confirmed that sentence yet, and the 10-20 close-out asked for one vision sentence next call.

Two measures: how good each step is, and whether the whole thing runs end to end with little supervision. The key number is how often Cletus was wrong and did not ask. Done: the grading code that scores each task is built (to-do T23). Reid finished the two pre-experiment tests and the write-up of what was done, why, how, and the results (to-do T21). That write-up is the model for how every arm reports. The early conflict checks are parked until the inventory and the experiment are done (to-do T20). The plain-code bar: for each document type, an AI arm counts only if it beats plain code's match rate for that type (decision D6; see The bar to beat: plain code). Where the checker stands (from Reid's notes, read back to Joe on 10-21): the checker branch is not merged, four measures are not scored yet, and its keys cover deal and type only. Joe's review of the branch is still owed (to-do T31). Until the four measures are scored, Round 1 can only report on part of what it should. New on 10-20: Joe has an eval criteria memo for Reid, but did not say which memo. Joe names it, then Claude puts it on the page. Suggestion: once it is on the page, this part should be rewritten from it rather than from what the calls said. Still to write: what counts as a pass for each measure (to-do T5, no owner). Suggested starting point: a field is right if it matches the answer sheet; asked about if Cletus flagged it for a person; wrong otherwise. A deal is complete if every required field is right or asked about and nothing is wrong. An arm passes a document type if its right rate beats plain code's rate for that type. A whole run also gets a count of how many times it needed a person, since the aim is fewer handoffs to Kelsey, Tana or Kathryn, not a new pile of work for them (Reid, 10-19). Reid follows up on the pre-experiment tests next: anything still open or blocking Experiment 1, and if nothing, what we can test next (to-do T34).

The experiment tries several ways (arms) of building the co-worker and runs each one against the same closed deal folders, the same written task and the same checks. Changed on 2026-10-20: Joe said there may be more than four arms, and the list may widen, never narrow, because a modular setup makes adding an agent cheap. The name four ways is now a working name, not a count. Joe's shortlist of AI co-worker versions, said on 2026-10-19: OpenHands, Claude Code, NemoClaw (NVIDIA's locked-down OpenClaw), Letta (long-term memory, built on the MemGPT research) and Hermes Agent (works with many models, memory and learning built in). What Joe wants from every version: it stops and asks when it is confused instead of guessing. Which of these become arms, and which AI model sits behind each, is still Joe and Reid's call (to-do T3). On 10-21 Joe asked for research on the most capable open source model, so an open-model arm is likely on the list. The rule that shapes the arms: between any two arms exactly one thing changes (R1). The group has not said which thing. Options: Option A: the agent framework changes, the model stays the same. Example: Claude Code, OpenHands and Hermes Agent all running the same model. Shows which way of working is best. Option B: the model changes, the framework stays the same. Example: one framework running a closed model, an open model and a small local model. Shows which brain is best. Option C: a ladder. Arm 1 is the simplest setup; each next arm adds one thing (memory, a tool, a better model). Fits the one-change rule and tells a clear story, but needs an agreed order. Plain code may sit beside the arms as an extra row, or be the pass line each arm must beat (see The bar to beat: plain code). Suggestion for the group: name the arms next call, each in one line that says what is different from the arm before it. The arms were never named on 10-20 because the call dropped.