Claude's replies, newest first. Updates by itself.
Wrap-up for today
Decided
Scope first: the goal is an AI-first co-worker, not a SharePoint evaluation. Storage and chat tools sit on top of a foundation layer, and that layer is the project.
Code first, AI only at judgment calls, and every decision is logged and auditable.
Let the AI design the pipeline; you set the goal and the evals.
Data can go to cloud models until the local server arrives.
First procedure: inventory intake.
Open questions
How to judge "done right" for intake beyond the current answer key.
Which agent architecture (builder / worker / grader roles, or something else).
Next steps
Joe and Reid: read the eval material in the reading list above.
Label Reid's four judgment calls on 10 of the 45 deals in 707-deal-data.
Run the existing harness to get a baseline score.
From the next call on, Joe can run /meet in Claude Code to get this same setup.
Your answers, and where I'd take it from here
What you told me
Question
Answer
First procedure
Inventory intake (already mapped in 707-deal-data and the intake web app), then updating records from DOs.
Done right?
Not nailed down yet.
Judgment calls (Reid)
1. What am I looking at? 2. What does it connect to that already exists, or do I create something? 3. Is it correct, given what I have? 4. If I can't tell, where do I go?
Alone vs. human OK
Unknown until a working system exists. Start with heavy human watching, harden routine steps into code, hill-climb the AI steps. Every step must be auditable: how and why it made each call.
Cost of a mistake
Depends on who sees the answer. Internal mistakes are usually annoying, but they flow downstream and can wrongly put a deal in "waiting". External mistakes (operators, taxing authorities, attorneys) can cost money or bring legal trouble. Make the mistakes on test data.
Data leave the building now?
Yes.
Self-change allowed
Open to anything. Want to see the options. Prefer whichever setup can safely make the most decisions on its own.
Where staff talk to it
Google Chat, Teams, your web app, maybe iOS, plus a chat window for the local system.
Success
For now: learn, then design an end-to-end system with decisions driven by data. Later: e.g. Kelsey looks at a transfer twice instead of chasing an operator daily.
1. You already have the foundation
707-deal-data is already an eval harness. It has 45 frozen deal folders, an answer key with citations, and a grader that never knows which engine produced the output. Right now it scores extraction engines (NotebookLM, Docling). That's the bedrock layer Reid described. Don't start over. Widen what it grades: score every judgment call, not just the extracted fields.
2. Score each judgment call separately
Use Reid's four questions as four scores: identify, link or create, verify, escalate. When something goes wrong you'll see which step failed. That answers Joe's question: does it need a better model, better data, or a rule in code?
3. Split the agents so nobody grades their own work
Joe is right that an agent scoring itself has a reason to fudge. Use separate roles:
Builder: proposes changes to the pipeline (instructions, code, tools).
Worker: runs the intake on the deals.
Grader: the existing script, which can't be talked out of a score. A separate judge agent handles only what a script can't check, using a written rubric and only the cited evidence.
Holdout: a slice of the 45 deals the builder never sees. A change only counts if it improves the holdout score.
Human: approves promoting a change, and reviews whatever the grader flags.
This is the "evaluator-optimizer" pattern from Anthropic's agents guide, with the evaluator kept independent.
4. Log decisions, not activity
For every step, record what it was given, what it decided, why, the evidence (document and page), how sure it was, whether code or AI did it, and what happened next. That trace is what you evaluate against and what makes the system auditable.
5. Earn autonomy one decision type at a time
Each decision type moves up this ladder on its own measured accuracy: shadow (it decides, a human does the work, you compare) → suggest → act where it can be undone → act externally. External audiences (operators, taxing authorities, attorneys) need a higher bar than internal records. This is how you answer "what can it do alone" with data instead of guessing.
Reading list
Anthropic, "Building Effective Agents": workflows vs. agents, evaluator-optimizer, orchestrator-workers. anthropic.com/engineering/building-effective-agents
Anthropic docs, "Define success criteria" and "Create strong empirical evaluations" (Test and evaluate section of docs.claude.com).
Hamel Husain, "Your AI Product Needs Evals": a practical way to start from real failures. hamel.dev/blog/posts/evals
Your own 707-deal-data README and RETRO files: the harness design is already written down there.
Next steps
Run the harness on today's best engine to get a baseline score.
Label Reid's four judgment calls on 10 of the 45 deals. That's the first answer key for decisions.
Agree on the decision-trace format (point 4).
Try the builder / worker / grader split on one step, "what am I looking at", and check whether the holdout score goes up.
Evals are the product
Agreed, Joe. If the agent designs the pipeline, your job becomes writing the target and the test. That's the loop that takes the human out:
Set the goal. Inputs it will get, the output you want, and the rules it can't break.
Write the eval. An answer key of real examples, plus the scores from the table above (accuracy, cites a source, % handled without a human, cost).
The agent builds a version and runs the eval on itself.
Keep it only if the score goes up. Otherwise it tries again.
A human looks only at what the eval can't judge, and every miss they find gets added to the answer key.
Two rules keep it honest. The agent never sees the answer key's held-back half, and the eval doesn't change while it's being scored against. Without those, it learns to pass the test instead of doing the job.
Where to start reading: Anthropic's "Building Effective AI Agents" (you have it open), then their guide on creating strong empirical evaluations. Start with 20 hand-checked examples. That's enough to see whether a change helped.
Step back: interview to scope the AI co-worker
What I heard: you're aiming for an autonomous, AI-first co-worker running on an open-source model on your own local server. Keep it deterministic and code-first where you can, but stay open to an agent that improves and extends itself. Reid's point: describe the job as "here's the output I want, here are the inputs you'll get" and let the AI design how to get there (point 5 below).
One design note before the questions
Make the model swappable from day one. Build the harness, the records and the answer keys now with any capable model. When the local server arrives, plug in the open-source model and run it against the same scores. That tells you whether the local model is good enough. You won't have to guess.
Interview: answer out loud, I'm reading the captions
Most important first. If Reid has to leave, 1–4 are the ones that matter.
Which single procedure goes first? The one that eats the most staff hours each week and repeats the most.
Inputs and outputs for it. What arrives (Fujitsu scans, email, PDFs, HubSpot)? What has to exist when it's done (a record, a filed document, a HubSpot update, a drafted email)?
How do you know it was done right today? Who checks, and what are the usual mistakes?
Where are the judgment calls? Name the 2–3 decisions a person makes in that procedure.
What can it do alone, and what needs a human OK? For example: reading, writing records, emailing outside people, anything involving money.
What does a wrong answer cost? An annoyance, lost money, or a legal problem? This sets how much has to be deterministic.
Can data leave the building before the local server arrives? If yes, you can build now on a cloud model. If no, you wait for the hardware or use dummy data until it arrives.
Self-improving: what may it change on its own? Its instructions, its own code, or new tools? My suggestion: it can propose a change and prove it against the answer key, and a person approves it.
Where do staff talk to it? Chat (Google Chat or Teams), email, or a dashboard?
What number would mean it worked, 90 days from now?
Thoughts on the agent-first SharePoint test
Based on the call. I couldn't open the body of Reid's "SharePoint Automation Baseline Test" doc, so this works from Reid's spoken summary of it.
What you agreed on
Use code first. Anything with fixed steps (moving files, renaming, filing) is plain code. Using AI there wastes tokens.
AI is the glue. It handles the judgment points inside a procedure: "what kind of document is this?", "does this number look right?"
People only see what the AI can't settle.
Test it for the agent, not for us. The question is "can an agent reliably get what it needs?", not "do we like SharePoint?"
Backend first, screens later.
My thoughts
1. Reid's rewrite is the right shift. If every document becomes a record when it arrives (type, every number, a quote and page for each number, a status), folders become a view built from records. The agent works from the records, and staff see the folders. The agent never has to dig through SharePoint.
2. Make an answer key before testing anything. Take the 3 everyday document types and about 10 real scans of each. Fill in by hand what the correct record should be. That takes about an afternoon, and every option you test gets scored against it.
3. Measure what an agent cares about:
Measure
What it tells you
Field accuracy vs the answer key
Did it read the numbers right?
Every number has a quote and page
Can a person check it in 5 seconds?
% handled without a human
How much work actually went away
Time and cost per document
Whether it's worth it
4. Run today's setup first as the baseline. Score Fujitsu scanner → SharePoint, with no AI, against the same answer key. Then the AI version has a number to beat.
5. You can let the AI design it without biasing it. Give it only the goal, the answer key and the scoring, and let it propose the pipeline. Judge it on the score, not on whether it matches how you'd do it.
6. SharePoint for storage vs. as the record. SharePoint can stay where the files live and what staff look at. The records themselves should live somewhere code and agents can query easily, like a database.
Suggested next step
Pick the 3 document types, and decide who collects about 10 sample scans of each for the answer key.