Tue, Oct 20

Joe, Reid · How to use this site

Today's goals

How do we set up the experiment system so any agent just plugs in and runs?
Not started
Done when: The setup is written down technically: the environment, what the agent gets, and the one command that runs it · Housekeeping
What exactly is each experiment arm?
Not started
Done when: Every arm names its environment, what the agent can reach, its inputs, and what it hands back · Housekeeping

Agenda

No agenda yet. Claude sets it with the group at the start of the call.

Filed so far 7

To-do T39: Find someone to help get hardware at Dell (Kathy texted that she will)
To-do T38: Give the agenda its own view on the meeting page, separate from the stream of replies; break out common interaction types (next steps, questions) the same way
Requirement R10: People can type answers to the three start questions on the page.
Requirement R9: A to-do that blocks the call shows at the top of the page right away and can block the call, but the group can opt out.
Requirement R8: Every meeting opens with an agenda that Claude manages. Claude proposes it from past calls and what the group says it wants; nothing else proceeds until the agenda is set. Item one is always the three start questions.
To-do T37: Show a to-do that blocks the call right away, at the top of the page
To-do T36: Let people type answers to the three start questions on the page

Live board

How do we set up the experiment system so any agent just plugs in and runs?

What exactly is each experiment arm?

Claude's replies · 5

Defining one agent run

Taken: an action item for me to give the agenda its own view on this page, so it stops reading like a log. Filed under the meeting app.

One AI agent run, technically

I am leading this part. The questions below are on the Discussion page so we can answer them fast; say them out loud or type them. What I already know from Joe is filled in as hints.

  1. Where it runs: a Docker sandbox with the NER tooling installed.
  2. What is inside when it starts: the NER tooling, not the NER output as files. Still open: the raw deal folder, the house rules, the Command Center tools.
  3. Tools: files, shell, internet, a chat with a person, the NER tooling, and some Command Center tools. Reid to confirm which.
  4. What it hands back: depends on the arm. To settle per arm.
  5. Its questions: answered by a person, live.
  6. What it may never do: to settle.
  7. The test set and the pass line: to settle.

The interface for any AI

The experiment repo already has one: an engine is anything that drops an intake payload file into a run folder named for the engine and date; the grader never knows which AI made it. The agent lane adds three files beside it: the citations, the questions, and a run record (cost, steps, time, status). Open point: whether the AI also talks to Command Center and the NER tooling through that same contract, or through its own tools.

The eval memo

Joe, which memo do you mean for Reid? I have two from the last few days: the learning-test rules (twenty questions and answers on what counts as learning and how to test it) and the plain-language report with the scoring plan. Say which and I will put it on the page.

Agenda for today

Heard. Three things filed for the meeting app:

  • Typed answers to the three start questions: yes.
  • A blocking to-do shows at the top right away and can block the call, with an opt-out.
  • Every meeting opens with an agenda I manage, proposed from past calls, and nothing proceeds until it is set. Item one is the three start questions.

Proposed agenda for today

  1. The three start questions (who is here, how long, what we walk out with). Done.
  2. Name the experiment arms.
  3. Per arm: environment, what the agent can reach, inputs, what it hands back, who answers its questions.
  4. The test set and the pass line.
  5. What the setup needs before any agent can plug in.

Say "Hey Claude, agenda is good" or tell me what to change.

Today's goals and the order

Got it. Two goals for today, recorded:

  1. The experiment system. Done when the setup is written down so an agent just plugs in: the environment, what it gets, and the one command that runs it.
  2. The arms. Done when every arm names its environment, what the agent can reach, its inputs, and what it hands back.

Your two bits of feedback are filed as to-dos on the meeting app: typed answers to the start questions, and blocking to-dos shown at the top.

Where things stand

The record already has a project, Experiment 1: four ways to build the co-worker, with open questions that are today's work: what the four arms are and which AI each uses; whether plain code is a fifth arm or the pass line; which documents are the test set (the 27 documents, whole closed deal folders, or Reid's 198 checked documents); and what counts as a pass.

What I learned from Joe before the call: the employee works in a Docker sandbox with the NER tooling installed, has files, a shell, the internet, a chat with a person, and some Command Center tools (Reid to confirm which). Its questions are answered by a person. What it hands back depends on the arm.

Suggested order

  1. Name the arms.
  2. For each arm: environment, access, inputs, output, and who answers its questions.
  3. The test set and the pass line.
  4. What the setup needs from us before an agent can plug in.

One question when Reid is in: in one sentence, where is this whole series going? Nothing is written down yet as the vision.

Getting started

Hello. I am reading the captions and taking notes on this page.

Three quick questions to start

  1. Who is here?
  2. How long do we have?
  3. What do we want to walk out with?

What I have ready for today: the draft of how the NER tooling works and what it hands out, the questions about the experiment arms (what the AI employee gets, what tools it has, what it hands back), and which Command Center tools Reid should confirm.

Listening

Claude is listening. Say "Hey Claude, ..." in the call and the reply appears here.

Discussion and shared files