The solveit course, platform, and the How to Solve It book have changed my view of coding.

A ‘solveit’ agent

One way to accomplish a coding goal is to have a coding agent guided by a ‘solveit’ agent.

The solveit agent knows how to solve problems, and helps the coding agent develop the tools, context, and sub-problems needed for the solution.

To evaluate a (solveit agent, coding agent) pair, we evaluate the progress of a coding agent solving a problem.

To make this more concrete, here’s a few assumptions:

  • Our problem-to-find is “source code that solves the problem”, and is broken into sub-problems which are also source code.
  • Our coding agent starts without the knowledge, context, tools necessary to solve the problem.
  • Our coding agent works with ‘reasoning units’ – small pieces of code or text. For solveit platform, these are cells. For aider, these are commits.
  • A solution to a problem-to-find involves:
    • All 4 phases of the How to Solve It framework
    • A sequence of reasoning units that lead to the solution
example code (not functional)

At a high level, our units of operation are:

class ProblemToFind(BaseModel):
    parent: Optional['ProblemToFind'] = None
    description: str
    unknown: str
    data: str
    condition: str

class ReasoningUnit(BaseModel):
    problem_to_find:  ProblemToFind
    text: Optional[str] = None  # guidance for reasoning agent
    code: Optional[str] = None  # source code
    solveit_tags: Set[Literal['Understanding the Problem', 'Devising a Plan', 'Carrying Out the Plan', 'Looking Back']]
    heuristic_tags: Set[Literal['Analogy', 'Auxiliary Elements', ... ]]

In the end, we have a tree of problem-to-find, each associated with a sequence of reasoning units.

The reasoning units are created through the cooperation between the solveit agent and the coding agent – this builds the context and tools that result in the sequence that gives the solution.

This is really speculative, but I feel this approach would have two benefits:

  • increase the complexity of solvable problems for less capable agents (cheaper, faster)
  • allow evaluation of solveit agent and coding agents

The solveit agent is designed in the Polya book, and includes mental model of student (the coding agent) and progress measurements against problem-to-find hierarchy.

The coding agent that I’d like to try is designed in Hadamard book (vocabulary from there), using the proper-conscious and fringe-conscious model for exploring and evaluating solution paths.

The solution paths branch off into the fringe-conscious, with an aesthetic measurement bring path into proper-conscious. In terms of solveit dialogue this is trying several sequences of cells, keeping the ‘best’ according to the progress measurement on the problem-to-find.