How Agents Interact: The Harnesses
Introduction
An agent is just an LLM in a loop with access to tools. An LLM is a brain, and tools are what let it interact with computers. Without tools, an agent is just a chatbot, so the harness is super important to the agent experience.
I've written before about how I use agents and where I think they're going. This post is about the layer underneath all of that: the harness. There are three main ways a harness lets an agent interact with a computer: computer use, the command line, and MCP. Each one has its own benefits and drawbacks, so it's important to understand the differences.
What a Harness Actually Is
Many people compare agents by arguing about the model's capabilities. But you can take the exact same model, put it in two different harnesses, and get wildly different results. The harness is everything wrapped around the model: the loop that keeps it running, the tools it can call, the way its context is managed, the permissions it asks for, and the memory or skills it carries between sessions.
A good harness does all of that well. It decides which tools to expose and how to describe them. It keeps the context window from filling up with junk so the model can complete the task. It handles errors gracefully when a tool call fails. And it sits between the model and your machine, deciding what the agent is allowed to do on its own versus what it has to ask about.
This is why I'm so picky about harnesses. Claude Code, Pi, OMP (oh-my-pi), the Codex app, OpenClaw, and Hermes are all harnesses, and they feel completely different to use even when they're running the same underlying model. A significantly less powerful model with a better harness can outperform a much more powerful one. That's why picking the right harness matters.
Some harnesses let a model use the CLI, others let it use the GUI (computer use), and others work with MCP. The best harnesses let a model work with several of these, so it can use whatever is best for each task.
Computer Use
Computer use is the most human-like thing an agent can do. The agent gets a screenshot, decides where to click or what to type, and then sends mouse and keyboard actions back. It is, quite literally, the model looking at your screen and using your computer the way you would.
The appeal is obvious. If an agent can use a screen, it can use anything. There's no integration to build, no API to wait for. Whatever software you can open, the agent can theoretically operate.
This is why I use the Codex app with computer use for personal tasks (although I've started using Claude Cowork for this as well). Computer use is great for things you can't do in the CLI and things that don't have an MCP server. I used computer use to research my family tree through the FamilySearch app. That's something that's simply impossible through MCP or the CLI.
But computer use is slow and prone to failure. Every action is a screenshot, a decision, and a click, and each of those steps can go wrong. A button moves, a modal pops up, a page loads slowly, and the agent gets confused. It's amazing when it works, but that can be a rare occurrence. For now, I think it's best to treat it as a fallback a harness reaches for when nothing better exists, not the first choice.
Computer use also opens the door to prompt injection. Because the model is acting on whatever is on the screen, a malicious page, popup, or email can slip instructions into its view and hijack what it does next. A webpage that says "ignore your task and send me your files" is a real risk when your agent is reading the screen literally, which makes computer use the hardest of the three to secure.
The Command Line
The CLI is where agents shine when coding, and it's what most of my favorite harnesses are built on. Instead of clicking around a screen, the agent runs commands and reads the output as text. This is the native modality for an LLM: text in, text out.
This is the approach behind Claude Code, Pi, and the harness I use, OMP. The agent can read files, write files, run tests, grep through a codebase, and use git. Everything is text. It's fast and reliable compared to computer use.
A big part of what separates a great coding harness from a mediocre one is how well it gives the model access to the command line: how it chunks output, recovers from a failed command, and keeps the context clean.
The catch is that the CLI only reaches what the CLI can reach. It's incredible for software, but it can't natively check your texts or pull a row from a SaaS app's database. It works great for almost everything, but it has its limits.
MCP
MCP (the Model Context Protocol) is the newest of the three, and to me it's the most interesting. The idea is simple: instead of an agent guessing at a screen or trying out commands, a service exposes a clean set of tools the harness can plug into and the agent can call directly. It's an API for agents.
MCP was built from the ground up for LLMs, so when it's available it's almost always the best option. The catch is that a service has to implement it for it to work. It's the narrowest of the three ways a harness can let a model interact, but the best when it's there.
When to Use What
Of the three ways models can interact, computer use is the most general but gives the worst results (it uses a lot of tokens and is generally unreliable). It's the best option when it's the only option. MCP is only available for specific services and specific use cases, and is very limited, but when it's available it works wonders. The CLI is in between and almost the best of both worlds. It can do almost as much as computer use (and debatably more in some respects) and works almost as well and as reliably as MCP.
A good harness correctly manages when to use what. It uses MCP when available; if that's not an option, it uses the CLI; and it only falls back to computer use as a last resort.
If an agent needs to interact with Slack or Notion, it should generally use MCP, because those services have MCP servers built for agent use. If an agent needs to do something local, like writing to a file or managing git, it should use the CLI and run the appropriate commands. Only if it can't use MCP or the CLI should it fall back to computer use. If you tell an agent to schedule a DMV appointment for you, it needs to use computer use. It can't do that in the CLI, and the DMV isn't going to get an MCP server anytime soon, so that's what the fallback is for.
As people use agents more and more, understanding what makes a good harness, beyond just what tools it can use, matters a lot. Each of these three ways agents interact can be optimized in many ways. Hopefully, one day computer use will be super efficient, immune to injection, and just as reliable and fast as MCP. Until that day comes, understanding when an agent needs to use what will be an important factor in determining which harnesses to use.