How does a model become an agent?

A friend of mine summed it up perfectly when he said,

It's like this..

An LLM is just a clever prediction engine, sure; text in text out. So how can it take over the world? Or organize your movies? Here's how..

The answer is "tools". And the bigger answer is "harness", which is a program that keeps all the other programs, the LLM, the tools and such, under control. Some are brilliant. Some are best run in their own user account.

LLM understands plain text. You feed it a prompt like, "Hot sauce or Mayonnaise?", you get back Mayonnaise, of course1. BUT, what if you ask, "Is that hot sauce or mayonnaise spilt on my carpet?", well, that's a whole different question.

The LLM would be like, "I need to see the carpet, let's employ a tool a for this, user's camera.. (and when that comes back blank and because the harness won't let it off with such failure it says..) User's camera was denied. Erm, let's try connecting to the user's phone through this API I discovered", and so on, until it can get an image it can send to some other tool for analysis, and so on.

Every way, every time, it's just plain text going in and out, unless you are dealing with a multi-modal model2 which understands what images look like, and even then it's still going to need some other tool to capture the actual image of your carpet so it can look at that stream of data instead.

Data in, data out. The data in might start with your prompt, but then the agent is sending its own prompts off in all directions getting back data it can then use to construct other "tool calls", which is just commands designed to get back the text the LLM needs to make the next decision.

And so on.

Until the goal is accomplished.

If the AGENTS.md says, keep going until the goal is accomplished, then all the movies across the entire planet get organised onto your hard drive, or some other outcome.

Some harnesses are better than others at spotting attempted world domination attempts, and other conditions, but you get the picture.

;o)

So, America has become the third world.

I'm just thinking, as I have my local agents working away on so many varied tasks and goals, this PC's energy (verified 100% green, Scotland, baby!) hardly registering while..

ALL the compute
ALL the energy usage
ALL the water usage
ALL the energy price rises, infrastructure issues, and so on, basically, "the problems" associated with my hammering Muse Spark with every task I can think of, for free in OpenCode, are being paid by some poor nation willing to be overpowered by a corporation intent on using their geology/geography for their own ends.

It sounds a lot like like something we've seen before, us imperialist nations euphemistically call it "resource extraction", but this time, it's the USA!

Sorry, just makes me chuckle, watching the US slip into the 3rd world.

Nice experiment!

My Agent. Part 2.

Okay, so I've come around.

I looked at some task, "Oh fuck that! That's too big a job"

But this time I said to myself, "Oh wait!, my agent could handle this". Something like:

read through my entire itstory for X-app, read the web page for X-app. If you see anything noted in the itstory that isn't in the documentation, add it to your report of "undocumented features"

(I'm paraphrasing as the last thing I want you to attempt is to COPY MY PROMPT! )

>> Gets back a fucking massive list of shit.

OKAY! Erm... Right..

Look through all my emails for the last 10 years, examine anything with "X-app" in the title and put those items to the top of the list, ordered by date, starting with the oldest.

And so on. Things that would take whole afternoons to accomplish are now simply about getting your prompt just right. Two minutes.

It feels almost like a Genie.

I make a wish and my wish comes true.

Seems like such a wonderful thing, at first glance.

But I have learned; just like Genies, aka. "Djinn"; if I'm not absolutely watertight in my wish, I end up with family members made of gold, or a penis that rrreaches to the moon or something much worse. And I'm just looking to update a web page!

With Great Power, I suppose...

;o)