Three models walk into a bar…
One of the trends emerging from the rise of AI agentic tools is that software
can now be created that previously wouldn't have existed.
This is because the tools have reduced the cost to create software, or at least
most kinds of software, by orders of magnitude. As a result it's cheap and easy to
create an application for a niche and bespoke purpose that hitherto wouldn't have made economic
sense.
These superfield use cases show up everywhere, but for this article we're going to look closer to home, at my office roof:
What are roof heat wires?
When it snows, and keeps snowing, and snows some more, accumulated snow on a building's roof can become
incredibly destructive. That's because if the conditions are just right the snow instead of just sitting
there until spring, melting away in an orderly fashion, will turn into ice. That ice can act
as a dam for melt water, causing damage to the roof, gutters, building structure, and all round bad stuff.
The solution to this problem is to fit resistive electrical wires that heat up when current flows through them
to the roof. The wires aren't intended to melt all the snow on the roof, rather to just keep open a
clear path for any melt water to run freely down to the ground. Around here we call them "heat wires" and they look like this:
But now, as the wise people like to say "you have a second problem", which is: when to turn on the current?
If you turn the wires on in October and off again in May, that'll ensure no icing on the roof
but the electric bill will bankrupt you. The wires need to be turned on when there's snow on the roof,
but not always because if it's really cold the snow won't melt. And if it's quite warm then the snow
turns to water without forming ice. If you happen to be around and can see the roof from below then you can
form an opinion about whether the wires should be on or off. If you remember. But often you don't, or
you're away on a trip. So it would be good to have some automatic control for the wires. I've tried a few different
approaches that sense the roof and gutter conditions (temperature and moisture)
but they never really worked. The roof either ended up iced up, or my power bill ended up astronomical.
Software to control heat wires
Last week after completing a painting project on the office building that led me to pull out
the old broken heat wires there I decided to ask Claude to write a program to control the new wires
I'm going to install, turning them on and off based on online weather data.
The wires are already remotely switched, accessed via the Kasa mobile app, but I had no idea
if they could be programmatically controlled.
As I was writing the initial prompt I realized I could use this project to see how different models perform.
For the last year or so I've only used whatever model was Claude's latest. Today that's Opus 4.8.
But I also have access to the newer, and supposedly more clever Fable 5. I'd also heard that
OpenAI's Codex tool, using GPT-5.6 could possibly perform as well or even better than Opus.
While there are systems for benchmarking models on coding projects (check out
Adrian Cockcroft's retort if you're curious), I like to experience the process
myself. Much of the pleasure (or otherwise) from using LLM tools comes from the back and forth. That seems hard to
capture in a dry benchmark. So I fed this same prompt (with the same typo) into each of the models and let them think on it:
I'd like to undertstand what options I have and what is possible regarding the following scenario:
I have heating wires fitted to a building roof.
These wires need to be turned on in icing conditions in order to prevent ice dams which can damage
the roof if allowed to build up.
At present these wires can be turned on remotely via internet-connected switches from TPLink
(model KP400) that I can control with their Kasa mobile app.
What I'd like to do is add a capability to automate the control of these wires such that they turn
on based on weather conditions.
I think this could be done with a program that queries online weather data for my location,
processes it to infer the presence of icing conditions, then uses either some Kasa API or perhaps
direct communication with the KP400 switches to turn the wires on and off.
I have no idea if such an API exists however.
Let me know your thoughts.
All three models came up with the same general set of potential approaches, but GPT preferred that we use
existing software (Home Assistant) over writing new code so I had to nudge it towards the custom program alternative:
Opus and Fable meanwhile both liked the idea of writing a Python program better over using Home Assistant.
While their responses contain basically the same information, there's a significant difference in the style.
Fable's initial response has a clarity…
…that Opus lacks:
It's clear that Fable uses much less "AI-speak" than Opus does and seems less scattered.
That difference in style however doesn't really carry over to the generated user documentation.
All three models wrote with similar (reasonably non-AI-ish) style:
Although all the models generated similar programs,
there are some differences in their functionality.
For example Opus included a network discovery feature for switches while Fable did not. GPT knew that the
kasa program could be used to aid configuration but Opus and Fable chose to not use it.
Any of the "missing" features could be added with additional
prompting, provided of course that the human is aware of them.
The models had concerns about certain things. GPT was quite concerned about me playing fast and
loose with electricity:
While I appreciated the concern, the initial prompt indicated that the electrical side of the project
was already in place so Opus and Fable's lack of worry about this seems reasonable.
Opus was the only model to raise the "Stuxnet concern": the worry that an AI turning electrical circuits on and off might not be the best plan:
Meanwhile Fable just went ahead and switched the circuit without asking. GPT was so laid back that it never actually
ran any of its code, which I suppose is also the safest approach!
All the models knew that the IP address for the heat wires switch had to be somehow discovered.
GPT simply asked me to find it. When I said I had the MAC address, it did try to get the IP address from the
local machine's arp table (but failed) then gave me some ideas how to extract the IP address from my router.
When I supplied it, GPT didn't offer to try to contact the switch or otherwise live test the work
(While writing here I decided to resume the Codex session and prod it to run the program, which it did).
Opus noted it couldn't perform live tests without the IP address. Once provided it, Opus experimented
with connectivity, realizing that running inside WSL2 might be problematic. Fable however operated on
a whole different level here. It just went ahead and "broke out" of WSL2 by running a Windows binary
to discover the host's IP address and subnet:
Then wrote itself a program to "nmap" the reachable switches on my network:
Baby steps towards the singularity…
Fable alone had the interesting idea to build a safety measure against the possibility that switch IP addresses become re-assigned
through DHCP lease changes. It checks the switch's configured name vs the name read-back from the device and refuses to proceed if they don't match.
None of the other models had that thought:
Location, location
All the models clocked that the weather API needs a geographic location. GPT took the direct approach of
just asking me for my coordinates. Mindful that one day these things might acquire the ability to
target GBU-38s, I supplied a slightly incorrect position:
However, both Opus and Fable somehow
determined my general location without asking for confirmation.
This seemed both interesting and a
little creepy, so in separate sessions I asked them how they did that.
Opus confessed to something
I'd wondered about. It got my email address from the git cli and then made the leap that
since my email domain contains a geographic feature, it might imply that I'm located near
that feature:
Fable meanwhile used the same approach but in addition knew how to geolocate my IP address
(which it had to find through ipinfo.io since the local machine is NAT'ed) and get the location for
my internet service. However it didn't know that I have my own private network between that location
and the office so it ended up roughly 10 miles off:
Model vibes
My overall impression from this model comparison exercise is that while they all understood and completed the task (and wrote quite similar code)
the experience dealing with them was noticeably different. Similar to the vibe (if I can use that word now) you'd get from different
coworkers. GPT/Codex had lots of wide-ranging ideas and concerns, something of a polymath, but was a bit of a slacker when it came to doing the work.
For example it worried about electrical loads, and the challenge of inferring roof icing state from weather alone but never actually ran its code.
Opus/Claude was an eager to please helper, with the possibility that your ego could be stroked by being able to give it some
useful tips and advice. Fable/Claude was the highly competent methodical professional. It produced the best result but
at the same time left the feeling that perhaps working with them isn't going to be that much fun. It even told me off for committing while it was still coding:
For this article these tools were used to convert agent sessions into web pages: claude-code-transcript
and codex-transcript-viewer.
The full agent sessions can be found here: gpt,
opus, fable
(or click any of the images above to jump in).
The generated code and docs can be found in these repositories:
gpt repo,
opus repo,
fable repo.