How I plan to use LLMs

What I have watched happen in my profession in the past two years, I am still struggling to describe. The first time I knew something was wrong, roughly a year and a quarter ago, I noticed a colleague replying to me using AI. His response was obviously generated by Claude.

That’s the opening of Appearing Productive in The Workplace. The entire essay is good, you should read it.

I’ve been noticing something similar, both at work and, worryingly, in personal spaces.

Tale of fixed disk size

A coworker recently told our entire team that some of our hosts had a fixed, tiny disk size. We could get more CPU and RAM provisioned, but disk size was consistent. The problem was, these hosts were consistently running out of disk space and our only way out of it was to do a lengthy migration.

That sounded wrong to me, but my coworker had confidently told the entire team, so who was I to question them? Coincidently, I had to touch that part of the infrastructure a few days later. So, I decided to take another stab.

I told the issue to our LLM and after 10–15 minutes of “deep research”, it concluded that yes, the hosts could not get extra disk space. Unconvinced, I asked it what other teams were doing and to refer to wikis and internal community questions portal. This time, it informed me that the disk size was actually changeable along with CPU and RAM.

I tried the solution and it worked. The new hosts have been chugging along fine without any issues.

I’m surrounded by smart people

My coworker has solved a lot of hard problems. This isn’t a jab at them. I don’t begrudge their LLM use, they’re going along with the incentives set forth by Extremely Upper Management ¹.

With this post, my goal is to set up a healthy LLM framework for myself instead of pointing fingers.

My LLM use framework

Software engineering: Learning first

This is inspired by Addy Osmani’s post Don’t Outsource the Learning. My main learnings from his post were:

1. If you start by asking the LLM, it frames the entire problem, producing worse decisions.
2. LLMs are tuned to produce output, not teaching.
3. What you prompt matters. Starting with “this doesn’t work, so let’s do that” makes the LLM take that as a fact. The disk space example showed all three of these learnings.

When I prompted the LLM for research, I started with the assumption that disk space was not possible, and it agreed. From there, it was focused on finding solutions to this issue, either via a migration or other workarounds.

Only after my pushback after 15 minutes, it found that our initial hypothesis was wrong. I should have either looked at the docs cursorily first, or asked it an open question instead of letting it drive all the research.

This quote from Addy’s post sums it up nicely:

Ship and learn are two separate metrics. Your manager and your customers will only ever ask about the first one. The second is on you.

Balancing learning is easier said than done. Here’s my plan:

1. Pay attention to framing: For LLMs, there’s a very thin line between hypothesis and fact. LLM might just accept a theory as a fact. I’ll start by only asserting what I know and frame everything else as a question.
2. Use examples: I learn best with examples, and I’ve found that LLMs are similar. I’ll aim at finding similar patterns and providing working examples with “this is how it’s being done in codebase” to steer the LLM in the right direction, instead of just giving a loose prompt.
3. Ask for explanation, with sources: Sometimes, examples are just not there. Documentation might not be there, or I could be looking at an edge case. In that case, asking for explanation helps. Even better, ask for sources. Current Claude models are good enough to admit when they can’t find an explanation and will tell you that they were mistaken. ²

Writing: No LLMs

I have found LLMs helpful for initial research and generating drafts of code. But writing is a hard line for me, and I will not use LLMs for creating anything from scratch.

Johanna Larsson did a wonderful job in The social contract of writing, and you can read the entire thing. My stance is the same as hers. A short version:

Social contract of writing has always been that the writer understands the ideas in their writing and put some effort into it. Using LLMs violates that social contract. Like Johanna, I commit to not using LLMs to write.

Images: No LLMs

I have written an entire post on AI Images in Blog Posts. I found them disappointing at that time. The models have gotten a lot better since then, but I’ve still not found a generated image that I liked. I tolerate them if writing around them is excellent, but such images usually signal image generation for the sake of an image instead of any meaning.

I don’t use generated images and I will not put them on this blog or anything else that I write. I’ll make something basic in Sketch or draw it, however bad it is.

Gaming: Hell no!

If the day ever comes when an LLM is playing my games for me, I authorize you to take away my console, my TV, and whatever I’m using to stream it away from me.

-—