[{"content":"From One Raspberry Pi to 100 Docker Containers The other day I was scrolling through container logs in Dozzle when I glanced at the sidebar.\n104 containers.\nI actually stopped for a second.\nSomehow, without ever setting out to build a homelab of this size, I\u0026rsquo;d ended up running well over a hundred Docker containers across my infrastructure. It made me think back to where all of this started.\nLike a lot of people, it began with a Raspberry Pi.\nBack then it was just a handful of Python scripts automating small jobs around the house. Before every second YouTube channel was talking about home automation and AI, I was happily SSH\u0026rsquo;ing into a little Linux box to see what else I could make it do. That single Pi eventually became a Hyper-V cluster built from four Intel NUCs. One of them even started life as an old Datto storage appliance. Rather than throwing perfectly usable hardware away, I kept finding another purpose for it. Later came three Dell Micro Form Factor PCs, now running as a Proxmox cluster, all mounted neatly inside a 3D printed rack (3D printed by Stephan, huge thanks btw!). It\u0026rsquo;s probably one of my favourite parts of the lab. It proves you don\u0026rsquo;t always need enterprise hardware or expensive cabinets to build something tidy and reliable.\nThat\u0026rsquo;s one thing I\u0026rsquo;ve grown to appreciate over the years.\nHardware generally lasts far longer than people give it credit for.\nIn enterprise IT we\u0026rsquo;re conditioned to think in three, five or seven-year refresh cycles. In a homelab, those same machines can often provide another five years of useful service with very little effort. Reducing e-waste isn\u0026rsquo;t just about buying efficient hardware. Sometimes it\u0026rsquo;s about simply making better use of what already exists.\nThe software has evolved just as much.\nHome Assistant has become so embedded into the house that if it goes offline, people notice. Lights stop behaving as expected, automations disappear and the family starts asking questions. At some point I realised I\u0026rsquo;d accidentally created a production environment at home.\nThat also meant introducing something I\u0026rsquo;d never expected to need outside of work.\nChange control.\nThese days I think twice before updating anything on a Friday evening. It\u0026rsquo;s the same instinct that eventually made me pin every image tag and update deliberately rather than let things drift under me.\nAlongside Home Assistant I\u0026rsquo;m running Portainer to manage the Docker environment, Dozzle for container logs, Dashy as my dashboard, Hugo for this website, Pi-hole for network-wide ad blocking, Open WebUI to give the kids a supervised AI interface, n8n for workflow automation, Caddy as a reverse proxy, Cloudflared tunnels, and dozens of small services I\u0026rsquo;ve written myself to solve specific problems.\nIndividually they\u0026rsquo;re tiny.\nCollectively they make the house run.\nSomething else has changed during all of this.\nI\u0026rsquo;ve realised just how many online services follow the same pattern. Start with a useful free tier, gradually introduce subscriptions, then if you stop paying you\u0026rsquo;re greeted with adverts or reduced functionality. Individually they\u0026rsquo;re inexpensive. Collectively they become another monthly bill that quietly grows over time.\nMy homelab has become the opposite of that.\nThe more I build, the more control I regain. My data stays on my equipment. My automations aren\u0026rsquo;t tied to someone else\u0026rsquo;s roadmap. If I want to experiment, I can. If I break something, it\u0026rsquo;s my fault and I get to fix it.\nMaybe this is my own version of declouding.\nNot abandoning cloud services entirely, because plenty of them still make sense, but being more deliberate about what belongs in someone else\u0026rsquo;s datacentre and what belongs in mine. It\u0026rsquo;s the same argument I keep coming back to when I write about owning your tools.\nIt reminds me of what the internet felt like years ago.\nPeople built things simply because they could. Websites were personal. Projects existed because someone wanted to learn something new. There wasn\u0026rsquo;t always a subscription attached to every feature.\nThat\u0026rsquo;s still the feeling I get every time I spin up another container.\nAlthough… perhaps 104 is enough for now.\n","date":"2026-07-17T00:00:00Z","permalink":"/infrastructure/from-one-raspberry-pi-to-100-docker-containers/","title":"From One Raspberry Pi to 100 Docker Containers"},{"content":"G98 vs G99 Explained Without the Industry Jargon When I first had solar and a battery installed, I thought the regulatory side of it was somebody else\u0026rsquo;s problem. There had been a form. The form had been dealt with. My system exported to the grid, the app showed nice green arrows, and I got on with the more interesting business of teaching a battery optimiser to be clever about when to charge.\nIt was only later, when I started seriously thinking about expanding the system — a bigger battery, more panels, folding the EV charger into the same energy strategy — that I realised the regulatory side had never been somebody else\u0026rsquo;s problem. It had been quietly deciding what I was allowed to do next, and I had signed up to those limits without understanding them. A decision made on the day of installation, in a box I never saw, was now the thing standing between me and the system I actually wanted.\nSo this is the article I wish someone had handed me before I started. It assumes you have never heard the terms G98 or G99, and it explains them in plain English, because they are not complicated once you strip away the jargon — and understanding them before you install is worth real money.\nThe one-sentence version Before anything else, here is the whole thing in a sentence, and if you remember only this you will already be ahead of most homeowners:\nG98 is \u0026ldquo;install it, then tell them.\u0026rdquo; G99 is \u0026ldquo;ask first, then install.\u0026rdquo; The line between them is about how big your system is and how much it can push back onto the grid.\nEverything else in this article is detail hanging off that sentence. Let us build it up properly.\nWhat these things actually are G98 and G99 are not laws exactly, and they are not product standards. They are Engineering Recommendations — documents published by the Energy Networks Association, the body that represents the companies that own and run the electricity networks in Great Britain and Ireland. When you see them written formally they carry an \u0026ldquo;EREC\u0026rdquo; prefix: EREC G98 and EREC G99. In conversation everyone just says \u0026ldquo;G98\u0026rdquo; and \u0026ldquo;G99.\u0026rdquo;\nTheir job is to define the rules for connecting generation to the distribution network — the wires that bring electricity to your house. Anything in your home that can push energy out onto those wires counts as generation for this purpose. Solar panels obviously do. A battery does too, because it can discharge to the grid. A hybrid inverter, which manages both, is the device that actually does the exporting, and it is the thing these rules are really written about.\nThe reason they exist is not bureaucratic obstruction, even though it can feel that way when a dialog box interrupts your installer. The grid was designed as a one-way street: power stations at one end, houses at the other, energy flowing downhill from big to small. Domestic generation reverses that. Suddenly thousands of houses are pushing energy back up the street, and the network has to cope with that without the voltage drifting out of safe limits, without equipment being damaged, and without a fault on the grid causing your inverter to keep energising a line that a network engineer believes is dead. That last one — anti-islanding — is a genuine safety issue. An inverter that kept feeding a \u0026ldquo;dead\u0026rdquo; cable could kill someone working on it. The grid codes are, at bottom, the rules that make sure your nice green arrows do not become somebody else\u0026rsquo;s electrocution.\nTake a look at just one weeks monitoring of my own platform, the voltage is within safe limits but clearly a balancing act in itself! So G98 and G99 both answer the same question — \u0026ldquo;under what conditions may this generation connect to the network?\u0026rdquo; — and they differ only in how big the generation is, which determines how much scrutiny the network operator wants before it says yes.\nThe DNO: the character you did not know was in the story To understand the difference between the two, you need to meet the party on the other end of the form. This is the Distribution Network Operator, universally abbreviated to DNO.\nThe DNO owns and runs the physical distribution network in your region — the substations, the transformers, the cables in the street. They are not your energy supplier. This confuses almost everyone, so it is worth being clear: your supplier (Octopus, EDF, British Gas, whoever) is who you buy electricity from and pay your bill to. Your DNO is who owns the wires. You do not choose your DNO — it is determined by where you live — and until you install generation you may never have knowingly interacted with them in your life.\nWhen you install solar and a battery, the DNO is the party that cares. They are responsible for keeping the network stable and safe, and your new ability to export is a change to their network that they have a legitimate interest in knowing about, or approving, depending on its size. The whole G98-versus-G99 distinction is really a distinction about your relationship with the DNO: whether you are obliged merely to inform them, or obliged to ask their permission first.\nYou buy power from You export power to ┌─────────────┐ ┌──────────────┐ ┌──────────────┐ │ Your supplier│◀──────│ Your home │───────▶│ The DNO │ │ (billing) │ │ solar+battery │ │ (the wires) │ └─────────────┘ └──────────────┘ └──────────────┘ the bill generation G98 / G99 lives here Keep that picture in mind. G98 and G99 are the terms of your relationship with the box on the right.\nNotification versus permission: the actual difference Now we can state the difference properly, because it is genuinely the heart of the whole thing.\nG98 is a notification process. If your installation is small enough to fall under G98, you are allowed to install it and connect it, and then tell the DNO — typically your installer submits the notification on your behalf, usually within a defined window after commissioning (28 days is the figure that gets quoted). You do not wait for permission. The DNO\u0026rsquo;s answer is assumed to be yes, because the equipment is fully type-tested to standards that guarantee it behaves safely on the network, and it is small enough that the network can absorb it without individual assessment. This is sometimes described as \u0026ldquo;connect and notify\u0026rdquo; or \u0026ldquo;fit and inform.\u0026rdquo; The dialog box in my first article in this series — the one my EV charger installer dismissed — was the system confirming its G98 compliance.\nG99 is an application process. If your installation is larger than the G98 threshold, you must apply to the DNO before you connect, and wait for their approval. They assess your specific case: what you want to install, how much you want to export, and what the network in your area can actually accommodate. They may approve it as requested. They may approve it with conditions — most commonly a cap on how much you are allowed to export. They may, in constrained areas, require network reinforcement before they will say yes, which can carry a cost and a delay. G99 is \u0026ldquo;ask first,\u0026rdquo; and the asking is a real assessment, not a rubber stamp.\nThe practical difference in your life is enormous. A G98 installation can be booked, fitted, and running in the time it takes to do the physical work. A G99 installation has a DNO application in the critical path, and that application takes time — potentially weeks, occasionally longer if the network is constrained — during which nothing can be commissioned. If nobody has told you which category you are in, you can get an unpleasant surprise about timelines, or worse, a system that gets installed and then cannot legally be switched on at full capacity until an application that should have gone in weeks earlier is finally approved.\nWhere the line sits So where is the threshold? This is where I have to be careful, because the exact numbers are the sort of thing that gets revised, and I would rather you understand the shape of the rule than memorise a figure that might change — always confirm the current thresholds with your installer or DNO. But the shape has been stable and it is what you need.\nThe dividing line is defined in terms of the generation\u0026rsquo;s rating per phase, and the number that matters is 16 amps per phase. Below that, on a fully type-tested system, you are in G98 territory. Above it, you are in G99 territory.\nFor a normal single-phase home — which is most UK homes — 16 amps at 230 volts works out to roughly 3.68 kW. That number, 3.68 kW, is the one you will hear again and again, and now you know where it comes from: it is the single-phase expression of the 16-amps-per-phase line.\nFor a three-phase supply, the same 16-amps-per-phase rule applies to each of the three phases, so the total is roughly three times as much — on the order of 11 kW before you cross into G99. This is one of the reasons a three-phase supply is quietly valuable for anyone with ambitions, and I will come back to it.\nHere is the comparison laid out, because a table is worth a hundred words of hedging:\nG98 G99 Process Notify after connection Apply and get approval before Timing Install now, inform within the window DNO assessment in the critical path Rough single-phase limit Up to ~3.68 kW (16 A per phase) Above ~3.68 kW Rough three-phase limit Up to ~11 kW (16 A per phase) Above ~11 kW DNO\u0026rsquo;s role Record-keeper Assessor and gatekeeper Typical answer Assumed yes Case-by-case, possibly with conditions Feels like A formality A negotiation A crucial subtlety that trips people up: the threshold is about the generation and export capacity, not simply the number of panels on your roof. A modern hybrid inverter\u0026rsquo;s rating, and specifically how much it can push to the grid, is what gets measured against the line. This is exactly why the export configuration matters so much, and it leads directly to the concept that ties the whole thing together.\nExport limitation: how to have a big system and stay small on paper Here is the clever bit, and the bit that most changed how I understood my own installation.\nSuppose you want a system that is genuinely larger than the G98 line — a decent-sized inverter and a big battery, because you want the ability to charge fast from cheap overnight electricity and run a lot of the house from stored energy. On paper that pushes you over 3.68 kW and into G99, with all the application and assessment that implies.\nBut what if you do not actually need to export at that full rate? What if the size is about your own use — self-consumption, fast charging, running the house — rather than about dumping large amounts back onto the grid? This is where export limitation comes in.\nAn export limitation scheme is a control function, built into the inverter or added alongside it, that actively caps how much power the system pushes to the grid, regardless of how big the system itself is. It measures the flow at your connection point and throttles export so it never exceeds a set limit — often, deliberately, set to the magic 3.68 kW. The relevant engineering recommendation here is EREC G100, which is the standard that governs how these export-limitation schemes must work to be trusted by the DNO. If you hear \u0026ldquo;G100\u0026rdquo; mentioned, that is what it is: the rulebook for capping export reliably enough that the network operator will believe your cap.\nThe result is that you can have a system that is physically capable of far more than 3.68 kW, but which is guaranteed never to export above the limit. Depending on the specifics and your DNO, this can keep you within a simpler connection process, or it can be the condition under which a G99 application is approved — \u0026ldquo;yes, you may install this larger system, provided export is limited to X.\u0026rdquo; Either way, the principle is the same: export limitation decouples how big your system is from how much it is allowed to push onto the grid.\nThis matters enormously if your goal is self-consumption and smart-tariff arbitrage rather than selling power back. My own battery optimiser mostly cares about importing cheaply and running the house from the battery, not about maximising export — for that kind of system, a sensible export limit costs you almost nothing in practice while keeping the connection simpler. But it is a decision, and it should be a conscious one, because the export limit you accept on installation day is a ceiling on part of your system\u0026rsquo;s future behaviour.\nA couple of worked examples Abstractions are slippery, so let me ground this in the kind of decisions real households actually face.\nExample one: the modest starter system. A single-phase house puts up a 3.6 kW array with a 3.6 kW hybrid inverter and a small battery. The inverter\u0026rsquo;s export sits at or below 3.68 kW. This is comfortably a G98 job: install it, notify the DNO within the window, done. No application, no waiting. Straightforward, and for a lot of households genuinely the right size. The catch, which nobody may mention, is that you have now built a system with essentially no headroom above the G98 line — which becomes the whole subject of the next article when you decide, two years later, that you want more.\nExample two: the ambitious self-consumer. A single-phase house wants a 5 kW inverter and a large battery, because they are on a half-hourly tariff and want to charge hard in cheap windows and run the house from storage. That inverter is over the G98 line. Two routes open up. One: apply under G99 and let the DNO assess it, possibly approving full export, possibly approving with a cap. Two: install the larger system but configure a G100-compliant export limit at 3.68 kW, so the system is big for the household\u0026rsquo;s own use but never exports beyond the line. Which route is right depends on whether they actually want to export at high power — and if they mostly want self-consumption, the export-limited route can be much simpler. The important thing is that this is a fork in the road with real consequences, and it should be chosen deliberately, not defaulted into.\nExample three: the three-phase household. A house with a three-phase supply has roughly three times the G98 headroom — around 11 kW — before G99 applies. This is why anyone with three phases, or the option to get them, has quietly more room to grow. A system that would force a single-phase home into a G99 application might sit comfortably under G98 on three phases. If you have three phases and did not know it mattered, it matters.\nThe misconceptions worth killing Because I held some of these myself, here are the ones I most want to correct.\n\u0026ldquo;The form is just a formality my installer handles.\u0026rdquo; Sometimes it is — for a small G98 job the notification genuinely is routine. But treating all of it as a formality is how you end up over the line without an approved G99 application, or accepting an export limit you never understood. The form is a formality only when you are firmly in G98 territory and someone has actually confirmed that. Otherwise it is a decision wearing the costume of a formality.\n\u0026ldquo;G99 means I can\u0026rsquo;t have a big system.\u0026rdquo; Wrong, and the opposite of the truth. G99 is the route to a big system. It is the process by which you get approval for something above the standard threshold. It is more work and it takes longer, but it is the door, not the wall.\n\u0026ldquo;Export limit is about being green / being nice to the grid.\u0026rdquo; The export limit is a technical and regulatory constraint that also happens to shape your economics. It is not a moral setting. Understanding it as \u0026ldquo;the maximum rate at which my system can sell or spill to the grid\u0026rdquo; is far more useful than thinking of it as some kind of environmental courtesy.\n\u0026ldquo;Battery capacity is what the DNO cares about.\u0026rdquo; Not directly. The DNO cares about the rate at which you can export — power, measured against that 16-amps-per-phase line — not how many kilowatt-hours you can store. You can have a very large battery behind a modest, export-limited inverter and remain small in the eyes of the grid code, because what matters is the size of the tap, not the size of the tank.\n\u0026ldquo;Once it\u0026rsquo;s set, it\u0026rsquo;s set.\u0026rdquo; Also wrong, and this cuts both ways. You can go back to the DNO to increase an export limit or upgrade a system — but it is another application, another assessment, and potentially another cost, especially if your local network is constrained. It is not that change is impossible. It is that change made later is more expensive and slower than headroom designed in earlier, which is the theme of the whole series.\nWhy understanding this before you install saves money Let me connect it back to the pound notes, because this is not an academic exercise.\nEvery decision in the G98/G99/G100 space sets a ceiling, and ceilings are cheap to raise before the build and expensive to raise after. If your installer sizes an inverter with no headroom above 3.68 kW because it kept you in the simple G98 process, that was probably the right call for the system you asked for — but if your real intention was to grow, you have just bought a component you will have to replace rather than one you can build on. Replacing an inverter to add capacity later can cost thousands. Choosing a slightly larger, export-limited inverter on day one might have cost very little and left the door open.\nSimilarly, if you know you will want to expand — a second battery, an EV charger folded into the same energy strategy, more panels — then the conversation about whether to go G99 from the start, or to design for a straightforward future application, is a conversation to have now, while the walls are open and the design is fluid, not in two years when every change is a retrofit. The DNO application you might need later is far easier to plan for than to discover you needed.\nAnd there is a quieter saving: understanding your own export limit lets you configure your system to actually exploit your tariff. If you know your system is export-limited to 3.68 kW, you design your self-consumption and battery strategy around that reality rather than around a fantasy of unlimited export. My battery makes money precisely because I understand what it is and is not allowed to do at the grid boundary. A system whose owner does not know its own limits cannot be optimised against them.\nThe lessons I took from getting this wrong I did not lose money on this, but only because my ambitions arrived late enough that I could still plan around the constraints I had unknowingly accepted. Here is what I would tell my earlier self.\nThe regulation is a design input, not paperwork. I filed G98/G99 mentally under \u0026ldquo;boring admin the installer does,\u0026rdquo; and that was the mistake. It is not admin. It is one of the constraints that shapes what your system can be, exactly like the size of your roof or the position of your consumer unit. Treat it as a design input from the first conversation.\nKnow your number. You should be able to say, out loud, what your export limit is and which process — G98 or G99 — your installation went through. If you cannot, you do not yet understand your own system, and neither, possibly, does whoever installed it. This is not hard information to have. It is just information nobody hands you unless you ask.\nHeadroom is the whole game. Almost every regret in this space is a headroom regret — a limit accepted, a component sized exactly to requirement, a ceiling set at today\u0026rsquo;s need with nothing above it. Understanding G98 and G99 early is really just understanding where the ceilings are early, so you can decide consciously how much room to leave above your head.\nSummary G98 and G99 are Engineering Recommendations from the Energy Networks Association that govern how generation — solar, batteries, hybrid inverters — connects to the distribution network in Great Britain. They exist for genuine reasons: voltage stability, equipment protection, and the safety of anti-islanding. The party on the other end is the DNO (Distribution Network Operator), who owns the wires — not your energy supplier. G98 and G99 define the terms of your relationship with them. G98 is notification: small, fully type-tested systems can be installed and then reported to the DNO. G99 is permission: larger systems must be applied for and approved before connection, with a real assessment in the critical path. The dividing line is 16 amps per phase — roughly 3.68 kW single-phase, roughly 11 kW three-phase. The threshold is about export/generation rating, not the number of panels or the size of the battery. Export limitation (governed by EREC G100) decouples how big your system is from how much it can push to the grid. A large, export-limited system can stay simple on paper — valuable for self-consumption and smart-tariff strategies where you do not need high-rate export. Every one of these decisions sets a ceiling, and ceilings are cheap to raise before installation and expensive afterwards. Understanding G98, G99 and export limits before you build is some of the cheapest insurance available. Know your own number: what your export limit is, and which process your system went through. If you cannot state it, you do not yet fully own your system. Next in the series: if I started my solar installation again, I\u0026rsquo;d design it very differently — the hindsight article, and the one where all these ceilings turn out to matter.\n","date":"2026-07-10T00:00:00Z","permalink":"/energy/g98-vs-g99-explained/","title":"G98 vs G99 Explained Without the Industry Jargon"},{"content":"The Solar Knowledge Gap Nobody Wants to Talk About The moment that made me start paying proper attention had nothing to do with my solar panels. It happened while an electrician was installing my EV charger.\nHe was competent, friendly, and clearly experienced with the physical work. He mounted the unit cleanly, ran the cable properly, and got everything terminated the way you would want. Then the commissioning app threw up a configuration screen, and one of the options was labelled G98. He paused. He read it twice. And then he did the thing that told me everything I needed to know: he took out his phone and rang another installer to ask what it meant.\nThe advice came back down the line, and I could hear it clearly enough to write it down.\n\u0026ldquo;Just press OK. It\u0026rsquo;ll go away.\u0026rdquo;\nHe pressed OK. It went away. He later admitted, without much embarrassment, that he did not actually know what G98 was. It was a box on a screen that appeared between him and finishing the job, and the fastest way through it was to make it disappear.\nI want to be very careful here, because this article is not about that electrician, and it is certainly not about the company he worked for. He was doing what almost anyone would do under time pressure on a job that was, from his point of view, essentially finished. The problem is not one person. The problem is that a screen asking about grid connection compliance appeared during a domestic installation, and nobody in the chain — not him, not the person he phoned — could say what it was for. That is not a personal failing. That is a knowledge gap, and it is systemic, and it is getting wider.\nWhy a single dismissed dialog box bothered me so much On its own, pressing OK on a G98 prompt during an EV charger install is probably harmless. An EV charger is a load, not a generator, and in most configurations the grid-code question is not the thing that will burn your house down. If that were the whole story I would have forgotten about it by the weekend.\nWhat bothered me was what it implied about everything else.\nI already had solar panels and a home battery. I run a battery optimiser that schedules charge and discharge against the Octopus Agile price curve and a solar forecast, so I had spent a fair amount of time living inside the technical detail of my own installation. I knew that G98 was not a nuisance dialog. It was a reference to the rules that govern how generation connects to the distribution network — the same rules that decide how much I am allowed to export, whether my setup was notified or approved, and crucially what I am allowed to add later without going back to the network operator for permission.\nIf the person installing energy equipment in my home treated that as a box to dismiss, the obvious question was: how much of the rest of it had been treated the same way? Not by him specifically — by the industry. How many systems have been signed off with a setting nobody understood, a limit nobody checked, and an export configuration that made sense to a piece of software and to no human being in the process?\nThe dismissed dialog was not the danger. It was the symptom. It told me that the complexity of these systems had outrun the average installer\u0026rsquo;s understanding of them, and that the gap was being papered over with \u0026ldquo;just press OK\u0026rdquo; rather than closed with knowledge.\nThe technology got hard while nobody was looking Here is the thing that I think genuinely explains the gap, and it is worth saying plainly because it is nobody\u0026rsquo;s fault in particular.\nA domestic solar installation used to be simple. Ten or fifteen years ago, a typical job was panels on a roof, a string inverter on a wall, and a connection that pushed whatever you generated back into the grid when you were not using it. The electrician\u0026rsquo;s job was electrical: mount, wire, terminate, test, certify. The \u0026ldquo;smart\u0026rdquo; part of the system was a generation meter and a Feed-in Tariff form. You could be a superb installer and never once have to reason about grid interaction beyond \u0026ldquo;it exports when the sun shines.\u0026rdquo;\nCompare that to what a modern installation actually contains. A current job might include:\nSolar panels, often more of them than the old Feed-in Tariff era, on multiple roof faces with different orientations. A home battery, with its own chemistry, its own state-of-charge management, and its own safety envelope. A hybrid inverter that does the job of a solar inverter and a battery inverter at once, and makes real-time decisions about where energy flows. Smart tariffs with half-hourly pricing, which turn \u0026ldquo;when should this charge\u0026rdquo; from a fixed setting into a moving optimisation problem. An EV charger, which is frequently the single largest load in the house and increasingly wants to coordinate with everything else. Home automation, because people want to see and control all of this, usually through something like Home Assistant. Backup or islanding capability, so the house can keep some circuits alive when the grid drops. Export limitation, where the system is deliberately configured to cap how much it pushes back to the grid. Every one of those is a discipline. A hybrid inverter is a power-electronics device running control software that decides, many times a second, whether to charge the battery, discharge it, import, export, or hold. Battery chemistry determines how hard you can push the cells and for how long. Export limitation is a grid-compliance function that has legal weight behind it. Smart tariffs turn the whole thing into a scheduling problem where the wrong default quietly costs you money every single day.\nThe installation is no longer an electrical job with some software bolted on. It is a small energy system, and it has the property that every good system has and every difficult one has too: the components interact. The battery\u0026rsquo;s behaviour depends on the tariff. The export limit depends on the inverter\u0026rsquo;s firmware. The EV charger\u0026rsquo;s load-management depends on whether it can see the rest of the system. You cannot fully understand any one part by looking at it in isolation, which is exactly the property that makes these systems hard to install well and easy to install badly.\nAnd here is the uncomfortable part. The trade that installs these systems is, for entirely understandable reasons, still largely trained and certified around the electrical work. The electrical work is genuinely the safety-critical bit, and I would never diminish it. But the electrical competence that gets someone qualified says very little about whether they understand hybrid inverter firmware behaviour, DNO application processes, battery cycle economics, or how a half-hourly tariff should shape the system\u0026rsquo;s configuration. The certification kept pace with the wiring. It did not keep pace with the system.\nThe customer now often knows more than the installer, and that is new There is a second shift that makes the gap more visible than it used to be, and it genuinely surprised me: the customer is frequently the most technically curious person in the room.\nThe people buying these systems now are often not passive. They have spent evenings reading forums. They have watched people on YouTube tear down inverters and benchmark battery round-trip efficiency. They arrive with a spreadsheet modelling their own consumption. Some of them — and I will happily put myself in this category — run home automation and monitoring that gives them more visibility into their own system\u0026rsquo;s real behaviour than the installer has ever had into any system they have fitted, because the installer moves on to the next job and never sees the year of data that follows.\nI do not say this to flatter customers or to embarrass installers. I say it because it inverts an assumption that both sides still quietly operate on: that the professional in the van is the most knowledgeable person present. For the electrical safety of the installation, they absolutely should be, and usually are. But for the system design — the sizing, the tariff strategy, the export configuration, the future expansion path — the balance has shifted in a way that neither party is quite prepared for.\nI have watched this play out. I have asked installers questions about export limits and future battery expansion and watched a flicker of \u0026ldquo;I\u0026rsquo;ll have to check\u0026rdquo; cross their face — which, to be clear, is the right answer, and far better than a confident wrong one. The failure mode is not \u0026ldquo;I don\u0026rsquo;t know.\u0026rdquo; The failure mode is \u0026ldquo;just press OK.\u0026rdquo; One is honesty about the edge of your knowledge. The other is the pretence that the edge is not there.\nThe dangerous installer is not the one who says \u0026ldquo;I\u0026rsquo;m not sure.\u0026rdquo; It\u0026rsquo;s the one who has never noticed there was anything to be unsure about.\nWhat actually goes wrong in the gap Let me be concrete, because \u0026ldquo;knowledge gap\u0026rdquo; is a soft phrase and the consequences are not soft. These are the failure patterns I have either seen, been warned about by people I trust, or come close to living myself. None of them are dramatic. All of them cost money, options, or both.\nThe system that cannot grow The most common one, and the subject of much of the rest of this series, is the installation that works perfectly on day one and quietly forecloses your future. An inverter sized with no headroom, so adding a second battery later means replacing it. Conduit packed exactly full, so pulling one more cable means lifting the drive. An export configuration chosen to make a form go away, which caps you at a limit you did not know you were accepting. The system meets the brief. The brief just never included next year.\nThe configuration nobody can explain A setting was chosen — an export limit, a charge rate, a grid-code parameter — and the reason lives in nobody\u0026rsquo;s head. When something needs to change, or when a second installer is called to expand the system, they inherit a black box. I have written before about the load-bearing thing nobody documented in the context of a homelab, and it is exactly the same disease here: a system whose current state cannot be explained is a system you cannot safely modify.\nThe tariff left on the table A battery and a half-hourly tariff are a money-making combination if the system is configured to exploit the price spread. Left on a naive default — \u0026ldquo;charge overnight, discharge in the evening\u0026rdquo; — the battery does something plausible and mediocre, and the owner never knows how much they are leaving on the table because nobody told them there was a table. This one is invisible precisely because the system still works. It just works worse than it should, every day, forever.\nThe compliance box ticked without being true This is the one the opening story points at. Somewhere in the chain a grid-code question was answered, a form was submitted or not submitted, an export limit was set or assumed — and the person who did it could not have told you what it meant. Most of the time nothing visible happens, which is the trap. The network operator has a record of your connection that may not match reality, and you find out only when you try to expand, or when someone eventually audits, or when a neighbour\u0026rsquo;s complaint about voltage brings scrutiny you did not expect.\nNotice the common thread. In every case, the system works. That is what makes the gap so hard to see and so easy to sell around. A dismissed dialog box does not throw an error. A too-small inverter still inverts. A naive tariff schedule still charges the battery. The cost is not a failure you can point at. It is an option you no longer have, a bill that is quietly higher than it needed to be, or a record that does not match your installation. The gap does not announce itself. It just sits there, priced into your next ten years.\nWhy the cheapest quote is a question, not an answer I understand the appeal of the cheapest installer. Solar is expensive, the panels are largely a commodity, and if two companies will bolt the same modules to your roof, why pay more? For a simple grid-tied array with no battery and no ambitions, that reasoning is often fine, and I am not going to pretend otherwise.\nBut the moment your system crosses into the territory above — battery, hybrid inverter, smart tariff, EV, any thought of future expansion — the cheapest quote stops being a straightforward answer and becomes a question you have to interrogate. Cheap on what, exactly?\nPrice competition pushes on the things a customer can compare on a quote: panel wattage, battery capacity, headline cost. It does not push on the things that are invisible on a quote and expensive later: whether the inverter has headroom, whether the export was applied for correctly, whether the cable routes leave room to grow, whether anyone in the company can explain the grid-code settings they applied. Those things cost the installer time and knowledge, which is precisely what gets squeezed out when the only competition is on price.\nI am not arguing that expensive means competent. Plenty of expensive installers are coasting on the same knowledge gap with a better van. The point is subtler and more useful: price tells you almost nothing about the thing that actually matters here, which is understanding. A cheap installer who can talk fluently about DNO applications and future expansion is a bargain. An expensive one who reaches for \u0026ldquo;just press OK\u0026rdquo; is a rip-off at any price. The number on the quote is not measuring the variable you care about.\nThis is the same instinct I bring to enterprise infrastructure, where the cheapest bid is routinely the most expensive decision once you price in what it cannot do later. I will draw that comparison out properly in what enterprise IT taught me about designing better solar systems, because it is not a loose analogy — the failure modes are structurally identical.\nBest practice, from someone who lives with the result If the gap is real, what do you actually do about it as a homeowner? Not become an expert — you do not have to. But you do have to change the shape of the conversation. Here is what I would tell anyone about to spend serious money on one of these systems.\nTreat the installer\u0026rsquo;s answers as data about the installer. When you ask a question, you are not only gathering information about your system. You are measuring whether the person answering understands what they are fitting. A good installer welcomes the questions and answers them, or tells you honestly where the edge of their knowledge is. A worrying one gets defensive, hand-waves, or reaches for reassurance instead of explanation. The questions are a competence probe as much as a fact-finding exercise.\nAsk about the future, not just the install. \u0026ldquo;Can I add a second battery later?\u0026rdquo; \u0026ldquo;What\u0026rsquo;s my export limit and why?\u0026rdquo; \u0026ldquo;If I want to add an EV charger next year, does this design leave room for load management?\u0026rdquo; These questions reveal whether the system was designed as a snapshot or as something that can grow. I have a whole checklist of the questions worth asking later in this series, precisely because the right questions are the cheapest insurance you can buy.\nGet the reasoning, not just the number. For every non-obvious choice — the inverter size, the export limit, the charge rate, the grid code — ask why that value. You are not trying to catch anyone out. You are trying to establish that a reason exists and lives in a human head, because a value chosen with a reason can be revisited and a value chosen to dismiss a dialog cannot.\nInsist on the paperwork being real. The DNO application, the commissioning certificate, the export configuration — these should be documents that describe your actual installation, handed to you, that you keep. If the grid connection was \u0026ldquo;notified,\u0026rdquo; you should have the notification. If it was \u0026ldquo;approved,\u0026rdquo; you should have the approval. A system whose paperwork matches its reality is a system someone understood well enough to document, which is most of the battle.\nThe lessons I actually took from it I took three things from watching a competent tradesman dismiss a grid-code dialog he could not name, and they have shaped how I think about my own system and how I would advise anyone else.\nThe first is that complexity migrated into these systems faster than expertise did, and the two have not caught up with each other. This is not cynicism. It is just what happens when a product category evolves from \u0026ldquo;panels and an inverter\u0026rdquo; into \u0026ldquo;a coordinated energy system\u0026rdquo; in the span of a decade while the training and certification evolve at the pace institutions evolve, which is slower. The gap is a timing problem, and timing problems close eventually — but not before a lot of systems get installed inside the gap.\nThe second is that the customer\u0026rsquo;s curiosity is now a load-bearing part of getting a good outcome. That is a strange thing to have to say. You should be able to hand the job to a professional and trust it. For the electrical safety you largely can. For the system design, the honest reality of this moment is that an engaged, questioning customer gets a materially better installation than a passive one, because the questions force the design decisions into the open where they can be got right. I wish it were otherwise. It is not, yet.\nThe third is the one I keep relearning across every technical thing I touch: a system that works is not the same as a system that was understood. The whole trap of the knowledge gap is that its failures are silent. Nothing catches fire. The panels generate, the battery charges, the app shows a reassuring flow diagram. And underneath, a limit you did not choose, a form that does not match, or an option you have quietly lost sits waiting to matter. Working and understood are different properties, and only one of them protects your next ten years.\nWhat I want you to do with this I am not trying to make you distrust installers. Most are decent, skilled people doing hard physical work in a trade that changed under their feet. I am trying to make you distrust the quiet. The renewable industry has a strong incentive to keep the conversation smooth and reassuring, because friction loses sales, and \u0026ldquo;just press OK, it\u0026rsquo;ll go away\u0026rdquo; is the customer-facing version of that same instinct — smooth the friction, close the job, move on.\nYour defence against the gap is not technical knowledge you do not have. It is the willingness to ask, and to keep asking until you get a reason rather than a reassurance. Ask what the grid code on your system is and why. Ask what your export limit is and whether it constrains your future. Ask whether the design leaves room to grow. Ask, when someone tells you to just press OK, what it is that you would be pressing OK to.\nThe marketing material will not prompt those questions, because the marketing material is designed to make you feel that everything is handled. Sometimes it is. But the only way to know the difference between a system that was understood and a system that merely works is to ask the questions that the \u0026ldquo;just press OK\u0026rdquo; school of installation is built to avoid.\nSummary A skilled installer dismissed a G98 grid-code dialog during my EV charger install because neither he nor the person he phoned knew what it meant. The dismissed box was harmless in itself; what it revealed was not. Domestic renewable systems have evolved from \u0026ldquo;panels and an inverter\u0026rdquo; into coordinated energy systems — solar, batteries, hybrid inverters, smart tariffs, EV charging, automation, backup and export limitation — whose components interact in ways that are genuinely hard to reason about. Installer training and certification kept pace with the electrical work, which is the safety-critical part, but not with the system design — the grid applications, tariff strategy, sizing headroom and expansion planning that decide whether the system is any good over ten years. The failures in this gap are silent: the system still works. A too-small inverter still inverts, a naive tariff still charges, a mis-set export limit throws no error. The cost shows up as lost options, higher bills, and records that do not match reality. The cheapest quote is not an answer but a question. Price competes on what a customer can see on a quote and never on the invisible understanding that actually matters. Your defence is not expertise you lack. It is asking for reasons instead of reassurances, and treating the quality of the answers as a measure of the person giving them. When someone tells you to just press OK, ask what you would be agreeing to. Next in the series: G98 vs G99 explained without the industry jargon — a plain-English guide to the exact regulation that dialog box was asking about, and why understanding it before you install can save you real money later.\n","date":"2026-07-03T00:00:00Z","permalink":"/energy/solar-knowledge-gap/","title":"The Solar Knowledge Gap Nobody Wants to Talk About"},{"content":"How I Broke Windows Update by Freeing Up Disk Space Every self-inflicted outage starts with a small act of tidiness. Mine started with me looking at a Windows Server 2025 box, noticing that C:\\Windows\\Installer had quietly swollen to several gigabytes of files I did not recognise, and deciding — reasonably, I thought — to clear it out and reclaim the space. The server thanked me by working perfectly for weeks. Then Windows Update tried to upgrade PowerShell, and the bill came due.\nThe machine in question was not a toy. It was running Veeam Backup \u0026amp; Replication — which is to say it was the one box in the lab whose entire job is to be trustworthy, the safety net under everything else. And it had started rebooting itself, unprompted, shortly after an update attempt. The thing that protects me from my own mistakes had been taken down by exactly one of them.\nC:\\Windows\\Installer is not the junk it looks like. It is the receipt Windows keeps for every MSI you have ever installed, and Windows Installer cannot repair, upgrade or remove what it has no receipt for.\nThe symptom that made no sense The first symptom was the alarming one: a production Windows Server 2025 VM restarting on its own. The second symptom is what turned a vague worry into a specific problem. Instead of a normal Windows Update failure — a hex error code, a line in the update history, something I could search — the installer kept stopping to ask me for something absurd:\nPowerShell 7-x64 is on a CD-ROM or other removable disk. Insert the disk and click OK.\nThere was no disk. There never had been a disk. PowerShell had arrived on that server the way everything does now — as a download, years ago. The update could not complete, the server was left in an unstable state, and the existing PowerShell still ran but reported versions that did not agree with each other. A prompt for media that has never existed is the kind of error that makes you doubt the machine before you doubt yourself, which, as it turned out, was exactly the wrong way round.\nWhat the packages were telling me, and why I didn\u0026rsquo;t believe them When three tools disagree about something as simple as \u0026ldquo;which version is installed\u0026rdquo;, the disagreement is the clue. So I asked all three.\nGet-Package cheerfully reported two versions living side by side, 7.4.13 and 7.4.14, as if the upgrade had half-happened:\nGet-Package -Name \u0026#34;PowerShell*\u0026#34; # reported both 7.4.13 and 7.4.14 The actual executable disagreed. Get-Command still pointed at the old binary, untouched:\nGet-Command pwsh # C:\\Program Files\\PowerShell\\7\\pwsh.exe → 7.4.13 And the MSI database — the thing Windows Installer itself trusts — only believed in one of them:\nGet-WmiObject Win32_Product | Where-Object { $_.Name -like \u0026#34;*PowerShell*\u0026#34; } | Select-Object Name, Version, IdentifyingNumber # PowerShell 7-x64 # 7.4.13 # {BD168355-B595-4C5B-B9ED-05C4714A01A5} (A small aside, because knowing your tools\u0026rsquo; sharp edges is half the job: Win32_Product is a slow and slightly dangerous way to ask this question — enumerating it makes Windows Installer run a consistency check against every installed MSI as it goes — but it reads from the same place the upgrade does, which is precisely why I wanted its answer here.)\nThree tools, three stories. The reconciliation was simple once I stopped arguing with it: the 7.4.14 upgrade had never actually landed. The package provider had noticed a newer thing being attempted and listed it; the filesystem and the MSI database both still belonged entirely to 7.4.13. The upgrade was stuck halfway through a door it could not open, and the whole estate was being held hostage by whatever was jamming that door.\nThe cause, which was me Here is the part I would bolt to the front of the runbook. The upgrade could not complete because Windows Installer could not get at the original 7.4.13 installation to remove and replace it — and it could not get at it because I had deleted the cache that held it.\nC:\\Windows\\Installer is not scratch space. It is where Windows Installer stashes a copy of the MSI (and any patches) for every product on the machine, so that later — when you repair, upgrade or uninstall something — it has a known-good source to work from without asking you for anything. A PowerShell point upgrade is, under the bonnet, a major upgrade: remove the old product, install the new one. The \u0026ldquo;remove the old product\u0026rdquo; step needed the cached 7.4.13 MSI. I had thrown it away to save a few gigabytes, so Windows Installer did the only thing it knows how to do when its source is missing — it fell back to asking for the original media. That is the \u0026ldquo;please insert the disk\u0026rdquo; prompt: not a bug, just Installer politely requesting the receipt I had binned. And every time the update retried and failed, it dragged the server through another reboot cycle.\nI had assumed C:\\Windows\\Installer was Microsoft\u0026rsquo;s mess to clean up. It is the opposite — it is Microsoft\u0026rsquo;s safety copy, and deleting it is one of those housekeeping moves that does nothing bad at all until the precise moment something needs to be repaired. This is the home lab\u0026rsquo;s favourite kind of lesson: the consequence arrives weeks after the decision, with no obvious thread connecting them, so you get to learn causation the hard way. Obvious in hindsight. Invisible at the time.\nThe fix: stop guessing and repair the MSI The temptation at this point is to start randomly repairing things — re-run the update, click retry, point the prompt at any MSI you can find and hope. That is how a one-evening problem becomes a three-evening one. The discipline is to fix the actual thing that is broken: give Windows Installer back a valid source for the product it is trying to upgrade.\nSo I worked from the registration I had already confirmed — that product code above, {BD168355-…} — and went and got a clean, verified copy of the installer rather than trusting whatever a browser handed me. Winget will do exactly this, and because PowerShell\u0026rsquo;s MSI is built with WiX I could ask it for the installer directly:\nwinget download --id Microsoft.PowerShell --installer-type wix That sidesteps the whole circus of browser downloads being intercepted by security software, and it checks the installer hash for me so I am not guessing whether I have the right, untampered package. Then I ran it by hand through msiexec, letting Windows Installer do the upgrade properly against the existing registration:\nmsiexec /i \u0026#34;\u0026lt;path to the downloaded MSI\u0026gt;\u0026#34; That single step did everything I needed. Installer used the new MSI to complete the major upgrade, re-cached the package back into C:\\Windows\\Installer — quietly rebuilding the receipt I had deleted — and finished cleanly. PowerShell came up on the new version, Windows Update stopped asking for a disk that does not exist, and the server stopped throwing itself on the floor. The safety net was back under the trapeze.\nWhat I\u0026rsquo;d tell past-me Three things, because the mistakes taught me more than the fix did.\nC:\\Windows\\Installer is not yours to tidy. It looks like wasted space and it is anything but. If a server is genuinely tight on disk, the answer is more disk, a bigger volume, or DISM/component-store cleanup — never reaching into the Installer cache with a delete key. The space you reclaim is a loan against the next repair or upgrade, and the interest is brutal.\nDon\u0026rsquo;t trust a single package view. Get-Package, Get-WmiObject Win32_Product (or its modern Get-CimInstance form), and pwsh -Version can and will disagree, because they read different sources of truth. When they diverge, the divergence is the diagnosis — and the view that matters for an upgrade is the one Windows Installer itself trusts, which is the MSI database, not the package provider.\nwinget download is a recovery tool, not just an install one. When a browser download is failing, or being quietly intercepted, or you simply cannot be sure you have the right untampered file, winget download hands you a verified installer and removes the guesswork about provenance entirely. It earned its place in my toolkit on this exact problem.\nAnd underneath all three, the meta-lesson that keeps this site honest: I got to learn all of it at the price of one evening on my own hardware instead of discovering it on a client\u0026rsquo;s production backup server at 3am. That is the entire argument for the home lab as a learning platform — you make the expensive mistakes where the only thing on fire is your weekend. The fix went straight into my second brain the moment it worked, because the version of me who meets this again in two years will have completely forgotten the product code, the WiX flag, and the housekeeping that caused it.\nUseful commands For future-me, and anyone else staring at a phantom CD-ROM prompt:\n# What does each source actually believe is installed? Get-Package -Name \u0026#34;PowerShell*\u0026#34; Get-WmiObject Win32_Product | Where-Object { $_.Name -like \u0026#34;*PowerShell*\u0026#34; } pwsh -Version # Which binary is actually on PATH? Get-Command pwsh # Get a clean, hash-verified installer (the WiX-built MSI) winget download --id Microsoft.PowerShell --installer-type wix # Let Windows Installer repair/upgrade against the existing registration msiexec /i \u0026#34;\u0026lt;path to the downloaded MSI\u0026gt;\u0026#34; Closing thought The unexpected restarts were the symptom that made me look. The lesson underneath was older and quieter: a backup server I had subtly broken, weeks earlier, with an act of tidiness I never connected to the failure. When Windows Installer tells you a product is \u0026ldquo;on a CD-ROM or other removable disk\u0026rdquo;, it is almost never talking about the application. It is telling you its own bookkeeping is no longer consistent — that the receipt is missing — and the only real fix is to give it a valid source and let it put its records back in order.\nThe irony is not lost on me that the machine I undermined was the one whose entire purpose is to recover from disasters, and that I did it not with a risky change but with cleanup — the kind of \u0026ldquo;responsible\u0026rdquo; maintenance that feels like the opposite of a risk. That, more than the msiexec incantation, is what I took away. The dangerous changes announce themselves. It is the housekeeping you should keep an eye on. This one fits neatly alongside backups, the boring discipline: the boring, well-meant, routine work is exactly where the quiet failures hide.\n","date":"2026-06-29T00:00:00Z","permalink":"/infrastructure/how-i-broke-windows-update/","title":"How I Broke Windows Update by Freeing Up Disk Space"},{"content":"Building COSMOS: A Front End for the Second Brain I have spent two years arguing that you should build knowledge instead of documents. I have the posts to prove it, a second brain in Git to live by it, and Atlas reading the whole corpus back to me on demand. The philosophy was settled. The plumbing worked. And yet, most evenings, building a piece of that knowledge still meant the same thing it always had: a blank Markdown file, a blinking cursor, and the whole journey from raw capture to something publishable resting entirely on my own discipline.\nThat is the gap COSMOS exists to close. The second brain had a memory and a voice. What it never had was a front door — a place to actually do the work of turning a heap of captured fragments into a finished, durable note. So I built one. COSMOS — Content Organisation, Semantic Memory \u0026amp; Operational System, because every project of mine apparently needs a self-important acronym — is a self-hosted studio for exactly that journey. It is the hardest single thing I have built for this site, and I am prouder of it than I have any right to be.\nA blank page is where good notes go to stall. I got tired of mine stalling, so I built the workbench I wished I had.\nWhat COSMOS actually is COSMOS is a small web application you run yourself, in Docker, alongside everything else in the homelab. You open it in a browser and it gives you a single, opinionated pipeline that takes a thought from \u0026ldquo;I scribbled this at 11pm\u0026rdquo; all the way to \u0026ldquo;this is live on the site and the knowledge base is richer for it.\u0026rdquo;\nThe shape of that pipeline is the whole point, so here it is end to end:\nflowchart TD A[Inbox: notes, links, uploads] --\u003e B[AI draft generation] B --\u003e C[AI review: a second model critiques] C --\u003e D[Knowledge scoring: originality, overlap, confidence] D --\u003e E[Revision, with full version history] E --\u003e F{Human approval} F --\u003e|approved| G[Publish into Hugo] F --\u003e|not yet| E G --\u003e H[Markdown in Git] H --\u003e I[Knowledge engine: search, related, graph] I --\u003e BYou capture raw material into an inbox — a pasted note, a saved link, an uploaded Markdown file — with no decisions required, because friction at the moment of capture is what kills every system like this. You point an AI writer at that material and it produces a first draft. A second model then reviews the draft critically, across multiple quality dimensions, the way a good editor would. The work gets scored — how original is this, how much does it overlap with things I already know, how confident should I be in it. You revise, with every version kept and restorable. And only when a human — me — clicks approve does anything get published, copied as a clean Markdown page bundle straight into this site\u0026rsquo;s Hugo content.\nNothing is automatic that should not be. COSMOS never publishes on its own and it never touches Git for me; it surfaces the exact command and lets me run it. The machine drafts, critiques and scores. The judgement stays mine.\nTaming the mountain The phrase I kept coming back to while building it was mountains of information. That is genuinely what the raw material is. Years of daily notes, half-finished thoughts, links I meant to read, transcripts, error messages and their fixes. The problem was never a shortage of input. It was that the input sat in a heap, and turning a heap into knowledge is slow, deliberate work that I kept not doing because the activation energy was too high.\nCOSMOS lowers that activation energy at every step. The inbox means capture costs nothing. The drafting means I start from a critiqued first pass instead of a blank page. The scoring means I can see, before I sink an evening into a piece, whether it is actually new or whether I am about to rewrite something I already said better eighteen months ago. The mountain stops being intimidating because the tool is built to climb it one switchback at a time, and to remember every step so the next climb is shorter.\nThat last part matters more than any single feature, and it is the same compounding argument I have made before: the work accumulates instead of evaporating. Every note that goes through COSMOS makes the next one faster, because the next one is partly assembled from notes that already exist, already corrected, already scored.\nKnowledge that knows itself The piece I am most quietly pleased with is the knowledge engine, because it is the bit that turns a publishing tool into something closer to a brain.\nCOSMOS embeds your entire Markdown corpus — this site\u0026rsquo;s content, plus everything it has published — into a local vector index. From that, it does the things a filing cabinet cannot. Semantic search, so I can ask for a concept and get the passages that mean it rather than the ones that happen to share a keyword. Related-article detection while I draft, so I am nudged toward the links I would otherwise forget to make. Duplicate and heavy-overlap warnings, so I stop accidentally re-deriving the Citrix sizing rationale for the fourth time. A similarity graph over the whole base, so the structure of my own thinking becomes something I can actually see. And a retrieval-augmented pass that reads a new draft against the existing corpus and tells me where it contradicts something I have already published, or leans on a reference I never wrote.\nI built the vector maths in plain Python with no heavyweight numerical dependencies, because at personal scale — tens of documents, hundreds of chunks — a careful dot product is more than fast enough, and I would rather own a hundred lines I understand than inherit a framework I do not. The on-disk Markdown is canonical; the index is disposable and rebuildable from scratch. That is the same principle as everything else I build: the plain text is the asset, and every clever layer on top of it has to be willing to be deleted and regenerated without losing anything that matters.\nThe boring stack, on purpose The architecture is deliberately unfashionable, and that is a feature. FastAPI on the back. HTMX and Jinja templates on the front, which means there is no Node build, no bundler, no thousand-dependency front-end toolchain quietly rotting — just HTML the server renders and swaps. SQLite for workflow state. A thin provider layer so the AI backend is swappable: local models on Ollama when I want privacy and zero marginal cost, an OpenAI-compatible endpoint or Anthropic when I want more power, and different providers for writing, reviewing and embeddings if I feel like mixing them.\nFastAPI · HTMX + Jinja · SQLite · Markdown on disk │ Provider layer Ollama · OpenAI-compatible · Anthropic Every layer is replaceable and the value lives in the bottom one. The models can change. The provider can change. The whole application could be thrown away tomorrow and the knowledge would be entirely intact, because it was never in COSMOS — it was always plain Markdown in Git, and COSMOS is just the best front door I have ever had to it. That is the same conviction that took this site off a platform and onto Hugo and Git, applied one layer up.\nWhat it cost, honestly I would be lying if I made this sound clean. It was not. It was weeks of evenings, more rewrites than I would like to admit, and a long tail of the unglamorous, fiddly problems that never make it into the architecture diagram.\nThe pipeline looks obvious in hindsight and was anything but while I was building it. Getting a second model to critique a first model\u0026rsquo;s output usefully — specific, grounded, not just a flattering paragraph of \u0026ldquo;this is great, here are three vague suggestions\u0026rdquo; — took a dozen passes at the prompts alone. The scoring had to be honest enough to tell me my draft was derivative, which meant resisting the temptation to build a tool that always told me I was clever. The publishing path had to be genuinely safe: never overwrite, never escape the content directory, handle image-bearing posts as proper page bundles, set the front matter correctly every time, because a publishing tool you cannot trust is worse than no tool at all. And the knowledge engine went through three shapes before I had one that returned results I actually believed.\nNone of that is hard in the way that sounds impressive at a conference. It is hard in the way real software is hard — a hundred small correct decisions, each boring on its own, that collectively decide whether the thing is a toy or something you reach for every day. I got plenty of them wrong first. I am sure some are still wrong. But it works now, it works well, and the version of me that started this with a vague idea and a blank repository would not quite believe how far it got. This is also exactly the failure mode I warned about in why most AI projects fail: the demo is easy and the last ten percent is the whole job. I tried very hard to do the last ten percent.\nWhy I am proud of it I do not say that easily, because pride is usually the noise you make right before something breaks. But I have used COSMOS to build real notes that are live on this site now, and the experience is the one I had been chasing for two years. The blank page is gone. The mountain of raw material has a path up it. The knowledge I build today genuinely makes tomorrow\u0026rsquo;s faster, and I can see it happening in the graph and the scores.\nIt is the front end the second brain was always missing. Atlas gave the corpus a voice; COSMOS gives it a workbench. Between them, the loop I have been describing across half this site — capture, build, connect, retrieve, act — is finally a loop I can stand inside rather than one I have to hold together by hand.\nCome and break it Here is the part I am slightly nervous about: COSMOS is open source, and you can run it yourself right now.\n→ github.com/sofapi/cosmos\nIt is a docker compose pull \u0026amp;\u0026amp; docker compose up -d away from running on your own machine, on your own models, against your own Markdown — nothing leaves your network if you point it at a local model. There is a setup guide in the repo. Bring an existing site and it will index your back catalogue and let you import your published posts as a starting point. Bring nothing and start from an empty inbox.\nWhat I would genuinely love is feedback. Tell me where the pipeline feels wrong. Tell me the prompt that produced a rubbish review. Open an issue when it breaks, because it will, and I would much rather hear about it than not. If you build something on top of it, or rip a piece out for your own setup, even better — that is the whole point of putting it in the open. I have spent a long time on this thing in private; I am ready for it to meet some people who did not build it and have no reason to be gentle.\nThe thing I keep relearning Every project on this site ends up teaching me the same lesson in a new accent, and COSMOS is no exception. The tool is not the asset. The Markdown is the asset, and it always was. What COSMOS changes is not what I own but how cheaply I can turn what I capture into what I keep — and lowering that cost, it turns out, is the difference between a knowledge base that compounds and one that quietly stalls at the blank page like mine used to.\nI built a memory. I built a voice. This year I finally built the front door, and I worked harder on it than on anything else here. It is open now, and the door is unlocked. Come in and have a look around.\n","date":"2026-06-28T00:00:00Z","permalink":"/projects/cosmos/","title":"Building COSMOS: A Front End for the Second Brain"},{"content":"AI Battery Optimiser For about a year my home battery did something almost criminally stupid every single night. It charged at full whack from the grid at 02:00, because that is when the cheap window used to be, and then it sat there proudly full while the sun did the actual work the next morning. By lunchtime the panels had nowhere to put their output and I was exporting electricity for pennies that I had bought, hours earlier, for more pennies.\nThe battery was working exactly as configured. That was the problem. It was following a fixed schedule I had set once and never revisited, and a fixed schedule is wrong on every day that does not look like the day you set it up on. Some days are sunny. Some days are filthy with cloud. And on a half-hourly tariff the price of grid electricity swings around so violently that \u0026ldquo;charge at night\u0026rdquo; is not a strategy, it is a superstition.\nSo I did what I keep doing in this lab: I replaced a static rule with something that looks at the actual conditions and makes a decision. This is the story of the battery optimiser I bolted onto Home Assistant — what it predicts, how it decides, and the unglamorous truth about how much \u0026ldquo;AI\u0026rdquo; is really in it. The honest answer is \u0026ldquo;not much, and that was the right call\u0026rdquo;, which is a theme I keep coming back to in why most AI projects fail.\nThe problem with a flat schedule Let me be specific about the money, because the money is the whole point.\nI am on a time-of-use tariff in the Octopus Agile style: the price of imported electricity is set in half-hour blocks and published the afternoon before for the following day. On a good day the overnight price might dip to a few pence per kWh, and the early-evening peak might be five or six times that. Occasionally, when the grid is awash with wind, the price goes negative and they pay you to consume. The spread between the cheapest and most expensive half-hour on the same day is routinely enormous.\nA flat schedule cannot see any of that. It charges in the same window whether that window is the cheapest of the day or merely average. Worse, it has no idea what tomorrow\u0026rsquo;s solar is going to do. If I fill the battery from the grid overnight and then wake up to a cloudless June day, I have paid for energy I did not need and I have left no room in the battery for free solar, so the surplus gets exported at the derisory export rate. I have managed to lose money twice with one decision.\nThe naive fixes are all rules, and rules accumulate. \u0026ldquo;Charge at night, but only to 80% in summer.\u0026rdquo; \u0026ldquo;Don\u0026rsquo;t charge if tomorrow\u0026rsquo;s forecast is sunny.\u0026rdquo; \u0026ldquo;Discharge during the evening peak, unless the battery is below 30%.\u0026rdquo; Every one of those is a reasonable instinct and every one is a hard-coded threshold that is wrong in some corner of the year. I have lived in that house of rules. It is exhausting to maintain and impossible to reason about once you have more than about four of them interacting.\nA schedule encodes a decision you made in the past. A forecast lets you make the decision with the information you actually have.\nThat sentence is basically the entire design philosophy, so I want to put it up front before I describe the build.\nThe decision: forecast-driven, not rule-based The core decision was to stop writing rules about when to charge and instead compute, fresh every evening, the cheapest possible way to get the house through the next 24 hours given everything I can forecast.\nThat reframes the whole thing as an optimisation problem. I am not asking \u0026ldquo;is it night-time?\u0026rdquo; I am asking \u0026ldquo;across the next 48 half-hour slots, when should the battery charge, when should it discharge, and when should it hold, so that my total cost of importing from the grid is as low as possible — subject to the battery never doing anything that harms it or leaves me stranded?\u0026rdquo;\nTo answer that I need three inputs, refreshed daily:\nA solar forecast for the next day, in roughly half-hourly resolution. I pull this from a Solcast-style forecast service that knows my panel orientation and tilt and gives me expected generation per slot. The price curve for the next day — the Agile import prices, and the export rate, fetched once the afternoon prices are published. My own household load profile — how much the house actually draws, slot by slot, learned from months of historical data rather than guessed. The third one is where the only genuine machine learning lives, and it is deliberately tiny. I will come back to that, because the restraint is the interesting part.\nI rejected two tempting alternatives. The first was buying a commercial \u0026ldquo;smart charging\u0026rdquo; add-on from the inverter vendor. It works, but it is a black box — I cannot see why it made a decision, I cannot tune the SoC targets, and it has no idea about my specific load shape. The whole reason I run a lab is so that I own the logic. The second was going the other way and building something far too clever: a reinforcement-learning agent that would \u0026ldquo;learn the optimal policy\u0026rdquo; over time. That is a thesis project, not a battery controller, and I talk below about why I am glad I did not.\nThe \u0026ldquo;AI\u0026rdquo; honesty This project lives under the AI category and I want to be straight about what that means, because the gap between what gets called AI and what is actually doing the work is the thing I find most tiresome in this industry.\nThere is no large language model anywhere near my battery. There would be no point. An LLM is a tool for turning language into language; deciding how many kWh to pull from the grid at 18:30 is a numerical optimisation over a known cost function. You do not need a 70-billion-parameter model that has read the entire internet to do arithmetic with constraints. You need a forecast and a solver.\nThe actual \u0026ldquo;intelligence\u0026rdquo; is two things working together. One is forecasting: the solar service predicts generation, and a lightweight model predicts my load. The load model is genuinely learned — it is a per-slot profile built from historical consumption, segmented by day-of-week and adjusted for recent drift, the sort of thing you can do with a few weeks of data and some sensible averaging, no GPU required. Two is optimisation: given those forecasts and the price curve, a small solver decides the charge/discharge/hold schedule that minimises cost.\nI resisted over-engineering this, and resisting was a conscious effort. It would have been more fun to train a neural net. It would not have been better. The optimisation problem here is small, well understood, and has a known structure, and throwing ML at a problem that linear programming already solves cleanly is how you end up with something impressive in a demo and unmaintainable in production. This is the same argument I make at length in why most AI projects fail: the failure is almost never the model, it is reaching for a model when you needed a function. Picking the right tool, even when the right tool is boring, is the actual skill. Or as I put it in AI is becoming infrastructure — the win is when the cleverness disappears into plumbing you stop thinking about.\nThe model is not the product. The decision is the product.\nHow it fits together The whole thing rides on the Home Assistant instance that already runs the real-world automation side of the lab I described in building an AI infrastructure lab at home. Home Assistant is the integration layer: it already talks to the inverter and battery over the vendor integration, it already has the solar forecast and Agile price sensors, and it already logs everything to its recorder database.\nThe optimiser itself is a Python routine that runs nightly under pyscript inside Home Assistant (AppDaemon would do the same job; I went with pyscript because it keeps the code inside the HA config and out of a separate container). It wakes up once the next day\u0026rsquo;s Agile prices are published, gathers the three inputs, runs the solve, and writes the resulting schedule into a set of helper entities. A handful of automations then read those entities and actually push charge/discharge commands at the inverter.\nflowchart TD A[Solcast solar forecast] --\u003e D[Nightly optimiser pyscript] B[Agile price curve] --\u003e D C[Learned load profile] --\u003e D H[Battery SoC and health limits] --\u003e D D --\u003e E[Schedule entities per slot] E --\u003e F[HA automation reads slot] F --\u003e G[Inverter charge or discharge or hold] M[Human override switch] --\u003e F G --\u003e N[Observability dashboard] N --\u003e CNotice the loop at the bottom: yesterday\u0026rsquo;s real outcomes feed back into the load profile, and everything that happens is logged to a dashboard so I can see what the optimiser decided and whether reality agreed. That feedback and that observability are not decoration. They are how I came to trust the thing enough to leave it running unattended.\nThe override deserves a mention too. There is a single input_boolean that, when on, freezes the optimiser and hands control back to a safe default. If guests are staying, or I know I am about to run the oven and the tumble dryer and charge a car all at once, I flip the switch and the system stops being clever. A system you cannot easily switch off is a system you will eventually rip out.\nHere is the automation that drives a single slot — deliberately dumb, because all the thinking already happened in the optimiser:\nautomation: - alias: \u0026#34;Battery - apply optimised slot\u0026#34; trigger: - platform: time_pattern minutes: \u0026#34;/30\u0026#34; condition: - condition: state entity_id: input_boolean.battery_optimiser_override state: \u0026#34;off\u0026#34; action: - variables: slot: \u0026#34;{{ now().hour * 2 + (now().minute // 30) }}\u0026#34; mode: \u0026#34;{{ state_attr(\u0026#39;sensor.battery_schedule\u0026#39;, \u0026#39;slots\u0026#39;)[slot] }}\u0026#34; - choose: - conditions: \u0026#34;{{ mode == \u0026#39;charge\u0026#39; }}\u0026#34; sequence: - service: number.set_value target: { entity_id: number.inverter_charge_power } data: { value: 3000 } - conditions: \u0026#34;{{ mode == \u0026#39;discharge\u0026#39; }}\u0026#34; sequence: - service: number.set_value target: { entity_id: number.inverter_charge_power } data: { value: -3000 } default: - service: number.set_value target: { entity_id: number.inverter_charge_power } data: { value: 0 } And here is the heart of the optimiser — a stripped-down sketch of the solve. The real version uses a small linear-programming library, but the shape is exactly this: walk the slots, respect the battery\u0026rsquo;s state-of-charge limits, and at each step prefer free solar, then cheap grid, then stored energy, never letting the battery go below its health floor.\n# nightly_optimiser.py (pyscript) # Inputs are 48 half-hour slots for the next day. BATTERY_KWH = 10.0 # usable capacity CHARGE_MAX = 1.5 # kWh movable per slot (3kW for 30 min) SOC_FLOOR = 0.15 # never deep-discharge below 15% SOC_CEILING = 0.95 # protect the cells at the top end SOC_TARGET_AM = 0.30 # reserve for the morning before solar kicks in def optimise(price, export, solar, load, soc_start): soc = soc_start * BATTERY_KWH schedule = [] # Rank slots by how cheap grid import is; charge in the cheapest # ones only if tomorrow\u0026#39;s solar will not already fill the battery. expected_solar = sum(solar) cheap_slots = sorted(range(48), key=lambda i: price[i]) plan = [\u0026#34;hold\u0026#34;] * 48 for i in range(48): surplus = solar[i] - load[i] # free energy this slot if surplus \u0026gt; 0: plan[i] = \u0026#34;charge\u0026#34; # soak up solar first, always elif price[i] \u0026gt; export * 3 and soc \u0026gt; SOC_FLOOR * BATTERY_KWH: plan[i] = \u0026#34;discharge\u0026#34; # beat the peak with stored energy # Only pre-charge from the grid if solar won\u0026#39;t get us to target. if expected_solar \u0026lt; BATTERY_KWH * (SOC_CEILING - SOC_TARGET_AM): for i in cheap_slots: if soc \u0026gt;= SOC_TARGET_AM * BATTERY_KWH: break if plan[i] == \u0026#34;hold\u0026#34;: plan[i] = \u0026#34;charge\u0026#34; soc += CHARGE_MAX return _enforce_soc_limits(plan, solar, load, soc_start) The _enforce_soc_limits pass is the safety net and it is non-negotiable: it walks the plan forward, tracks the running state of charge, and downgrades any instruction that would push the battery below the floor or above the ceiling to a hold. The optimiser is allowed to be wrong about price and weather. It is never allowed to deep-discharge the battery. Safety constraints are not part of the objective function — they are hard limits that sit outside it, because a constraint you can trade away for a few pence is not a safety constraint at all.\nWhat I got wrong, and what the data taught me The first and biggest lesson: forecasts are wrong, so build for the error, not the forecast. My early version trusted the solar forecast as if it were a measurement. It would decline to pre-charge overnight because Solcast promised a brilliant day, and then a band of cloud would roll in and I would spend the morning importing at full price with an empty battery. The fix was not a better forecast. The fix was humility: I now keep a morning reserve (SOC_TARGET_AM above) so that even if tomorrow\u0026rsquo;s free energy fails to show up, I have a buffer to ride through the morning without paying peak rates. I treat the forecast as the central guess of a distribution, and I leave headroom for the times it is wrong. That single change did more for real-world savings than any tuning of the optimiser.\nThe second lesson: modelled savings and real savings are different numbers, and only one of them pays your bill. My optimiser will happily tell me it saved me a tidy sum versus a flat schedule. But that figure is computed against the same forecasts the optimiser used to make its decisions, so of course it looks good — it is marking its own homework. The number I actually trust is measured: total metered grid import cost this month versus the same month last year on the old schedule, normalised for the weather. It is a smaller, messier, more honest number, and watching it is the only thing that told me the project was genuinely worth it.\nThe edge cases are where the design earned its keep. Cloudy days are handled by the morning reserve. Export pricing turned out to matter more than I expected: on days with a high export rate, there are slots where it is genuinely better to discharge the battery to the grid for profit than to hold it, which is why the discharge condition compares import price against a multiple of the export rate rather than treating export as worthless. And battery wear is a quiet, long-term cost that no single day\u0026rsquo;s optimisation can see — every full cycle ages the cells, so I cap how aggressively the system cycles the battery for marginal pennies. Chasing a 4p arbitrage that costs me a measurable slice of battery lifespan is a bad trade dressed up as a clever one.\nUnderpinning all of it is observability, and I cannot overstate how much this mattered. I built a Grafana panel — fed from the same Prometheus stack I use for the rest of the lab — that overlays four lines for each day: the price curve, the forecast solar, the actual solar, and what the battery actually did. The first week I had that dashboard up, I caught two bugs in an afternoon that I would never have found from the bill alone. You cannot trust an automated system you cannot see inside. Building the dashboard before fully trusting the optimiser is the same instinct I apply to everything I automate, and it is the difference between a tool and a liability. If I cannot watch it think, I do not let it touch the inverter.\nWhere this goes next There are three concrete things on the roadmap, in roughly the order I will tackle them.\nThe first is tighter ML on the load profile. Right now the household model is a fairly blunt per-slot average segmented by weekday. It does not know that a cold snap means the heat pump will run hard, or that a Friday in term time looks different from a Friday in the holidays. Feeding in weather and a calendar signal and using a proper short-horizon regression would sharpen the load forecast, and the load forecast is currently the weakest of my three inputs. This is the one place I would let a bit more ML in — but only because I can measure whether it actually beats the simple average, and I will rip it out if it does not.\nThe second is EV charging coordination. At the moment the car charger and the battery optimiser are two systems that do not talk, which means they occasionally fight over the same cheap slots. Folding the car\u0026rsquo;s charging demand into the same optimisation — it is just another load with a deadline and a flexible schedule — would let me fill the car and the house from the same cheapest half-hours instead of competing for them. This is the upgrade I am most excited about, because the car is by far the biggest movable load in the house.\nThe third is dynamic export. Some tariffs now pay a variable, half-hourly export rate, and the moment that is true, the discharge decision becomes a proper arbitrage problem: buy low overnight, hold through the cheap-export daytime, sell into the export peak. My current logic treats export as a flat fallback. Making it forecast-driven on both sides — import and export — is the natural completion of the original idea.\nI would also like to wire the daily schedule into Atlas, my local assistant, so I can ask \u0026ldquo;why did the battery not charge last night?\u0026rdquo; and get a plain-English answer pulled from the optimiser\u0026rsquo;s own logs rather than squinting at a Grafana panel. That is a small, genuinely useful place for an LLM — explaining a decision after the fact, not making it.\nClosing thought The thing I keep turning over about this project is how little of it is the part everyone wants to talk about. The \u0026ldquo;AI\u0026rdquo; headline is a per-slot average and a small solver. The real engineering is in the boring scaffolding: the morning reserve that admits the forecast will be wrong, the hard SoC floor that sits outside the objective function, the override switch, the dashboard that let me trust it. None of that is clever. All of it is necessary.\nI think that is the actual lesson of building useful automation, and it generalises well beyond batteries. The intelligence is cheap and getting cheaper. The judgement about where to put a hard limit, how much to trust a forecast, and how to make a system you can see inside and switch off — that is the work, and that is the part no model is going to do for you. My battery still does something every night. These days it is the right thing, most of the time, and I can see exactly when it is not.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/ai-battery-optimiser/","title":"AI Battery Optimiser"},{"content":"AI is Becoming Infrastructure There is a moment in every technology shift where the thing stops being interesting in its own right and starts being assumed. Nobody runs a \u0026ldquo;networking project\u0026rdquo; any more. Nobody pitches a board on the strategic value of having a database. These things became substrate — the floor you stand on, not the building you point at. You only notice them when they break.\nI think we are at exactly that moment with AI, and most organisations have not realised it yet. They are still treating AI the way you treat a feature: a project with a start date, a sponsor, a demo, and a hoped-for outcome. A team gets some budget, stands up a proof of concept against a hosted API, wires it into one workflow, and calls it an AI initiative. Then another team does the same thing, with a different model, a different key, a different prompt convention, and no idea the first team exists. Multiply that by a dozen teams and you do not have an AI strategy. You have a sprawl.\nThis is the argument I keep coming back to, and it is the spine that several other things I have written hang off. AI has crossed the line from application to infrastructure. You do not \u0026ldquo;do an AI project\u0026rdquo;. You run an AI platform, the same way you run a network or an identity service or a virtualisation estate. And the organisations that internalise that early will spend the next few years compounding, while the ones still treating it as a feature will spend the same years re-solving the same problems in a dozen incompatible ways.\nThis is the constructive companion to the autopsy I wrote in why most AI projects fail. That piece is about how the application mindset kills projects one at a time. This one is about what you build instead.\nThe thesis: AI is now a layer, not a feature Let me state it plainly, because the rest follows from it.\nAI has become a dependency layer. It belongs next to networking, identity and the database — not in the application tier on top of them.\nA feature is something one team owns and one workflow consumes. A layer is something everything depends on and nobody owns in isolation. The test is simple: if three different teams need the same capability and would each build it badly on their own, that capability has become infrastructure whether you have named it or not. Authentication crossed that line years ago — we stopped letting every app invent its own login and we built an identity platform. Storage crossed it. Messaging crossed it. AI is crossing it right now.\nThe signal is everywhere once you look. The marketing team wants summarisation. Support wants classification and drafting. The data team wants extraction. Engineering wants code assistance. Finance wants document parsing. These are not five AI projects. They are five consumers of one capability — access to language models, governed, observed and paid for centrally. When you see the same need arriving from five directions, you are not looking at a use case. You are looking at a utility.\nAnd utilities have a particular shape. You meter them. You secure access to them. You plan capacity for them. You version them. You do not let every tenant build their own power station in the car park.\nWhat treating AI as infrastructure actually means This is where it gets concrete, because \u0026ldquo;platform\u0026rdquo; is one of those words that means everything and therefore nothing. So here is what an AI platform is, in components, stripped of the brochure language.\nIt starts with a platform team — a small group whose product is not an AI feature but the ability of every other team to safely ship AI features. They own the gateway, the policies, the golden paths and the bill. This is the single most important and most resisted idea, because it requires someone to stop thinking of AI as their project and start thinking of it as everyone\u0026rsquo;s substrate.\nThe technical heart of it is a model gateway — a shared access point that every application talks to instead of talking to model providers directly. This is the equivalent of putting a load balancer or an API gateway in front of a fleet of services, and it is the highest-leverage thing you can build. Route all model calls through one place and you suddenly get, for free, the things you cannot retrofit later: a single point for authentication, for rate limiting, for cost attribution, for logging, for content filtering, for swapping one model for another without touching a single application. I run a small version of this pattern at home through n8n as the spine, and the principle scales: applications should depend on an abstraction, never on a specific model endpoint.\nAround the gateway you need identity and access control over models and data. Not every team should reach every model. Not every model should reach every data source. The same Conditional Access thinking I apply to a Microsoft tenant in the 365 health check applies here — who can call what, with which data, under what conditions. The blast radius of a leaked model key is not \u0026ldquo;a few wasted tokens\u0026rdquo;. It is potentially your entire prompt history and whatever the model was allowed to retrieve.\nThen observability and cost control, which in AI are the same problem wearing two hats. You cannot manage what you cannot see, and AI spend is uniquely good at hiding. Tokens are cheap individually and ruinous in aggregate, and a single badly written retrieval loop can quintuple a bill overnight. You want per-team, per-application, per-model spend visible on a dashboard, with alerts, the same way I watch everything else through Prometheus and Grafana. Latency, error rates, token throughput, cost per request — these are platform SLOs now, not curiosities.\nYou provide golden paths — paved, opinionated, well-documented ways for a team to go from \u0026ldquo;I want to add summarisation\u0026rdquo; to \u0026ldquo;it is in production and governed\u0026rdquo; without inventing anything. A template repository, a client library that already points at the gateway, a sanctioned retrieval pattern, an evaluation harness. Golden paths are how a platform team scales without becoming a bottleneck: you make the right way the easy way, and most teams take it gratefully.\nYou manage lifecycle and versioning of models and prompts. Models deprecate. Providers retire endpoints with weeks of notice. Prompts are code — they have versions, they regress, they need testing. Treating a prompt as a configuration string pasted into an application is the AI equivalent of hardcoding a connection string, and it ages just as badly.\nAnd you do capacity planning, especially the moment any local inference enters the picture. A GPU is a finite, schedulable resource. I sized mine — a single RTX 3090 with 24GB of VRAM — deliberately, for VRAM-per-pound rather than raw speed, because the constraint that actually bites is memory, not throughput. At enterprise scale the question becomes how many concurrent requests a model server holds, how you queue, where you burst to a hosted provider, and what it costs when you do.\nHere is the layered picture I have in my head when I say \u0026ldquo;AI platform\u0026rdquo;.\nflowchart TD Teams[Product teams and apps] --\u003e GW[Model gateway] GW --\u003e IAM[Identity and access control] GW --\u003e OBS[Observability and cost control] GW --\u003e ROUTER[Model router] ROUTER --\u003e HOSTED[Hosted model APIs] ROUTER --\u003e LOCAL[Local inference on GPU] GW --\u003e DATA[Governed data and retrieval] IAM --\u003e POLICY[Policy and audit] OBS --\u003e POLICY PLAT[Platform team] --\u003e GW PLAT --\u003e PATHS[Golden paths and templates] PATHS --\u003e TeamsNotice what the applications at the top depend on. Not a model. A platform. The model is an implementation detail behind the router, where it belongs.\nWe have been here before If this feels familiar, it should, because the industry has run this exact play three times in fifteen years and the shape never changes. Each time, a capability arrived as a novelty, got treated as a per-project curiosity, and then quietly became the substrate that everything sits on.\nVirtualisation started as a clever trick for consolidating a few servers. For a while you \u0026ldquo;did a virtualisation project\u0026rdquo;. Then it became the floor — you stopped asking whether a workload was virtual because virtual was simply how servers worked. Cloud arrived as shadow IT on a corporate credit card, one team at a time, until the sprawl forced the creation of landing zones, cloud platform teams, guardrails and FinOps — in other words, until cloud stopped being a project and became a governed platform. Containers were a developer toy, then a deployment detail, then Kubernetes became the substrate and platform engineering became a discipline whose entire job is to give product teams paved roads onto shared infrastructure.\nEvery one of those followed the same curve: novelty, sprawl, governance, substrate. AI is somewhere around the sprawl-to-governance inflection right now, and the lesson from the previous three is unambiguous. The organisations that got ahead of the curve built the platform before the sprawl became unmanageable. The ones that waited spent a painful year clawing back control from a hundred unsanctioned deployments they could no longer even enumerate.\ngraph LR N[Novelty] --\u003e S[Sprawl] S --\u003e G[Governance] G --\u003e Sub[Substrate]The cloud taught us this most expensively. The \u0026ldquo;shadow cloud\u0026rdquo; of 2014 — untracked accounts, no tagging, no cost control, no security baseline — is being re-enacted right now as shadow AI, and for exactly the same reasons. We are simply substituting model endpoints for EC2 instances.\nWhy the application mindset fails I have watched the application mindset play out and it fails in a consistent, predictable set of ways. These are the same failure modes I catalogued in why most AI projects fail, but seen through the platform lens they stop looking like a dozen separate problems and resolve into one: the absence of a platform.\nWhen every team rolls its own, you get duplicated effort — five teams independently solving retrieval, five subtly different and all slightly wrong. You get duplicated spend, because nobody is negotiating a single committed-use rate and nobody can even see the aggregate bill until finance asks why the cloud invoice grew by a third. You get no governance — no consistent answer to \u0026ldquo;what data is this model allowed to see\u0026rdquo;, because the answer is whatever each team happened to decide. You get shadow AI, the most dangerous of all: employees pasting sensitive material into whatever consumer tool is open in a browser tab, because the sanctioned path is harder than the unsanctioned one. And you get no audit — when a regulator or a customer asks \u0026ldquo;what did your AI tell people and on what basis\u0026rdquo;, the honest answer is a shrug across a dozen systems that log nothing in common.\nNone of these are model problems. Every one of them is an organisational problem that a platform solves structurally rather than heroically. You do not fix shadow AI with a policy memo any more than you fixed shadow cloud with one. You fix it by making the governed path the path of least resistance.\nHere is the contrast I keep in front of me, because it clarifies almost every decision.\nApplication thinking Platform thinking \u0026ldquo;We are doing an AI project\u0026rdquo; \u0026ldquo;We run an AI platform\u0026rdquo; Each team holds its own model keys Access brokered through a shared gateway Model chosen and hardcoded per app Model selected behind a router, swappable Cost discovered on the monthly invoice Cost metered per team in real time Prompts pasted into application code Prompts versioned, tested, owned Governance is a policy document Governance is enforced at the gateway Success is a working demo Success is teams shipping safely on paved roads Security reviewed once, at the end Identity and audit built into the substrate If your AI work lives entirely in the left column, you do not have an AI capability that will survive contact with scale. You have a collection of demos with a shared invoice.\nBuild versus buy at the platform layer Treating AI as infrastructure does not commit you to running your own models, and this is where people get the decision backwards. The build-versus-buy question is not \u0026ldquo;hosted API or local GPU\u0026rdquo; answered once, globally. It is answered per workload, behind the router, and the router is the entire point.\nThink of hosted APIs as a utility — like grid electricity. You pay per unit, you get effectively infinite capacity on demand, you get frontier capability you could never reproduce, and you accept that your data leaves your boundary and your unit cost is set by someone else. For most workloads, most of the time, that is exactly the right trade. You would not generate your own electricity to boil a kettle.\nThink of local inference as the generator and the battery — the thing you run for control, privacy, predictable cost at volume, and independence from a provider\u0026rsquo;s roadmap and rate limits. I run local models for precisely these reasons: data that must not leave the building, and high-volume batch work where per-token pricing would be punishing. The model is not the product — it never was — so I pick the model for the job and the hosting for the constraint.\nThe platform\u0026rsquo;s job is to make that a routing decision, not an architecture decision. An application asks for a capability. The router decides — by policy, by data sensitivity, by cost, by latency — whether that request is served by a frontier hosted model, a cheaper hosted model, or a local one on the GPU. Change the policy, change the routing, and not one line of application code moves. That is the dividend of having built the gateway: build-versus-buy stops being a fork in the road you commit to once, and becomes a dial you turn per workload, forever.\n# The router mindset: applications ask for a capability, not a model. def route(request): if request.data_classification == \u0026#34;restricted\u0026#34;: return local_model(\u0026#34;qwen2.5\u0026#34;, quant=\u0026#34;Q4_K_M\u0026#34;) # never leaves the building if request.needs_frontier_reasoning: return hosted_model(\u0026#34;frontier-tier\u0026#34;) # pay the utility for capability return hosted_model(\u0026#34;cheap-tier\u0026#34;) # the boring default The application that called route() knows nothing about which model answered, and that ignorance is the feature.\nWhat infrastructure and platform engineers should do now If you build infrastructure for a living, this shift is the best news you have had in years, because it is your discipline applied to a new substrate — not a new discipline you have to learn from scratch. Everything you already know about running a shared service is the thing that is missing from most AI efforts.\nSo start where the leverage is. Build the gateway before you need it. Even a thin proxy that every team agrees to route through is worth more than the most sophisticated single application, because it is the hook on which all future governance hangs. Get it in early, while there are three consumers and not thirty.\nGet cost visible immediately. Tag and meter from day one. The cost conversation is the one that gives a platform team its mandate: the moment finance can see per-team AI spend, the value of central control becomes self-evident and the political argument wins itself.\nTreat models and prompts as versioned artefacts in Git, tested in a pipeline, deprecated on a schedule — the same plain-text, version-controlled discipline I apply to everything I build. A prompt with no version history and no evaluation is a liability waiting to regress silently in production.\nBring identity and audit to the front. Decide who can call what, with which data, and log every call in a common format before you have a hundred consumers, not after. Retrofitting audit onto a sprawl is the most miserable project I can imagine, and shadow AI is how you end up needing to.\nAnd learn enough of the AI stack to be dangerous — tokens, context windows, quantisation, retrieval, evaluation. You do not need to train models; you need to understand the workload well enough to run a platform for it, the same way I argue every infrastructure engineer should learn Python. This is also the centre of gravity in presales and consultancy now — the customers worth helping are not asking \u0026ldquo;can we do an AI project\u0026rdquo;, they are asking \u0026ldquo;how do we run AI as a platform\u0026rdquo;, and most do not yet know that is the question.\nWhere this leaves us The framing matters more than any individual tool, which is why I keep returning to it and why so much else I write leans on it. If you take AI to be a feature, every decision that follows is shaped wrong: you optimise for the demo, you let teams sprawl, you discover the bill and the risk far too late, and you re-solve the same problems a dozen times in a dozen incompatible ways. If you take AI to be infrastructure, the decisions arrange themselves — gateway, identity, observability, golden paths, lifecycle, capacity — because they are the same decisions you have always made for any shared service, applied to a new and unusually expensive one.\nWe have run this play before. Virtualisation, cloud, containers — each went from novelty to sprawl to governance to substrate, and each time the organisations that built the platform ahead of the sprawl were the ones still standing and compounding when the dust settled. AI is on the same curve, only faster and with a worse failure mode, because it fails confidently and it fails in prose.\nSo stop doing AI projects. Start running an AI platform. The model is not the product, and it never was. The platform is.\n","date":"2026-06-25T00:00:00Z","permalink":"/ai/ai-is-becoming-infrastructure/","title":"AI is Becoming Infrastructure"},{"content":"Building a Second Brain I have solved the same problem more than once. Not a similar problem — the same one. The same broken Citrix StoreFront error, the same Caddyfile tweak that finally got HTTPS working on an internal host, the same n8n quirk where a webhook node silently swallows a malformed payload. I worked it out, felt clever, moved on, and six months later sat staring at the identical symptom with no idea what I had done. Past-me had the answer. Past-me did not write it down.\nThat is the failure this project exists to fix. Not a productivity hack, not a journalling fad — a deliberate piece of personal infrastructure that captures what I learn and makes it retrievable, including by a machine. The notes are plain-text Markdown in a Git repo. The retrieval and reasoning layer is Atlas, my local AI assistant. Together they turn a static pile of notes into something that can answer questions, summarise, and occasionally surprise me by connecting two things I had forgotten were related.\nI want to be honest up front about the thing most \u0026ldquo;second brain\u0026rdquo; content skips. The system is worthless without the habit. The architecture is the easy part. I will get to the discipline, because it is the part that actually decides whether any of this works.\nThe problem with where knowledge lives An engineer\u0026rsquo;s knowledge is scattered. Mine was spread across a notes app on my phone, a different notes app on my laptop, forty browser tabs I was afraid to close, screenshots of error messages in a folder called Screenshots, Slack DMs to myself, a few Google Docs, and — mostly — my own memory, which is a lossy cache with no backup. When I needed something, finding it was a search across five tools, none of which talked to each other, and a lot of the time the answer was only in my head and had already evaporated.\nThe deeper problem is that this scattering makes you solve things twice. The cost of writing something down is paid once, immediately, when you are tired and just want the problem to be over. The cost of not writing it down is paid repeatedly, later, with interest, every time you rediscover the same fix from scratch. Past-you was lazy and present-you is paying the bill. I have done this enough times to take it personally.\nThen there is the tool-of-the-month trap. I have migrated my notes more than once because the shiny new app promised to organise my life. Evernote, then a wiki, then one of the block-based outliner tools everyone raved about. Each one locked my notes inside its own database and its own export format that never quite round-tripped. One of them changed its pricing to something absurd. Another simply shut down and gave me sixty days to get my data out. Every migration lost something — formatting, links, attachments, the will to live. I learned the hard way that the app is not the asset. The notes are the asset. If the notes only exist inside someone else\u0026rsquo;s application, you do not own your knowledge, you are renting access to it.\nIf your knowledge only opens in one company\u0026rsquo;s app, you do not own it. You are renting it.\nThe principles I settled on After the third migration I stopped chasing tools and wrote down the principles instead. These are non-negotiable now, and everything else is an implementation detail.\nOwn your data. The notes live in a Git repository I control. Not a service. A repo. I can clone it, back it up, grep it, and read every file with cat on a machine that has never heard of my note-taking app. If every tool I currently use vanished tomorrow, the knowledge would be completely intact.\nPlain text, forever. Markdown. Nothing else. Markdown is readable as raw text, renders nicely when you want it to, diffs cleanly in Git, and will still open in fifty years. Proprietary formats are a bet that a company outlives your career. Plain text is a bet that text files keep working, which is the safest bet in computing. This is the same conviction that made me move this site off Blogger and onto Hugo and Markdown in Git.\nAtomic, linked notes. One idea per note, where practical, with links between them. A note about a Caddy directive links to a note about the reverse proxy migration, which links to the homelab overview. The value is not in any single note; it is in the connections. Knowledge is a graph, not a filing cabinet, and I want the structure to reflect that.\nCapture first, curate later. The biggest killer of any note system is friction at the moment of capture. If writing something down requires choosing the right folder and the right tags and the right title, I will not do it when I am mid-incident at 11pm. So capture is dumb and instant: throw it in the daily note, no decisions required. Curation — turning that raw capture into something durable — happens later, deliberately, when I have the headspace.\nWrite for future-you. Every note is a letter to a version of me who has forgotten everything. That means context, not just the answer. Not \u0026ldquo;add the directive\u0026rdquo; but \u0026ldquo;Caddy wasn\u0026rsquo;t issuing a cert for the internal host because the ACME challenge couldn\u0026rsquo;t reach it — switch that host to the DNS challenge with a build that includes the provider, here\u0026rsquo;s the working version and why\u0026rdquo;. Future-me is a stranger. Be kind to the stranger.\nThe system that actually works The repository structure is deliberately boring. Boring survives. I tried clever taxonomies before and abandoned all of them; the only structure I have kept for more than a year is the one that maps to how I actually think.\nsecond-brain/ ├── daily/ # one file per day, the inbox: YYYY-MM-DD.md ├── projects/ # active work, one folder per project │ ├── atlas/ │ └── energy-optimiser/ ├── references/ # facts, configs, how-tos — the lookup layer │ ├── caddy.md │ └── citrix-storefront.md ├── evergreen/ # distilled, durable notes — my actual thinking └── meta/ # templates, conventions, the README Four kinds of note, each with a job. Daily notes are the inbox — every capture lands here first, timestamped, unprocessed. Project notes are working memory for whatever is live, and they are allowed to be messy because they are temporary. Reference notes are the lookup layer: the working Caddyfile, the Graph app-registration steps, the exact ollama command that pulls the right quant. Evergreen notes are the crown jewels — distilled, rewritten, linked ideas that represent what I actually understand about something, not just what I copied from a terminal.\nNaming and linking conventions matter more than they look. Files are lowercase-kebab, dated notes are YYYY-MM-DD, and links are relative Markdown links between files, [reverse proxy](../references/caddy.md). I avoid spaces in filenames so the repo plays nicely with every tool. Tags live in front matter, not scattered through the text, so they are machine-readable. That front matter is the same idea as on this site, and it is what lets a program reason about the notes:\n--- title: \u0026#34;Caddy not issuing a cert for an internal host\u0026#34; type: reference created: 2025-11-03 updated: 2026-02-18 status: evergreen tags: [caddy, docker, tls, homelab] links: - \u0026#34;[[reverse-proxy-migration]]\u0026#34; - \u0026#34;[[homelab-overview]]\u0026#34; summary: \u0026gt; The ACME challenge could not reach an internal-only host, so no cert ever issued. Fix: switch it to the DNS challenge. Reasoning below. --- That summary field is not decoration. It is the thing the AI layer leans on hardest, and writing a one-line summary forces me to know what the note is actually about.\nThere is also the question of what overlaps with the public site. I run a private brain and a public notebook, and the line between them is real. The private brain holds client specifics, credentials-adjacent detail, half-formed opinions, and anything with someone else\u0026rsquo;s name on it. The public site — this site — is the curated, sanitised, generalised version: the lessons with the names filed off. An evergreen note about why most AI projects stall in the private brain becomes, after a lot of editing, a public article like why most AI projects fail. The private brain is where I think; the public site is where I publish thinking I am willing to stand behind. Plenty crosses the line; plenty never does, and that is the point of having two.\nThe AI layer that makes it a second brain A Git repo of Markdown is a very good filing cabinet. It is not a second brain. A filing cabinet only gives back exactly what you put in, exactly where you put it, and only if you remember it exists. The thing that turns the filing cabinet into a brain is the layer that can reason over the whole of it at once — and that layer is Atlas.\nAtlas runs retrieval-augmented generation over the notes. The repo is chunked and embedded into a vector store; when I ask a question, the relevant chunks are pulled back and handed to a local model running on Ollama on the GPU box along with my question. The architecture and the tool-calling are described in full in the Atlas write-up; here I care about what it does for the notes specifically.\nThree things, mainly. It retrieves — I ask \u0026ldquo;how did I fix the StoreFront login loop?\u0026rdquo; in plain English and get the answer with the source note linked, instead of guessing which of forty files it lived in. It summarises — point it at a noisy project folder and it gives me the state of play without my re-reading three weeks of daily notes. And occasionally it surfaces forgotten connections — because retrieval works on meaning, not exact words, it pulls back a note I had completely forgotten was relevant. That is the closest this gets to feeling like a genuine second brain: it remembers things I have stopped remembering.\nThen there is the part that makes it more than a search box. Atlas can act, through n8n. A query is not just answered, it can trigger a workflow — pull the latest tenant findings, kick off a health check, file a new note from a chat, fire a webhook. The note system becomes both the memory and a control surface.\nflowchart TD A[Capture into daily note] --\u003e B[Curate into reference or project] B --\u003e C[Promote to evergreen and link] C --\u003e D[Embed into vector store] D --\u003e E[Atlas RAG retrieval] E --\u003e F[Summarise and surface connections] F --\u003e G[Act via n8n workflows] G --\u003e A E --\u003e AThe loop is the whole idea. Capture, curate, connect, retrieve, act — and acting or querying often generates the next capture, so it feeds itself.\nI have to be honest about the limits, because the hype around this stuff is dishonest. RAG is only as good as the notes. If a note is wrong, the AI confidently tells me the wrong thing, now with a citation that makes it look authoritative. If a note never got written, retrieval returns nothing useful and the model fills the gap with a plausible guess. And critically, AI does not fix bad capture habits. It cannot retrieve what you never captured. A second brain built on an empty first brain is just an expensive way to generate confident fiction. The AI is a multiplier on the quality of the underlying notes, and a multiplier works in both directions.\nThe discipline, which is the actual product Here is the part I would bolt to the front door. The system is worthless without the habit. I have built three versions of this with progressively nicer tooling, and the version that worked was not the one with the best architecture. It was the one I actually used every day.\nThe discipline has three loops, on three timescales.\nDaily capture. Every working day, everything interesting goes into that day\u0026rsquo;s daily note. No filing, no decisions, no friction — just dump it. An error and its fix, a thing I learned, a half-idea, a link worth keeping. The bar is deliberately on the floor, because a note captured badly is infinitely more valuable than a note captured perfectly tomorrow, which is to say never.\nWeekly review. Once a week — Friday afternoon, usually, when I am too fried for real work but fine for tidying — I go through the week\u0026rsquo;s daily notes and process them. Most lines get deleted; they were noise. Some get promoted into a reference note. A few project notes get updated. This is where dumb capture becomes structured knowledge, and skipping it is how the inbox becomes a landfill nobody can use.\nPromoting to evergreen. The rarest and most valuable move. Occasionally a reference note has been touched enough times, or an idea has recurred enough, that it deserves to become evergreen: rewritten properly, linked into the graph, made durable. This is the same instinct as the difference between building knowledge and building documents — a document is written once and rots; an evergreen note is alive and gets better each time you return to it.\nMiss the daily capture and the well runs dry. Miss the weekly review and the inbox rots. Miss the promotion and you have a swamp of raw notes with no distilled understanding on top. I have failed at all three at various points, and every time the system\u0026rsquo;s value collapsed within a fortnight. The architecture does not save you. The habit is the product. Everything else is plumbing.\nWhat I got wrong I over-engineered the early versions. The first attempt had a fourteen-category taxonomy I designed before I had written a single note, and I spent more time deciding where things went than writing them down. I abandoned it. The structure should grow from the notes, not the other way round.\nI also tagged everything obsessively and linked almost nothing, which is exactly backwards. Tags are weak; they group things that happen to share a word. Links are strong; they encode an actual relationship I decided was real. The graph lives in the links. I now spend my curation effort on linking and barely touch tags.\nAnd I reached for the AI layer too early. I built embeddings and retrieval over a corpus of about thirty thin notes and wondered why the answers were rubbish. They were rubbish because the corpus was rubbish. RAG over a small, shallow set of notes is worse than just reading them. The AI only started earning its place once there were enough good notes for retrieval to have something real to find.\nWhere this goes next The honest near-term roadmap is mostly about closing the loop between capture and the AI. Right now I curate by hand; I want Atlas to do a first pass on the weekly review — proposing which daily-note lines look like reference material and drafting the promotion — leaving me to approve rather than to do. That is squarely on the list of what I\u0026rsquo;m building now.\nI want capture from more places. A Telegram bot that drops a line straight into today\u0026rsquo;s daily note. A clipper that saves a page as Markdown with its source. Voice capture transcribed locally, so a thought on a walk becomes a note without me touching a screen. The principle holds — lower the friction of capture and the well stays full.\nAnd I want Atlas to get more proactive about surfacing connections without being asked — a weekly digest of notes it thinks are related but I have not linked, turning the forgotten-connection trick from something I have to prompt into something that arrives on its own. Carefully, because a system that nags is a system you start ignoring.\nThe thing I keep relearning The point of a second brain is not to remember everything. It is to make it cheap to remember the right things, and trivial to find them again, so the expensive part of my mind is free to think rather than to store. I do not need to hold the exact Caddy directive in my head if I trust that ten seconds of asking Atlas will return it with its reasoning intact.\nBut the trust is earned by the boring daily work, not by the architecture. I built a clean repo, a sensible structure, and a genuinely useful AI layer on top, and none of it mattered until I started actually writing things down, every day, even when I did not feel like it. The second brain is real. It just turns out the first brain still has to show up and do the writing.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/building-a-second-brain/","title":"Building a Second Brain"},{"content":"Building an AI Consultancy Toolkit Most of what passes for \u0026ldquo;AI consulting\u0026rdquo; right now is one of two things. It is either a hype-led demo — someone wires a chatbot to a sample PDF, the room makes impressed noises, and nobody asks what happens on day ninety. Or it is a strategy deck: forty slides of maturity curves and \u0026ldquo;transformation pillars\u0026rdquo; written by people who have never deployed a model, never watched a retrieval pipeline return confidently wrong answers, and never had to explain to a CFO why the proof of concept that wowed everyone cannot go live.\nI sit in the middle of that, in technical presales and solutions architecture, and I have come to believe the gap between those two failure modes is where the actual job lives. The demo people can build but cannot scope. The strategy people can scope but cannot build. The work that matters — and the work customers will pay for and trust — is qualifying hard, discovering honestly, and turning that into something defensible an engineering team can actually deliver.\nThe single most useful thing in my toolkit is not a framework. It is the willingness to say no. Most requests that arrive with the words \u0026ldquo;can you add AI to this?\u0026rdquo; should not become projects, and a consultant who cannot disqualify is just an expensive order-taker. Everything below is built around that discipline. The frameworks exist to make a defensible \u0026ldquo;no\u0026rdquo; — or a defensible \u0026ldquo;yes\u0026rdquo; — repeatable instead of a matter of mood.\nI write this as someone who actually builds the stuff. I run local models at home on a single RTX 3090, an Ollama-based assistant I call Project Atlas, and a stack of retrieval and automation that I have broken and rebuilt enough times to know where the bodies are. That hands-on credibility is not a vanity point. It is the thing that lets me sit across from a customer and tell the difference between a five-day integration and a six-month research project, because I have personally been burned by mistaking one for the other.\nThe problem with how this gets sold The typical AI enquiry arrives backwards. A customer has read that competitors are \u0026ldquo;using AI\u0026rdquo;, a board member has asked an awkward question, and someone has been told to \u0026ldquo;look into it\u0026rdquo;. So the request lands as a solution looking for a problem: we want a copilot, we want to chat with our documents, can the system summarise tickets. There is no use case underneath it, only a capability they have heard exists.\nWhat is almost always missing at this point is the unglamorous half of the conversation. Nobody has decided what decision the AI would actually change. Nobody knows whether the data it would need is clean, accessible, or even allowed to be used that way. There is no definition of \u0026ldquo;good\u0026rdquo; — no number, no threshold, no acceptance criteria — so the project can never be declared finished or successful. And crucially there is rarely a sponsor with the authority to make the trade-off decisions the work will demand.\nI have written separately about why most AI projects fail, and the post-mortems nearly always trace back to this opening moment. The failure is designed in before a line of code is written, because the enquiry was treated as a brief instead of a symptom. The job of good consulting is to refuse the brief and diagnose the symptom.\nIf you cannot name the decision the model will change, you do not have a use case. You have a wish.\nQualification — telling an opportunity from a science project Qualification is where the money is saved or lost, and it happens before discovery proper. I am trying to answer one question: is there a real opportunity here, or a science project dressed as one? A science project is interesting, open-ended, and funded by enthusiasm; it has no owner and no end state. A real opportunity has tension behind it — someone is genuinely losing time, money, or sleep — and someone with authority wants it fixed.\nFive things have to be present, and I score them deliberately. Value: is there a quantifiable outcome, or at least a defensible estimate of one? \u0026ldquo;Save advisors ten minutes per case across two hundred cases a day\u0026rdquo; is value. \u0026ldquo;Be more innovative\u0026rdquo; is not. Data availability: does the data the use case depends on actually exist, in a place we can reach, in a state we can trust? Ownership: is there a named business owner who will live with the result, not just an IT contact relaying messages? Risk appetite: is this a domain where a wrong answer is recoverable, or one where a single hallucination is a regulatory incident? Sponsor: is there someone who will make decisions and defend a budget when the novelty wears off?\nHere is the scoring model I actually use. It is deliberately blunt.\nDimension 0 — disqualify 1 — caution 2 — strong Value No measurable outcome Plausible but unquantified Named metric and baseline Data readiness Doesn\u0026rsquo;t exist or off-limits Exists but messy or siloed Accessible, governed, trustworthy Ownership IT-only, no business owner Business owner, low engagement Engaged owner who feels the pain Risk fit Zero tolerance for error Errors need human review Errors are cheap and recoverable Sponsor Curiosity, no budget Budget but no decision power Funded sponsor who decides I total the five. Anything scoring a zero on a single dimension is a hard stop regardless of the total — a brilliant use case with no usable data is not a project, it is a data project pretending to be an AI one. Eight to ten and I will write a proposal with confidence. Five to seven and I will propose a paid discovery to resolve the unknowns, never a build. Below five I say no, and I say it plainly, because letting a doomed engagement start is the most expensive kindness a consultant can offer.\nSaying no well is a skill. I do not say \u0026ldquo;this is a bad idea\u0026rdquo;. I say \u0026ldquo;here is what would have to be true for this to work, and here is which of those things is missing today\u0026rdquo;. That reframes the no as a map. Half the time the customer comes back six months later with the missing piece solved, and now they trust me because I did not take their money the first time.\nflowchart TD A[AI enquiry arrives] --\u003e B{Real decision identified} B -- No --\u003e X[Disqualify or reframe] B -- Yes --\u003e C{Data exists and usable} C -- No --\u003e D[Data readiness engagement first] C -- Yes --\u003e E{Owner and sponsor present} E -- No --\u003e X E -- Yes --\u003e F{Risk tolerance fits errors} F -- No --\u003e X F -- Yes --\u003e G[Paid discovery workshop] G --\u003e H[Use case scoring and architecture] H --\u003e I{Score eight or above} I -- No --\u003e J[Smaller pilot or stop] I -- Yes --\u003e K[Defensible proposal]The discovery workshop Once an opportunity qualifies, discovery is where I earn the right to propose. I run it as a structured workshop, not a casual chat, because the structure is what surfaces the things nobody volunteers. I want the business owner in the room, a couple of the people who actually do the work, and someone who knows where the data lives. Half a day, sometimes a full day. It is paid, and saying so up front is itself a qualifier — people who will not fund discovery were never going to fund delivery.\nI move through five things in order. First, process mapping: walk the real workflow end to end, the version that happens on a Tuesday when someone is off sick, not the tidy diagram in the quality manual. Second, where decisions are made: every step where a human judges, chooses, or interprets is a candidate for AI assistance and, equally, a candidate for AI to get dangerously wrong. Third, what data exists and whether it can be trusted: not just \u0026ldquo;do you have the documents\u0026rdquo; but are they current, are they contradictory, who updates them, and what happens when they are wrong. Fourth, what good looks like: I push hard for a number and an acceptance threshold, because a use case without a definition of success is a use case that never ends. Fifth, build versus buy: whether this capability is differentiating enough to build or whether a product already does it for less than our day rate.\nThe most valuable output of discovery is often a smaller, sharper problem than the one we walked in with. \u0026ldquo;Chat with all our documents\u0026rdquo; becomes \u0026ldquo;answer the twelve questions the support desk asks most, from these four documents, with a citation\u0026rdquo;. That narrowing is the work. It is also where I find the landmines — the data that turns out to live in someone\u0026rsquo;s personal mailbox, the decision that is actually governed by a regulation nobody mentioned, the \u0026ldquo;simple\u0026rdquo; classification that requires judgement no current model can reliably make.\nThe data readiness assessment I treat data readiness as a gate, not a footnote, because it is the most common reason a confident proposal turns into a quiet disaster. The questions are unglamorous and that is the point.\ndata_readiness: exists: true # does the data physically exist accessible: true # can we reach it via api or export governed: false # is there a clear owner and lawful basis current: true # is it kept up to date, or stale structured: partial # structured, semi, or free text trustworthy: unknown # is it correct, or full of contradictions volume: sufficient # enough to be useful, not so much it is noise sensitivity: high # pii, commercial, regulated Any false or unknown on governed, trustworthy, or accessible stops the build conversation and starts a data conversation instead. I have learned the hard way that a retrieval system over untrustworthy source data does not fail loudly. It fails politely, with fluent, well-formatted, completely wrong answers, and by the time anyone notices, trust in the whole project is gone. This is the same instinct I bring to repeatable customer health checks — measure the ground truth before you promise anything built on top of it.\nBuild, buy, or integrate The next decision is whether we build anything at all. My default bias is do not build what you can buy, because bespoke AI carries a maintenance tail most customers underestimate. I frame it as three options. Buy when a mature product already does the job and the use case is not a competitive differentiator — most \u0026ldquo;summarise my tickets\u0026rdquo; and \u0026ldquo;draft my emails\u0026rdquo; requests are already features in tools the customer owns. Integrate when the value is in connecting existing capabilities to the customer\u0026rsquo;s specific data and workflow, which is the sweet spot for most engagements. Build only when the use case is genuinely differentiating, the data is proprietary, and no product fits — the rarest case, and the one people reach for first.\nThe honest version of this conversation often loses me the bigger project and wins me a customer for life. Telling someone their idea is already a checkbox in their Microsoft 365 licence is not a lost sale. It is the reason they call you for the hard thing next year.\nThe architecture pattern catalogue When we do build or integrate, I reach for a small catalogue of patterns and resist the urge to over-engineer. The mistake I see most often is reaching for an autonomous agent when an API call would do, because agents demo well and feel modern. They are also the hardest thing to make reliable.\nPattern Use when Cost and risk Avoid when Just an API The model needs no private context, task is one shot Lowest, mostly prompt design You need grounding in private data RAG Answers must be grounded in a known document set Moderate, retrieval quality is the work The knowledge is procedural not factual Fine-tune You need a consistent style or format at scale High, needs data and re-training Facts change often, RAG is cheaper Agent or tool-calling The task spans multiple steps and systems Highest, reliability is hard A deterministic workflow would do Most of the value I deliver is plain retrieval-augmented generation over a curated, governed document set, with citations, and a human in the loop. It is unfashionable and it works. I reach for fine-tuning rarely, because the moment the underlying facts change, a fine-tuned model is confidently out of date and RAG is not. I reach for agents only when the workflow genuinely needs to plan and act across systems, and even then I bound them tightly with deterministic tools rather than letting them improvise. I build mine on Ollama and n8n precisely so I can keep the clever bits small and the plumbing inspectable. This is also why I keep arguing that AI is becoming infrastructure: the durable engineering is in retrieval, data pipelines, and orchestration, not in the model itself, which is increasingly a commodity you swap out.\nRisk, governance, and the boring checklist that saves you Before anything goes in a proposal I run a governance pass, because the questions that sink AI projects in regulated organisations are never technical. Where does the data go, and does that cross a boundary the customer\u0026rsquo;s legal team would object to? What is the lawful basis for using this data this way? How do we handle a wrong answer — is there a human review step, an audit trail, a way to trace why the system said what it said? Who is accountable when it fails, and have they agreed to be? What is the fallback when the model or the provider is unavailable?\nI keep this as a literal checklist and I do not skip it because the customer is enthusiastic. Enthusiasm is exactly when the boring questions get waved away, and exactly when they matter most. A consultant who raises governance early looks slower than the demo merchant in the next meeting. They also look a great deal smarter eighteen months later.\nFrom discovery to a defensible proposal Everything above feeds one output: a proposal that survives scrutiny. By defensible I mean every claim in it traces back to something we established in discovery — the value to the scored metric, the architecture to the data assessment, the timeline to the genuine unknowns rather than to optimism. It states what we are not doing as clearly as what we are, because uncontrolled scope is the quiet killer. And it sets expectations honestly: where the risks are, what could push the timeline, and what \u0026ldquo;good enough to go live\u0026rdquo; actually means.\nThe handover from a signed proposal to a delivery team is its own discipline, and I have written about that journey from proposal to production — the short version is that a proposal which hides the hard parts to win the deal simply moves the failure to the delivery team and burns the relationship anyway. A defensible proposal is one I would be happy to be held to in twelve months, which is a much higher bar than one that wins the meeting.\nWhat I have got wrong I have made every mistake in this article at least once. I have let a project start that I knew in my gut was a science project, because the customer was keen and the number was big, and it ended exactly as my gut predicted — a beautiful proof of concept that could never survive contact with real data. I have under-weighted governance because the use case was exciting, and watched it stall in legal for three months. I have proposed an agent where a scheduled script would have been more reliable and a tenth of the cost, because the agent was more fun to build.\nThe pattern is the same each time: I let enthusiasm override the discipline of the frameworks. The frameworks are not there for the easy decisions. They are there to hold the line when everyone in the room, including me, wants to skip to the fun part.\nWhere this goes next The toolkit keeps evolving. I am turning the qualification scorecard and the data readiness assessment into actual artefacts — structured templates that feed a workflow rather than living in my head and a slide. There is a natural project in connecting them to the same retrieval-and-automation stack I use for everything else, so that discovery notes become searchable, comparable knowledge across engagements rather than documents that die in a folder. That is the same instinct that drives my whole approach to building knowledge instead of documents, and it is where I think presales is heading anyway, as I argued in the future of technical presales: less polished pitching, more genuine engineering judgement applied early and honestly.\nThe thing I will not automate is the no. Disqualifying a bad opportunity depends on reading a room, hearing what a sponsor does not say, and having the credibility to be believed when you tell someone their favourite idea will not work. That credibility comes from having actually built the thing. It is the reason I keep running models on my own hardware, breaking my own pipelines, and writing it all down — not because the homelab is the job, but because it is the thing that lets me sit across a table and tell the difference between what is possible and what merely demos well.\nThe best AI consulting I do looks, from the outside, like talking people out of things. That is not a failure of ambition. It is the whole value.\n","date":"2026-06-25T00:00:00Z","permalink":"/ai/building-an-ai-consultancy-toolkit/","title":"Building an AI Consultancy Toolkit"},{"content":"Building an AI Infrastructure Lab at Home I wanted to understand AI properly.\nNot from a slide deck. Not from vendor messaging. Not from someone else\u0026rsquo;s curated demo environment where everything works because someone removed all the parts that don\u0026rsquo;t.\nI wanted to build something, break it, rebuild it, and understand what was actually happening underneath. So I built a small AI lab at home, in Docker, on hardware I own, and I have been living with it ever since. This article is the origin story of almost everything else in this notebook. Most of the other projects here grew out of this one room of equipment and a fairly stubborn refusal to accept \u0026ldquo;it just works\u0026rdquo; as an explanation.\nThe thesis is simple, and I will keep coming back to it: AI is a workload, not a feature you switch on. It needs infrastructure, data, access control, governance, monitoring, storage and a sensible operating model — the same as any other platform that matters. Once you accept that, the whole conversation changes, and most of the marketing falls away.\nWhy I bothered The day job is technical presales and solutions architecture. I spend my time helping organisations work out what to build and how to run it. And for the last couple of years, nearly every one of those conversations has eventually arrived at AI. Usually in the same shape: someone has been told AI will transform the business, a budget has appeared, and now somebody needs to make it real.\nThe problem is that most people talking about enterprise AI have never actually run any. They have run a chat window. That is not the same thing. Running a chat window tells you nothing about where the data goes, how access is controlled, what happens when the model is confidently wrong, or what it costs to keep the lights on when fifty people use it instead of one.\nI did not want to be one of those people. If I am going to stand in front of a customer and talk about AI as part of their infrastructure, I want to have built it myself first — including the unglamorous parts. The bits the demos skip. The reverse proxy, the GPU scheduling, the moment you realise your knowledge base is a mess and the model is faithfully repeating the mess back to you.\nSo the lab is partly selfish. It is the fastest way I know to learn something real. A home lab teaches you things slideware never will, which is most of the argument I make in the home lab as a learning platform. You cannot fake having run the thing.\nThe problem: businesses treat AI as a feature, not a platform Here is what I keep seeing. A business decides it wants AI. The conversation starts with a product:\nWhat AI product should we buy? Can we turn on Copilot? Can we use ChatGPT for this process? Can we bolt a chatbot onto the website? None of those are stupid questions. But they are not the starting questions, and treating them as the starting questions is how you end up with an expensive disappointment. They all assume AI is a feature — a thing you procure, enable, and tick off. Switch it on and value falls out.\nThat is not how it works, and it is a large part of why so many AI projects fail. When you treat AI as a feature, you skip the questions that actually determine whether it works:\nWhat data do we trust, and where does it live? Who is allowed to see what, and does the AI respect that boundary? Which decisions can AI assist with, and which must stay human-led? How do we audit an outcome after the fact? What happens when the model is wrong — and it will be wrong? How do we stop this becoming another unsupported shadow IT project nobody owns? Those are infrastructure and governance questions. They are boring, and they are the entire game. A business that can answer them will get value out of a mediocre model. A business that can\u0026rsquo;t will get nothing out of the best model on the market, because the model was never the bottleneck.\nAI is not a feature you enable. It is a platform you operate.\nI built the lab to test that belief in the only way that counts: by having to operate the platform myself, with no vendor to hide behind when something breaks at eleven at night.\nDesign decisions A few principles shaped the build, and I made each of them deliberately. They are worth stating plainly, because the trade-offs are the interesting part.\nLocal-first. The default is that data and inference stay on hardware I control. This is not anti-cloud dogma. I use cloud models when they are the right tool. But starting local forces you to confront the questions that matter — where does this data live, who can reach it, what leaves the building — instead of waving them away because the API endpoint is somebody else\u0026rsquo;s problem. Local-first is a forcing function for good design. Once you have run a workload locally and understood it, choosing to put it in the cloud becomes an informed decision rather than a default.\nEverything as code. Nearly every service runs in Docker, defined in docker-compose. The compose files live in Git, and Git is the source of truth — not the running container, not a setting I clicked in a UI eighteen months ago and have since forgotten. I run Portainer for visibility, but it is a window onto the system, not the system of record. If a machine dies, I want to rebuild from text, not from memory. This discipline is the spine of my Docker homelab, and it is the single highest-leverage decision in the whole lab.\nA consumer GPU, chosen for VRAM per pound. The inference box runs a single NVIDIA RTX 3090 with 24GB of VRAM. People are sometimes surprised it isn\u0026rsquo;t something newer or faster. The reasoning is straightforward: for local LLM work, the constraint that actually bites is VRAM, not raw throughput. 24GB is the practical line where useful models run at a sensible quantisation without constant out-of-memory juggling. A used 3090 gives more usable VRAM per pound than almost anything else, and a home lab does not need to serve a thousand concurrent users — it needs to teach. I sized it the same way I describe in designing infrastructure for AI workloads: start from the model you want to run, work back to the memory it needs, and buy that. The datacentre cards are wonderful and I am not paying datacentre prices to learn.\nBare metal underneath. The main server runs Ubuntu straight on the metal — AMD Ryzen, plenty of RAM — with the always-on services in Docker on top. I tried virtualising the base first, for the snapshots, but for a single-operator lab the hypervisor cost more in fragility than it returned, so disposability comes from elsewhere: the compose files live in Git and a broken host rebuilds from text, not from a VM snapshot. A small fleet of N100-class mini PCs handles the lightweight always-on bits, and a NAS holds bulk storage and backups, because everything important follows 3-2-1 thinking. The network is a flat home LAN being slowly carved into VLANs — trust, IoT, lab — which is one of the security boundaries I am still actively working on.\nThe thread running through all of it: build it the way I would tell a customer to build it. No special cases I would be embarrassed to defend.\nArchitecture: how it actually fits together None of the components are exotic. That is deliberate, and it is the point. The interesting part is never one tool — it is how the tools connect.\nDocker is the substrate; almost everything is a container — the one deliberate exception is the model runtime. Ollama runs natively on the bare-metal GPU box, serving local models over its API close to the hardware with nothing in the way. Open WebUI is the chat front end — the part that looks like the demo, and the least important part of the system. n8n is the automation spine: the thing that turns a chat window into a system by giving the model tools, triggers and a way to reach into other services. Home Assistant brings the real world in — sensors, energy, switches — which is how the lab connects to physical things rather than just text. Caddy sits in front, terminating TLS with automatic HTTPS; I moved to it from Nginx Proxy Manager once I wanted the routing defined as code in one file alongside everything else. Uptime Kuma watches up/down, and Prometheus with Grafana handles metrics, because a platform you can\u0026rsquo;t observe is a platform you don\u0026rsquo;t really run.\nHere is how the pieces relate:\nflowchart TD User[User] --\u003e Caddy[Caddy reverse proxy] Caddy --\u003e WebUI[Open WebUI] Caddy --\u003e N8N[n8n automation] Caddy --\u003e HA[Home Assistant] WebUI --\u003e Ollama[Ollama native on GPU box] Ollama --\u003e GPU[RTX 3090 24GB] Ollama --\u003e Models[Local models] N8N --\u003e Ollama N8N --\u003e Graph[Microsoft Graph] N8N --\u003e HA N8N --\u003e KB[Knowledge base in Git] HA --\u003e Sensors[Energy and sensors] subgraph Observability Kuma[Uptime Kuma] Prom[Prometheus and Grafana] end Ollama --\u003e Prom N8N --\u003e KumaThe shape that matters: the user does not talk to a model. The user talks to a system, and the model is one component inside it — reachable both directly through the chat front end and programmatically through n8n. That second path is where the real value lives, because n8n is what lets the model do something other than produce text.\nA small but representative slice. Ollama is not in here, because it runs natively on the GPU box — bound to the LAN as a host service so the containers can reach it:\n# On the bare-metal GPU box (systemd, not Docker): # OLLAMA_HOST=0.0.0.0:11434 # OLLAMA_MODELS=/mnt/nvme/models # # The container side is the front end, pointed at that box: services: open-webui: image: ghcr.io/open-webui/open-webui:main container_name: open-webui restart: unless-stopped environment: - OLLAMA_BASE_URL=http://gpu-box.lab.internal:11434 - WEBUI_AUTH=true volumes: - openwebui_data:/app/backend/data networks: - web - ai_net volumes: openwebui_data: networks: web: external: true # shared with Caddy; routing lives in the Caddyfile ai_net: driver: bridge That is the front-of-house in a readable amount of YAML. The model runtime runs natively on the GPU box, the front end talks to it over the LAN, Caddy publishes the front end from a single Caddyfile, and the container side is a git pull and a docker compose up -d away from existing on another machine. That property — reproducibility — is worth more than any individual feature.\nThis stack is also the foundation of Project Atlas, my local AI assistant. Atlas is what happens when you take this base and add a proper Git-backed knowledge layer, retrieval, and tool-calling into n8n workflows. Everything in this article is the ground Atlas stands on.\nWhat it changed about how I see enterprise AI This is the part I did not expect, and the part that has been most useful in the day job.\nBefore the lab, if you had asked me what made an AI project succeed, I would have talked about the model. Capabilities, benchmarks, context windows. Having actually run the thing for a while, I think that focus is almost entirely wrong, and I would have argued the wrong case in front of customers who deserved better.\nThe model is not the product.\nIt is the easiest thing to obsess over. Llama, Qwen, Mistral, gpt-oss, the odd DeepSeek, whatever lands next week. I run several of them, usually at Q4_K_M, sometimes higher when there is VRAM to spare, and I pick the model for the job rather than crowning a favourite — the longer version of that journey is in my work with local LLMs. But swapping models, in the end, changes surprisingly little about whether the system is any good.\nWhat changes everything is the context and the plumbing. A model on its own is just a text box. A useful AI system needs the right data in front of it, tools it can call, limits on what it can reach, memory in the right places and — just as importantly — no memory in the wrong places. Get the knowledge layer clean and a modest model performs well. Leave it a mess and the best model on earth will confidently recite your worst, most out-of-date document, because it has no way of knowing the document is wrong.\nSo the questions that actually determine enterprise success are not about the model at all. They are about data, governance and ownership:\nIs the data we are feeding it trustworthy, current and well-structured? Does the system respect who is allowed to see what? Can we explain and audit what it did? Who owns this platform when the person who built it is on holiday? Those are infrastructure questions. They are the questions I have always cared about as an infrastructure person. The lab made me realise that AI did not introduce a new discipline — it just raised the stakes on the old one. This is most of what I mean when I say AI is becoming infrastructure: the model is the part everyone looks at, and the platform around it is the part that decides whether it works.\nLessons learnt Document earlier. This is the big one, and it is the lesson I learnt the hard way. When you are learning fast you change things constantly and you do not write down why. Six weeks later you find a setting you clearly chose on purpose and you have no memory of the purpose. That is fine for a weekend hack. It is not how a platform should be run, and it is exactly the failure mode I would criticise in a customer. This site exists partly to fix that in myself — capturing why, what, what worked and what failed, which is the whole argument behind building knowledge instead of documents.\nThe model is not the product. Worth repeating because it is the lesson that took longest to truly believe. I spent early effort chasing models when I should have spent it on the knowledge layer and the workflows. The model was never the bottleneck.\nContext and tooling are the real work. Connecting the model to n8n, giving it clean data to draw on, and bounding what it can touch — that is where the value and the difficulty both live. It is unglamorous. It does not demo as well as a clever answer in a chat window. It is also the entire job.\nLocal-first surfaces the hard questions early. Running things myself forced me to confront access, storage and segmentation up front, instead of discovering them in production. That has been uncomfortable and completely worth it.\nThings I would not do again. I left the network flat for too long, which meant my IoT devices and my lab sat on the same trust level — fine until you think about it for ten seconds. I also ran for months without proper monitoring, so I was guessing at GPU memory pressure instead of looking at a graph. Both were false economies. Both are now fixed, or being fixed.\nWhere this goes next The lab works, but it is still a collection of good tools rather than a properly engineered platform. The next phase is about turning it into the latter. Concretely:\nA cleaner knowledge layer. This is the highest priority, because it is the thing that most limits quality right now. Structured, version-controlled, deduplicated source material that retrieval can draw on without dragging in noise — the spine of a proper second brain.\nGit everywhere. Compose files already live in Git. I want the knowledge base, the n8n workflow exports and the Home Assistant config there too, so the entire lab is reproducible from a repository and nothing important exists only inside a running container.\nReal security boundaries. Finishing the VLAN segmentation, tightening what the AI services can reach, and treating the lab as something with an attack surface rather than a trusted playground. Secrets stay in .env files and a vault, never in Git, and I want that discipline enforced rather than merely intended.\nEvaluations. Right now I judge quality by feel, which does not scale and is not honest. I want a small, repeatable eval set — fixed prompts with known-good answers — so that when I change a model or a prompt I can measure whether it got better or just different. This is the same instinct behind repeatable customer health checks: if you cannot measure it the same way twice, you are guessing.\nThe goal is not to build the biggest AI lab. It is to build one that teaches the right lessons and reflects how I would tell a customer to do it.\nA real closing thought The thing I keep returning to is that the lab did not make AI feel more magical. It made it feel more ordinary — in the best possible way. It is another layer of technology. A powerful one, but a layer, with the same needs as every other layer: somewhere to run, data to trust, access to control, and someone to own it when it breaks.\nThe organisations that get the most from AI will not be the ones who shout about it loudest. They will be the ones who understand their processes, their data, their risks and their infrastructure well enough to put AI to work quietly and sensibly. That understanding does not come from a slide deck. For me it came from a noisy box with a 3090 in it and a stack of compose files, and from being on the hook when the thing fell over.\nI did not build this to learn how to use AI. Plenty of people can use AI. I built it to learn how to design around it — and that, it turns out, is the part that actually matters.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/building-an-ai-infrastructure-lab-at-home/","title":"Building an AI Infrastructure Lab at Home"},{"content":"Building Knowledge Instead of Documents I have written the same proposal four times. Not the same client, not even the same product, but the same underlying thing: the sizing logic, the design rationale, the \u0026ldquo;here is why we recommend this rather than that\u0026rdquo; section. Each time I started from a blank page or, worse, from someone else\u0026rsquo;s deck that I half-trusted. Each time the knowledge that should have accreted into something durable instead evaporated the moment the engagement closed and the file went to sleep in a SharePoint folder nobody would ever open again.\nThat is the quiet tragedy of how most of our industry treats its own thinking. We produce enormous quantities of genuinely valuable analysis and then bury it in formats designed for printing, not for remembering. We confuse the act of writing a document with the act of building knowledge. They are not the same thing, and the difference compounds — for you and against you — over a career.\nThis is the argument for treating your knowledge as plain text under version control: Markdown, Git, and a static site to read it. It is not Markdown-purism. Word and PowerPoint still win in the places they win. But for the body of work that is supposed to make you faster next year than you were this year, the document was always the wrong container.\nThe document graveyard Walk into any consultancy\u0026rsquo;s SharePoint and you are walking through a graveyard. Thousands of .docx and .pptx files, each a frozen snapshot of a moment that has long since passed. They were written once, read once — by the client, at the review meeting — and then never opened again. The effort that went into them was real. The retained value is close to zero.\nThe symptoms are familiar to everyone and fixed by no one. There is the version chaos: Proposal_v2.docx, Proposal_v2_JS_edits.docx, Proposal_FINAL.docx, and the immortal Proposal_v3_FINAL_FINAL.docx. Nobody knows which is canonical. The \u0026ldquo;truth\u0026rdquo; is whichever copy happened to be attached to the last email. There is no history you can trust, only a sediment of near-identical files differing in ways no one can reconstruct.\nThen there is the format itself. A Word document is a proprietary binary blob. You cannot grep it. You cannot diff two versions and see, line by line, what actually changed and who changed it. You cannot link from a paragraph in one document to a paragraph in another and have that link mean anything. And you certainly cannot feed a folder of decks to a retrieval system and get sensible answers back, because the content is wrapped in layout — text boxes, slide masters, embedded images of diagrams — rather than expressed as content.\nSo the same things get written from scratch, engagement after engagement. The Conditional Access best-practice paragraph. The Citrix delivery-group sizing rationale I keep re-deriving instead of looking up — exactly the kind of thing I now keep as canon in my notes on modern Citrix architecture. Each is rewritten because finding the previous version, trusting it, and extracting it from its surrounding layout is slower than starting again. That is the tell. When reuse is harder than rewriting, you do not have a knowledge base. You have a graveyard with good search disabled.\nSharePoint is not where knowledge goes to live. It is where knowledge goes to die quietly, with full audit compliance.\nA document is not knowledge The root confusion is treating two different things as one. A document is a deliverable — a frozen artefact produced for a specific moment and a specific audience. A proposal, a design pack, a board deck. It has a date, a recipient, and a shelf life. Once delivered, it is done. Its job was to communicate something at a point in time, and it did.\nKnowledge is the opposite of frozen. Knowledge is living. It is linked to other knowledge, it is reusable across contexts, and crucially it accretes value over time rather than losing it. A good note about how Microsoft Graph authentication actually works does not expire when the engagement ends. It gets better as I add the edge case I hit last week, the gotcha about app-registration consent, the link to the n8n workflow that uses it.\nThe mistake is producing only documents and assuming the knowledge will somehow precipitate out of them. It does not. The document is the snapshot; the knowledge is the negative it was printed from, and most people throw the negative away. The discipline I am describing is keeping the negative. You write the durable knowledge first, in a form built to last and to be reused, and you generate documents from it when a moment demands one — not the other way round.\nflowchart TD subgraph GRAVE[Document graveyard] A[Word file] --\u003e B[SharePoint folder] C[PowerPoint deck] --\u003e B B --\u003e D[Never opened again] D --\u003e E[Rewritten next time] E --\u003e A end subgraph GRAPH[Knowledge graph] F[Atomic note] --\u003e G[Linked note] G --\u003e H[Linked note] F --\u003e H G --\u003e I[Generate deliverable] H --\u003e I F --\u003e J[Feeds Atlas RAG] G --\u003e J endThe left loop is a circle: write, file, forget, rewrite. The right loop is a spiral: write, link, reuse, and feed the same notes into something that makes them more useful still.\nWhy plain text and Markdown The first decision is the format, and it is the one people resist most because it feels like a downgrade. Plain text looks primitive next to a styled Word document. It is precisely that primitiveness that makes it durable.\nMarkdown is readable by a human with no tooling at all — it is just text with light, obvious conventions. It is readable by every editor, every operating system, every programming language, forever. There is no version of the future in which you cannot open a .md file. I cannot say the same about a .pptx from 2009 with embedded fonts and a long-dead plugin. Plain text is the only format I trust to outlive the tool that created it, which is the whole point of a knowledge base.\nIt is diff-able. Two versions of a note produce a clean, line-by-line comparison. You see exactly what changed. It is grep-able and machine-readable, which matters more every year, because the same notes that I read are the notes that feed retrieval. When I built Project Atlas, my local AI assistant, the knowledge base it retrieves from is this same Markdown — chunked, embedded, and queried. There is no export step, no conversion, no fidelity lost pulling text out of slide layouts. The thing I write for myself is the thing the model reads. A Word-and-SharePoint estate cannot do that without a brittle extraction pipeline bolted on the side, and even then it inherits all the layout noise.\nThat dual readability — humans and machines, the same source — is the quiet superpower. You are not maintaining one corpus for people and another for the RAG system. There is one corpus, in the most boring, most durable format there is.\nWhy Git, not a shared drive Format solves durability and reuse. Git solves trust. A shared drive gives you files; Git gives you history.\nEvery change is a commit, with an author, a timestamp, and a message saying why. git log is the audit trail that documents never had — not a sediment of near-identical files, but a precise, ordered record of how the thinking evolved. git blame tells me who wrote a particular line and when, which is invaluable when I am staring at a sizing assumption and trying to remember whether it was deliberate or a copy-paste accident. Branching lets me draft a major revision without touching the version of record, then merge it when it is ready. There is exactly one canonical state — main — and the history behind it is real, not reconstructed from email attachments.\nThis is the same approach I describe in building this site: Markdown content, Git for history, Docker to build and serve it. The repository is the source of truth. Everything else is a view onto it.\n# What actually happened to this design decision, in order git log --oneline -- citrix/delivery-group-sizing.md # a1f2c3d Correct session density after real-world load test # 9e8d7c6 Add gotcha: profile container IOPS ceiling # 4b3a2f1 Initial sizing rationale from ProjectName engagement # Who wrote this line and why git blame -L 40,48 citrix/delivery-group-sizing.md You will never get that from Proposal_v3_FINAL_FINAL.docx. The version chaos is not a discipline failure on the part of the people using Word. It is the inevitable result of a tool that has no concept of history. Git makes the right behaviour the default.\nWhy Hugo to publish it Plain text in Git is excellent to write and to machine-read, but humans also need to browse, and a flat repository of .md files is not pleasant to navigate by eye. That is where a static site generator earns its keep. I use Hugo with the Stack theme — the same engine behind this site.\nHugo takes the Markdown and renders a fast, searchable, linked website. Nothing is locked away. The source stays plain text in Git; Hugo is purely a rendering layer on top, and if it vanished tomorrow the knowledge would be entirely intact. I get full-text search, taxonomies, automatic cross-linking, and rendered Mermaid diagrams from the same files I write in any text editor. The build is a Docker container, so publishing is one command and the output is static HTML that will serve from anywhere with no database to corrupt and no runtime to patch.\nThe architecture is deliberately layered so that each layer is replaceable and the value lives in the bottom one.\nflowchart LR W[Markdown notes] --\u003e G[Git repo] G --\u003e H[Hugo build] H --\u003e S[Static site to browse] G --\u003e R[RAG ingest] R --\u003e A[Atlas answers questions] W -.source of truth.-\u003e WThe notes are the asset. Git is the history. Hugo is one consumer; Atlas is another. Tomorrow there might be a third. None of them own the knowledge — the plain-text repository does.\nKnowledge as a graph, not a folder tree Format, history, and rendering still leave the most important question: how the knowledge is structured. The instinct from the document era is to file things — folders, sub-folders, one big document per topic. Folders are a tree, and a tree forces every idea to live in exactly one place. Real knowledge does not work like that. An idea about Conditional Access belongs to identity, to Microsoft 365, to security, and to that specific health-check engagement all at once.\nSo I structure the base as a graph, not a tree. The unit is the atomic note: one note, one idea, small enough to link to precisely and reuse without dragging half a document along with it. Notes cross-reference each other directly. Taxonomies — tags and categories — let the same note surface under multiple themes without being duplicated. The structure that emerges is a web of connections, which is far closer to how I actually think and, not coincidentally, far better for retrieval. This is the second-brain principle I go deeper on in building a second brain: small, linked, durable units beat large, isolated documents every time.\nAn atomic note carries enough front matter to be found, filtered, and trusted, then says one thing well:\n--- title: \u0026#34;Delivery group sizing — session density\u0026#34; tags: [citrix, sizing, cvad, performance] status: validated sources: - 2026-04 ProjectName load test related: - \u0026#34;[[profile-container-iops]]\u0026#34; - \u0026#34;[[cvad-reference-architecture]]\u0026#34; updated: 2026-06-20 --- Single-server session density is bounded by the profile container IOPS ceiling long before CPU saturates. Plan to 8–10 heavy users per vCPU pair as a starting point, then validate under real load — the synthetic numbers lie. See the IOPS gotcha in [[profile-container-iops]]. That note is durable, linkable, machine-readable, and honest about its own provenance and status. When I write the next Citrix proposal, I do not re-derive any of this. I link to it, and the deliverable assembles from validated pieces — exactly the reusable-framework approach I argue for in building repeatable customer health checks.\nThe compounding effect This is the part that matters most and is hardest to feel in the first month. A knowledge base built this way compounds. Every note makes the next piece of work faster, because the next piece is partly assembled from notes that already exist and have already been corrected. The graveyard does the opposite: every engagement starts near zero, so your speed in year five is roughly your speed in year one with better war stories.\nCompounding is the difference between linear and exponential effort. The fiftieth proposal in a graveyard system costs about what the first did. The fiftieth proposal drawn from a living graph costs a fraction, because the sizing logic, the design rationale, the standard caveats, and the diagrams are all already written, already validated, already linked. You are composing, not authoring from scratch.\nOver time that becomes a genuine moat. Not the individual notes — anyone can write a note. The accumulated, interlinked, version-controlled, machine-readable body of corrected thinking is the thing that is hard to replicate and hard to take away. It is the asset I am actually building, and the documents are just exhaust from using it.\nWhere Word and SharePoint still win None of this is a case for never opening PowerPoint. That would be ideology, and ideology makes you worse at your job. There are jobs the document does better, and refusing to use the right tool is the same mistake as the graveyard, just pointed the other way.\nA polished client deliverable is one of them. When I hand a board a proposal, it needs to look considered, branded, and finished — and a styled Word or PowerPoint document does that far better than rendered Markdown. Collaboration with non-technical people is another. Track Changes and a comment thread in Word are how most of the world reviews a document, and asking a procurement lead to raise a pull request is absurd. Governance and formal sign-off — the legal weight of a named, dated, approved artefact — is genuinely what the document format is for. A frozen snapshot is exactly the right thing when you need a frozen snapshot.\nThe model that works is not Markdown instead of Word. It is Markdown underneath, Word on top. The knowledge lives as plain text in Git and compounds there. When a moment needs a deliverable, I generate one — assembling it from validated notes and polishing it in the tool the audience expects. The deliverable is downstream of the knowledge, disposable by design, and the next deliverable starts from the same improved base rather than from the last frozen file. This is precisely the flow I describe in from proposal to production: the proposal is a render of the knowledge, not the knowledge itself.\nWhat it costs, honestly I would be lying if I called this free. The cost is discipline, and it is paid up front, every day, before the payoff arrives.\nYou have to actually write the note, in the moment, when the lazy option is to finish the deliverable and move on. You have to keep notes atomic when the temptation is to dump everything into one sprawling file. You have to maintain links and resist letting the graph rot into orphaned fragments. You have to learn enough Git to be comfortable, which is a real barrier for people who came up entirely in the Office world. And the compounding is invisible early — for the first few months it genuinely feels like extra work for no return, because the base is too small to compose from yet.\nI got the granularity wrong at first. My early notes were really documents in disguise — long, multi-topic, impossible to link to precisely — and they retrieved badly in Atlas, pulling back a 2,000-word note when a query needed two sentences. I had to go back and atomise them. The discipline is not just \u0026ldquo;write it down\u0026rdquo;. It is \u0026ldquo;write it down small, link it, and keep it honest.\u0026rdquo; That is the actual price of admission.\nWhere this goes next The direction I am pushing is to close the loop between writing and generating. Right now I assemble deliverables semi-manually from notes. The next step is templated generation: a proposal skeleton that pulls the relevant validated notes by tag and produces a first-draft document automatically, ready for the human polish that Word is good at. The knowledge stays canonical; the document becomes genuinely disposable.\nI also want richer status and provenance on every note — draft, validated, deprecated — surfaced both in Hugo and in retrieval, so that Atlas can weight a battle-tested note above a half-formed one and tell me when it is leaning on something I never finished checking. And I want better detection of graph rot: orphaned notes, dead links, and ideas that have quietly contradicted each other. A knowledge base is a living thing, and living things need tending or they decay back into a graveyard with extra steps.\nThe real point The shift is not really about Markdown, Git, or Hugo. Those are implementation details, and you could make similar arguments with different tools. The shift is in what you think you are producing. If you believe you are producing documents, you will optimise for the moment of delivery and your knowledge will keep dying behind you. If you believe you are producing knowledge, you will optimise for durability, reuse, and compounding, and the documents will fall out of that almost for free.\nI spent years building a graveyard without noticing, because every individual document felt like progress. It was not. It was the same work, rewritten, on a treadmill. Building knowledge instead means the work accumulates — the version of me doing this in five years inherits everything the version doing it today figures out. That is the whole game. The document was always meant to be the output. The knowledge was always meant to be the asset. Most of us had it exactly backwards.\n","date":"2026-06-25T00:00:00Z","permalink":"/posts/building-knowledge-instead-of-documents/","title":"Building Knowledge Instead of Documents"},{"content":"Building krishaynes.co.uk The article you are reading is part of the thing it describes. That feels worth admitting up front, because it is the whole point.\nThis site is not a blog in the sense that word usually means. It is a public engineering notebook, a place where every project I actually build gets written up properly, in plain text, under version control, on infrastructure I own. The page rendering these words was a Markdown file before it was a web page. It lives in a Git repository next to every other thing I have written. If this domain disappeared tomorrow, the content would not, because the content is not really \u0026ldquo;on\u0026rdquo; a website at all. It is a folder of text files that happens to be published as a website.\nI did not start here. For years I had a Blogger site, like a lot of people who wanted somewhere to put their thoughts and did not want to think about hosting. Moving off it was one of those decisions that felt like procrastination at the time and turned out to be one of the better engineering choices I have made, precisely because it stopped being about a website and started being about how I keep knowledge. This is the honest account of why I moved, what I built, and the trade-offs I accepted.\nWhat was actually wrong with Blogger Blogger worked. That is the uncomfortable part. It did exactly what it promised. You log in, you click into a WYSIWYG editor, you type, you hit publish, and a post appears on the internet. For a long time I told myself that was enough.\nThe problem is not that these platforms are bad at publishing. The problem is what they do to the content itself, and it took me a while to see it clearly.\nStart with ownership, because everyone says they own their content and almost nobody does. On Blogger my words lived in Google\u0026rsquo;s database in a format I never saw and could not meaningfully export. There was an export button, of course, that produced an XML file full of platform-specific markup and embedded styling decisions I never made. That is not ownership. Ownership is being able to pick up your work and walk to a completely different tool without a migration project. If leaving costs you days of cleanup, you do not own your content, you are renting space in someone else\u0026rsquo;s schema.\nThen there is fragility. A WYSIWYG editor hides the actual content behind a wall of generated HTML. Paste something in from Word and you inherit a tangle of \u0026lt;span style=\u0026quot;...\u0026quot;\u0026gt; tags that follow you forever. The text you see is not the text that is stored. You are editing a rendering of your content, not the content, and the gap between those two things is where rot sets in. I had posts where a theme change broke the formatting in ways I could not fix without going into raw HTML the platform did not really want me touching.\nAnd there is no real history. This is the one that finally got me. If I edited a post and made it worse, there was no clean way to see what it used to say, no diff, no \u0026ldquo;what changed and when\u0026rdquo;. The current state was the only state. For a personal blog that sounds like a non-issue. For an engineering journal, where the value is in being able to look back and see how my thinking evolved, it is fatal. I want to be able to ask \u0026ldquo;what did I believe about local models a year ago\u0026rdquo; and get an answer from the record, not from memory.\nThe deepest problem, though, is existential. A SaaS blogging platform is a building you do not own, and one day the landlord decides to demolish it. Google has a long, well-documented history of closing products people depended on. Blogger has survived, but I no longer wanted to bet a decade of writing on the continued enthusiasm of a team inside a company that does not particularly care about it. When the platform dies, the content dies with it, unless the content was never really trapped inside the platform in the first place.\nIf leaving a tool is a project, you never owned your work. You were renting.\nSo the brief I set myself was simple. I wanted a way of writing that would outlive whatever I used to write it.\nThe choices, and the ones I rejected Three decisions sit underneath this site, and they are deliberately boring, which is how you know they will last.\nMarkdown, because plain text outlives tools. I write everything in Markdown now, and the reason is not that Markdown is elegant, though it is. It is that a Markdown file is a plain text file. I can open it in anything, on any operating system, in forty years, with no special software. It is human-readable as source and renders cleanly as output, so there is no gap between what I store and what I mean. It diffs beautifully, line by line, which makes change history actually useful. It is the lingua franca of technical writing, so it moves between tools without translation.\nThere is a fourth reason that matters more every month: Markdown is the format machines read best. The same notes I write here feed Atlas, my local AI assistant, whose knowledge base is a folder of Markdown under Git. Plain text is what retrieval systems want, what models were trained on, what tooling parses without heroics. By writing in Markdown I am not choosing between writing for humans and writing for machines. I am writing once and getting both. That single decision is why this site and my second brain are, underneath, the same thing, which is the argument I make in building knowledge instead of documents.\nGit, because content deserves the same rigour as code. Once the content is plain text, putting it in Git is obvious, and it changes everything. The content is a repository. Every change is a commit with a message and a date. I can branch, draft a controversial post on a feature branch, sit on it for a week, and merge it when I am happy, or throw the branch away with no trace. I can recover anything. A post I deleted two years ago is one git log away. The history is not a feature bolted onto the content, it is inseparable from it. This is exactly the discipline I keep my homelab in, where compose files in Git are the source of truth and the running containers are just the current checkout, as I describe in lessons from building a Docker homelab.\nHugo, because I wanted a website, not a database. With plain text in Git, I needed something to turn it into a site. The honest alternative was WordPress, and I rejected it deliberately. WordPress means a database, a PHP runtime, a plugin attack surface, and a backup story that is genuinely hard to get right. It reintroduces every fragility I was trying to escape. A static site generator does the opposite. Hugo takes my Markdown and a theme and produces plain HTML files. There is no database. There is nothing to hack at runtime because there is no runtime, just files a web server hands out. It is absurdly fast, builds the entire site in well under a second, and it has a mature theme ecosystem. I chose hugo-theme-stack because it is clean, card-based, handles categories and tags well, and renders Mermaid diagrams and code blocks without me fighting it. The diagram below this paragraph is just a fenced code block in the source; the theme turns it into a picture.\nI considered Jekyll and Eleventy. Jekyll\u0026rsquo;s Ruby toolchain has caused me more grief than Hugo\u0026rsquo;s single binary ever has. Eleventy is lovely but I did not want to assemble a site from JavaScript plugins. Hugo is one binary, no dependencies, no node_modules, and it is fast enough that the build never breaks my train of thought.\nDocker, because the build should be reproducible. I did not want to install Hugo on my laptop and discover in a year that a version bump had broken my site. So Hugo runs in a container, defined by the compose file that lives in this very repository. The toolchain is pinned and reproducible. Anyone, including future me on a fresh machine, can clone the repo, run one command, and get an identical local preview. This is the actual service:\nservices: hugo: image: ghcr.io/gohugoio/hugo:latest container_name: hugo-site working_dir: /src volumes: - .:/src ports: - \u0026#34;1313:1313\u0026#34; command: - server - --bind - 0.0.0.0 - --baseURL - http://192.168.100.250:1313 - --buildDrafts - --disableFastRender restart: unless-stopped docker compose up and I have a live-reloading preview on the lab network. Save a Markdown file, the browser refreshes, the loop is instant. For a production build I swap the server command for a plain hugo invocation and the container spits out a public/ directory of static files, which is all a web server ever needs to see.\nHow it actually fits together The architecture is deliberately almost nothing, and that is the feature. There are no moving parts at serve time. All the cleverness happens at build time and then gets frozen into static files.\nflowchart TD A[Markdown files] --\u003e B[Git repository] B --\u003e C[Hugo in Docker] D[hugo-theme-stack] --\u003e C C --\u003e E[Static HTML in public] E --\u003e F[Web server] F --\u003e G[Reader] B --\u003e H[Atlas knowledge base] H --\u003e I[Local LLM retrieval]The same Markdown that flows down the left into a published page also flows right into the knowledge base my assistant reads. One source, two consumers. The website is the human-facing output; the retrieval index is the machine-facing one. Neither is the canonical store. The Git repository is.\nA single article is just a file with a block of YAML front matter at the top, which is how Hugo knows the title, date, and where to file it under categories and tags. Here is the shape every post takes:\n--- title: \u0026#34;Building krishaynes.co.uk\u0026#34; date: 2026-06-25 description: \u0026#34;Why I moved off Blogger and built this site on Hugo and Git.\u0026#34; draft: false categories: - Projects - Infrastructure tags: - hugo - markdown - git - static-site --- The information architecture matters more than it looks. The content is organised into sections, which are just top-level folders: projects, ai, infrastructure, automation, citrix, notes, and posts. Sections are the spine, the structural home of a piece. A write-up of something I built goes in projects. A piece of thinking about where the industry is going goes in posts. A half-formed working note goes in notes, which is where you will find what I\u0026rsquo;m building now, the rolling log of current work.\nCategories and tags do a different job, and conflating them is a classic mistake. A piece lives in exactly one place on disk, but it can touch several themes. Categories are the broad buckets, capitalised, a handful of them: AI, Infrastructure, Automation, Projects, Knowledge. Tags are fine-grained and lowercase: hugo, ollama, docker, n8n. The section says where the article lives. The categories say what kind of thing it is. The tags say what it touches. Get that separation right and the site stays navigable as it grows; get it wrong and you end up with forty near-identical tags and no categories that mean anything.\nHow I actually intend to use it A blog is something you maintain. A notebook is something that compounds. The distinction is the entire reason this exists.\nThe discipline I am holding myself to is straightforward: every project gets written up here. Not a polished marketing case study, but the real account, including the parts where I got it wrong. The lab I run, described in building an AI infrastructure lab at home, is the raw material; this site is where that work gets turned into something durable. When I size a GPU, configure Caddy, debug an n8n workflow, or work out a battery charging strategy, the writing-up is part of finishing the job, not an optional extra after it.\nThat is the second-brain argument I make at length in building a second brain. Knowledge you do not write down decays. Knowledge you write down into a trapped platform decays slightly slower and then dies with the platform. Knowledge you write down into versioned plain text, that both you and your AI can read, accumulates. Each article makes the next one easier to write, because I can link to settled thinking instead of re-deriving it. The cross-references in this very paragraph are the mechanism: the site is becoming a graph, not a pile.\nIt also forces clarity. You cannot write up a thing you only half understand. The act of explaining the build, in prose, to a hypothetical reader, repeatedly exposes the bit I was hand-waving over. Writing is how I find the holes.\nWhat I got wrong, and the honest trade-offs I will not pretend this is free. The move off Blogger bought me durability and ownership and cost me convenience, and anyone considering the same should know the bill.\nThere are no comments out of the box. A static site has no server to receive them. You can bolt on a third-party comment system, but that is exactly the kind of external dependency I left Blogger to escape, so I have simply gone without. The conversation happens elsewhere, and I have made my peace with that.\nYou write Markdown, not click buttons. For me that is a feature, but it is a real shift. There is no toolbar. If you want a table you write a table in pipes and dashes. For anyone not already comfortable in a text editor, that is friction, and pretending otherwise would be dishonest.\nThere is a static-site learning curve, and I underestimated it at first. Hugo\u0026rsquo;s templating, the way sections and taxonomies and front matter interact, the occasional cryptic build error, the theme\u0026rsquo;s configuration options buried in a TOML file. None of it is hard once it clicks, but the first weekend was not the frictionless experience the \u0026ldquo;just write Markdown\u0026rdquo; pitch implies. My first attempt at the section structure was wrong. I dumped everything into posts and used tags to fake structure, and the site became an undifferentiated stream within about fifteen articles. Reorganising into real sections later meant moving files and fixing internal links, which Git made survivable but which I should have got right from the start.\nI also over-tuned the theme early, fiddling with colours and card layouts before I had written anything worth reading. Classic procrastination dressed as work. The lesson, which I keep relearning, is that the content is the product and the presentation is a distant second.\nWhere this goes next The roadmap is modest and concrete, which is how I like roadmaps.\nThe most obvious next step is closing the loop with Atlas. Right now the same Markdown feeds both the site and the assistant\u0026rsquo;s knowledge base, but the wiring is manual. I want the publish step and the index step to be one action, so that anything I write is immediately retrievable by the assistant without me copying files around. That is a small n8n workflow away.\nI want to automate the build and deploy properly, so that a git push triggers a container build and pushes the static output to where it is served, with no manual steps. The pieces are all in the lab already; it is a matter of joining them.\nLonger term I would like a lightweight way to surface related articles automatically, generated from the actual content rather than hand-maintained tags, which is the kind of thing a local model is genuinely good at. And I want to be more disciplined about the rolling notes section, treating it as a daily-ish log rather than something I tidy up only when a project is finished.\nNone of this is exotic. It is all the same pattern: plain text, in Git, processed by reproducible tooling, serving both me and the machine I am building.\nThe meta point I will end where I started, because the recursion is the argument. This article is a Markdown file in a Git repository, built by Hugo in a Docker container defined by a compose file three folders away, and it exists to explain why it is a Markdown file in a Git repository built by Hugo in a Docker container. The medium is the message, and the message is that the medium should outlive the platform.\nBlogger gave me somewhere to publish. This gives me somewhere to keep. The difference between publishing and keeping is the difference between a stream that flows past and a body of work that accumulates, and once I saw that difference I could not unsee it. A platform that can disappear is not a place to put a decade of thinking. A folder of plain text under version control is the closest thing I have found to permanence in software, and permanence is exactly what an engineering journal needs to be worth keeping.\nEvery project from here gets written up in this format. The notebook compounds. That was the whole point.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/building-krishaynes-co-uk/","title":"Building krishaynes.co.uk"},{"content":"Building Repeatable Customer Health Checks Most of my career has a recurring scene in it. A customer asks for a \u0026ldquo;health check\u0026rdquo; of something — their Microsoft 365 tenant, their VMware estate, their backup posture, their Citrix farm — and a good engineer spends two or three days clicking through admin consoles, copying screenshots into a Word document, and writing the same observations they wrote for the last customer with the names changed. The customer pays a day rate for someone to look at a screen and type. The output is a forty-page document that is already going stale by the time it lands in their inbox.\nI have done this dozens of times. I have also been the person who reviewed those documents, and the uncomfortable truth is that the quality varied wildly depending on who ran it. One engineer would catch a missing Conditional Access policy; another would miss it but write three paragraphs about mailbox sizes nobody asked about. Same service, same price, completely different value. That is not a delivery problem. That is a design problem.\nThis article is about the pattern I have been building to fix it. The Microsoft 365 AI health check is one concrete instance of it, but the interesting thing was never the M365 part. It was realising that almost every infrastructure review is the same shape underneath, and that the shape can be turned into a product instead of a heroic effort.\nWhy the manual way is broken A manual health check fails on four fronts at once, and they compound.\nIt is slow. Days of an experienced engineer\u0026rsquo;s time go into navigating interfaces, and most of that time is data gathering, not thinking. The judgement — the bit the customer is actually paying a senior person for — is maybe ten percent of the effort. The rest is clerical.\nIt is inconsistent. The findings depend on which consoles the engineer remembered to open and what they happened to notice. There is no guarantee that two reviews of identical tenants produce the same report, which means the report tells you as much about the reviewer as about the environment.\nIt goes stale instantly. A screenshot of a Conditional Access policy is true for exactly as long as nobody changes it. The document is a photograph of a moving thing. A month later it is fiction, and nobody re-runs it because re-running it means another three days.\nAnd it does not scale. The only way to do more health checks is to consume more engineer-days, which is exactly the resource you are short of. The economics are linear and the margin is thin, because you are essentially reselling a person\u0026rsquo;s afternoon.\nThe customer is not paying for someone to click through consoles. They are paying for an opinion they can trust. Everything between those two things is waste.\nOnce I framed it that way, the goal became obvious. Collect the facts by machine, deterministically, every time. Spend the human — and the AI — only on the judgement and the narrative. Make the whole thing re-runnable so that \u0026ldquo;what changed since last quarter\u0026rdquo; is a button, not a project.\nDesign principles I settled on The first and most important decision was to separate fact-collection from judgement, completely and structurally. These are different kinds of work with different failure modes, and mixing them is what makes manual reviews unreliable.\nFacts are collected programmatically, through APIs, never through a human reading a screen. If Microsoft Graph can tell me whether security defaults are enabled, I do not want an engineer eyeballing a toggle. The collector asks the API and records the answer. This is the same argument I made in why every infrastructure engineer should learn Python — the API is the source of truth, and driving it by hand is the slow, unauditable path.\nJudgement on those facts is deterministic wherever it can be. \u0026ldquo;MFA is not enforced for admins\u0026rdquo; is not an opinion; it is a rule applied to a fact. Rules like that belong in code, where they are explicit, testable, and identical for every customer. I do not want a language model deciding whether something is a finding. I want it deciding how to explain a finding that the rules have already established.\nThat leaves a clear division of labour. Deterministic rules decide what is true and what is wrong. The AI decides how to say it — the narrative, the prioritisation, the readability, turning a list of thirty raw findings into a report a busy IT manager will actually read and act on. The model is a writer, not an auditor.\nOutput is templated, so quality does not depend on who ran the tool. The branding, the section structure, the executive summary format — all fixed. The engineer running it this week and the engineer running it next month produce documents that look and read the same, because the document is generated, not authored.\nAnd everything is versioned and re-runnable. Each run is stored against the customer with a timestamp. Because the data model is stable between runs, I can diff them and show drift — the policy that was disabled last month, the new licences that appeared, the Secure Score that quietly dropped. That diff is something a manual process simply cannot produce, and it is often the most valuable page in the report.\nThe reusable framework The architecture is deliberately boring, because boring is what makes it reusable. It is a pipeline of five stages, and the whole point is that only the first stage knows or cares which product is being checked.\nflowchart TD A[Collectors] --\u003e B[Normalised data model] B --\u003e C[Rules and scoring engine] C --\u003e D[AI report writer] D --\u003e E[Branded output] F[Check definitions] --\u003e C G[Local LLM via Ollama] --\u003e D B --\u003e H[Historical store] H --\u003e CCollectors are the only product-specific part. An M365 collector talks to Microsoft Graph. A VMware collector talks to vCenter via the REST or PowerCLI surface. A Veeam collector hits the Veeam REST API. A Citrix collector queries the CVAD or DaaS APIs. Each one has a single job: pull raw configuration and posture data and hand it onward. Nothing in a collector makes a judgement. It does not decide what is good or bad. It just gathers.\nThe normalised data model is the heart of the whole thing, and I will say more about why below. Every collector writes into the same structured shape — entities, properties, relationships, a score-able set of facts — regardless of which platform produced them. A \u0026ldquo;control\u0026rdquo; from M365 and a \u0026ldquo;control\u0026rdquo; from Veeam land in the same schema. This is what lets the rest of the pipeline stay product-agnostic.\nThe rules and scoring engine reads the normalised data and the check definitions, and produces findings. A check definition is a small, declarative description of one thing worth evaluating: what to look at, what counts as a pass, how severe a failure is, and which category it rolls up into. Rules are deterministic and unit-tested. The scoring engine aggregates findings into category scores and an overall posture.\nThe AI report writer takes the findings — never the raw data — and writes the human-facing document: an executive summary, prioritised recommendations, and readable explanations of each finding. It runs against a local model through Ollama, for reasons I will get to.\nBranded output renders the writer\u0026rsquo;s prose into the deliverable: a templated document with the consultancy\u0026rsquo;s styling, consistent every time.\nA check definition looks like this The unit of work is the check definition, and keeping it small and declarative is what makes the system extensible. Here is roughly the shape I use, in YAML:\n- id: m365.identity.admin_mfa title: MFA enforced for privileged roles product: m365 category: identity severity: critical source: graph.conditional_access.policies rule: \u0026gt; any(policy.state == \u0026#34;enabled\u0026#34; and \u0026#34;All\u0026#34; in policy.conditions.roles and policy.grant_controls includes \u0026#34;mfa\u0026#34;) pass_message: All privileged roles require multi-factor authentication. fail_message: One or more admin roles can sign in without MFA. remediation: \u0026gt; Create a Conditional Access policy targeting privileged directory roles that requires multi-factor authentication. weight: 10 And the engine that consumes it stays tiny and generic, because all the product knowledge lives in the definition and the collector, not in the loop:\ndef evaluate(check, data_model): facts = data_model.resolve(check[\u0026#34;source\u0026#34;]) passed = run_rule(check[\u0026#34;rule\u0026#34;], facts) return Finding( check_id=check[\u0026#34;id\u0026#34;], category=check[\u0026#34;category\u0026#34;], severity=check[\u0026#34;severity\u0026#34;], passed=passed, message=check[\u0026#34;pass_message\u0026#34;] if passed else check[\u0026#34;fail_message\u0026#34;], remediation=None if passed else check[\u0026#34;remediation\u0026#34;], weight=check[\u0026#34;weight\u0026#34;], ) Adding a new check is adding a YAML block. Adding a new product is writing one collector that fills the data model, then writing check definitions against it. The M365 instance and a hypothetical Veeam instance share every line of the engine, the writer and the renderer. That is the leverage. It is the same instinct as turning a one-off engagement into a system that I wrote about in from proposal to production — the value is in the repeatable machinery, not the heroic delivery.\nGrounding the AI so it cannot invent findings The single biggest risk in putting a language model anywhere near a customer-facing audit document is that it confidently writes something that is not true. A report that invents a finding is worse than no report, because it destroys trust in the whole exercise. So the writer is constrained hard.\nThe model never sees the customer\u0026rsquo;s raw data and is never asked to discover problems. It receives a structured list of findings that the deterministic engine has already decided are true, and its instructions are explicit: write only about these findings, do not introduce new ones, do not speculate about anything not present in the input. Every claim in the prose must trace back to a finding object with an ID.\nIn practice this means the prompt carries the findings as data and the model\u0026rsquo;s job is transformation, not generation of fact. If a finding is not in the list, the model has nothing to say about it, and the template will not have a slot for it. I also do a cheap post-check: the rendered report\u0026rsquo;s claims are cross-referenced against the finding IDs, and anything that does not map gets flagged. This is the same discipline that separates the AI projects that survive contact with reality from the ones that do not, which is most of them — a theme I went into in why most AI projects fail. The model is a writer working from a brief. It is not allowed to do research.\nMaking it safe enough to point at a customer You cannot build a tool that hoovers up a customer\u0026rsquo;s tenant configuration without taking the security of the tool itself seriously. Three things matter.\nIt is read-only. Every collector authenticates with credentials scoped to read, never write. The M365 app registration requests read permissions on Graph and nothing else. There is no code path in any collector that modifies the customer environment, because there is no business reason for one and every reason against.\nIt uses least privilege. Rather than a single god-mode account, each collector gets exactly the permissions it needs to read exactly what it reads. If a customer wants to review the consent grant before they approve it, it should be short and obviously harmless.\nAnd the data goes nowhere it should not. The findings are written by a local model running on my own hardware through Ollama, on the GPU box I built for exactly this kind of work — a single RTX 3090, 24GB of VRAM, sized for VRAM-per-pound rather than raw speed. A customer\u0026rsquo;s identity and infrastructure posture is precisely the sort of data you do not want flowing into a third-party API to be summarised. Keeping the model local means the sensitive part of the pipeline never leaves a machine I control. That is not a performance decision; it is a trust decision, and it is one of the strongest arguments for running your own models that I make when customers ask why I bothered.\nThe business case The reason I care about this is not technical elegance. It is margin and consistency.\nA manual health check is a bespoke services effort: linear cost, thin margin, quality that depends on the individual. The automated version is a product. The expensive engineering happens once, in building the collectors and the data model and the check library. After that, running a check for a new customer costs minutes of compute and a short review, not days of senior time. The same deliverable, produced at a fraction of the cost, at higher and more predictable quality.\nThat changes what the engagement is. The engineer is freed from data-gathering and spends their time on the part that genuinely needs a human: interpreting the findings in the customer\u0026rsquo;s context, having the conversation, deciding what actually matters for this organisation this quarter. That is where presales value lives, and it is exactly the shift I argued for in the future of technical presales and stocked the toolbox for in building an AI consultancy toolkit. The tool does the clerical work. The human does the judgement. Both are now doing the thing they are good at.\nIt also turns a one-off into a relationship. Because the check is re-runnable and shows drift, it naturally becomes a recurring service — a quarterly posture review that gets more valuable each time because the history accumulates. That is a far better business than selling the same forty-page snapshot to a new customer every week.\nWhat I got wrong, and what I learnt The data model is the hard part. I went in thinking the collectors would be the difficult bit — wrangling Graph, vCenter, Veeam, all their quirks. They are fiddly but tractable. The genuinely hard, genuinely intellectual work is designing a normalised shape that an M365 control and a VMware setting and a backup policy can all live in without being mangled. Get that right and everything downstream is easy and shared. Get it wrong and you end up with product-specific code leaking into the engine and the writer, and you are back to building a separate tool for every platform. I rebuilt the data model twice. I should have spent longer on it before writing a single collector.\nCustomers trust consistency more than they trust brilliance. I expected pushback on the reports being \u0026ldquo;machine-generated\u0026rdquo;. The opposite happened. The fact that the same check produces the same structured assessment every time, that two environments can be compared on the same scale, that last quarter\u0026rsquo;s report and this quarter\u0026rsquo;s are directly comparable — that is what built confidence. A slightly less eloquent report that is rigorously consistent beats a beautifully written one-off that nobody can reproduce.\nThe AI is a writer, not an auditor, and the moment I forgot that I got burned. An early version let the model reason a little more freely about the data, and it produced lovely prose containing a finding that was simply not true — it had pattern-matched a plausible-sounding problem. Nobody shipped it, but it was the wake-up call that hardened the grounding. The model\u0026rsquo;s freedom ends at how to phrase what the rules already decided.\nWhere this goes next The obvious next step is making the checks continuous rather than episodic. The whole pipeline is already re-runnable, so scheduling it is mostly plumbing — the same n8n orchestration spine that runs my other automations can trigger a check on a cadence, store the result, and alert when a score drops or a new critical finding appears. A health check that runs itself every night and only speaks up when something changes is a different and better product than a document.\nAfter that, benchmarking. Once you have run the same check across many environments, you can tell a customer not just \u0026ldquo;your Secure Score is 62\u0026rdquo; but \u0026ldquo;that sits in the bottom quartile for organisations your size\u0026rdquo;. Anonymised, aggregated, careful with the data — but genuinely useful context that no single review can offer.\nAnd eventually, self-service. A portal where a customer connects their own tenant under a read-only consent and runs the check themselves, gets the report, and watches their posture over time. The engineer steps in for the conversation, not the click-through. That is the end state I am building towards: the tool handles the facts and the writing, and the human is reserved entirely for judgement.\nA closing thought The thing I keep coming back to is that the health check was never really a document. The document was just the artefact we produced because producing it by hand was all we knew how to do. What the customer actually wants is a trustworthy, current, comparable opinion about whether their environment is in good shape — and the manual process was a slow, expensive, inconsistent way of approximating that.\nOnce you collect the facts by machine, decide the findings by rule, and use the model only to make it readable, the document stops being the work. It becomes a side effect. And the engineer who used to spend three days producing it gets to spend those three days being the thing a machine cannot be: someone with an opinion worth paying for.\n","date":"2026-06-25T00:00:00Z","permalink":"/automation/building-repeatable-customer-health-checks/","title":"Building Repeatable Customer Health Checks"},{"content":"Designing Infrastructure for AI Workloads There is a comfortable lie going around that infrastructure no longer matters for AI. You just call an API. Someone else owns the GPUs, the cooling, the network, the lot. Your job is a prompt and a credit card.\nI do not buy it. I have spent enough time as a solutions architect, and enough nights in my own home lab, to know that the API is the thin layer on top of a very physical, very opinionated stack. The moment you care about cost per token, data residency, latency, or running anything yourself, the infrastructure underneath reasserts itself with force. AI did not abolish infrastructure. It raised the stakes.\nThis article is the technical backbone for a lot of what I write elsewhere. When I size a GPU in another piece, or wave my hand at \u0026ldquo;the inference box\u0026rdquo;, this is where the reasoning lives. It is deliberately opinionated, because the worst infrastructure decisions I have seen came from people refusing to take a position.\nWhy this still matters when you can \u0026ldquo;just call an API\u0026rdquo; The API model is genuinely good for a class of problems. Spiky demand, frontier-model quality, no desire to own hardware — call the API and move on. I do exactly that for plenty of things.\nBut \u0026ldquo;just call an API\u0026rdquo; quietly assumes your data is allowed to leave the building, your latency budget tolerates a round trip to someone else\u0026rsquo;s data centre, your costs scale linearly and forgivingly, and the model you depend on will still exist, unchanged, next quarter. For a homelab those assumptions are mostly fine. For a regulated enterprise tenant they are frequently false, which is half the reason most AI projects fail — they treat a production system as a demo with a bigger prompt.\nThe interesting truth I keep coming back to is that AI is becoming infrastructure in the same way databases and networks did. It stops being a feature you bolt on and becomes a substrate everything else assumes is there. And substrate has to be designed, not summoned.\nSo I will treat an AI workload the way I treat any other tier-one system: what does it consume, where does it hurt, and what happens when it grows.\nThe real constraint is VRAM, not FLOPS Everyone fixates on GPU compute. The marketing is all TFLOPS and tensor cores. In practice, for the workloads most of us actually run, the wall you hit first is video memory.\nA model has to fit in VRAM to run well. If it does not fit, you either offload layers to system RAM — which collapses your throughput as data shuffles across the PCIe bus — or you drop to a smaller model or a harsher quantisation. VRAM is the gate. Everything else is negotiable.\nThis is why I chose a single RTX 3090 with 24GB for the inference box rather than something newer and faster with less memory. I sized for VRAM-per-pound, not raw speed, and I have never regretted it. Twenty-four gigabytes is, in my experience, the practical line where local models stop being toys. Below it you are constantly compromising; above it you are paying data-centre prices. I explain the model side of this in more depth in my journey into local LLMs, but the hardware logic is simple: buy memory.\nA rough way to think about what fits. A model\u0026rsquo;s weights consume, in bytes, roughly the parameter count multiplied by the bytes-per-parameter of the quantisation. At Q4_K_M — around 4.5 bits per weight, my usual default — an 8B model is comfortably under 6GB of weights, a 13B sits around 8GB, and a 70B lands near 40GB and simply will not fit on a 24GB card without aggressive offload or a brutal 2-bit quant that hurts quality.\nVRAM budget on a 24GB card (Q4_K_M, rough) 8B ~5-6 GB weights + KV cache → easy, long context 13B ~8 GB weights + KV cache → comfortable 34B ~20 GB weights + KV cache → tight, short-ish context 70B ~40 GB weights → does not fit, don\u0026#39;t pretend But weights are only half the story, and the half everyone forgets is the KV cache. Every token in the context window has to keep its key and value tensors resident in VRAM so attention can see them. That cache grows linearly with context length and with batch size. Push the context from 4k to 32k tokens and the KV cache can swallow several gigabytes on its own. So your usable context length is not a free model property — it is a memory budget you spend out of the same 24GB the weights already claimed.\nThat gives a real design rule. Inference wants enough VRAM for weights plus your worst-case KV cache (context length times concurrency). Fine-tuning is a different animal — full fine-tuning needs memory for weights, gradients, and optimiser states, often three to four times the inference footprint, which is why I do parameter-efficient tuning (LoRA / QLoRA) at home and rent a fat cloud GPU for the rare full run. Sizing for fine-tuning on consumer hardware means sizing for QLoRA or not at all.\nBatching is the lever that turns VRAM into throughput. A single request barely tickles the GPU; batching many concurrent requests amortises the cost of streaming weights and keeps the tensor cores fed. The catch is that batching multiplies the KV cache. So the honest sizing question is not \u0026ldquo;will the model fit\u0026rdquo; but \u0026ldquo;will the model plus the KV cache for N concurrent users at C context length fit\u0026rdquo;. For a household, N is tiny and it all just works. For an enterprise serving hundreds of sessions, N dominates and you are buying VRAM by the rack.\nOn consumer versus data-centre cards: a data-centre card buys you ECC memory, much more VRAM per card, NVLink for pooling memory across cards, and a support contract. A 3090 buys you 24GB at a fraction of the price with no ECC, no sanctioned NVLink in newer generations, and a power-hungry consumer board. For learning, prototyping, and personal inference, the consumer card wins on every metric that matters to me. For multi-tenant production with uptime guarantees, it does not. Know which problem you are solving before you spend.\nStorage: three different jobs people lump into one \u0026ldquo;Fast storage\u0026rdquo; is not a requirement, it is a slogan. AI workloads have three distinct storage profiles and conflating them is how you end up with an expensive array that is wrong for all three.\nFirst, model weights. These get loaded into VRAM at startup and on every model switch. What you want here is low-latency, high-throughput sequential read — fast NVMe. Loading a 40GB model off a spinning disk is a coffee break; off a decent Gen4 NVMe it is a few seconds. I keep all the Ollama models on NVMe for exactly this reason. It is the difference between a model swap being invisible and being a wince.\nSecond, datasets for training or RAG ingestion. This is a throughput game, often sequential, often large. Bulk capacity matters more than latency. This is what the NAS is for — datasets and corpora live there, get pulled to fast local storage when actively worked on, and everything important is backed up with proper 3-2-1 thinking.\nThird, and the one people get wrong, the vector database. A vector DB doing similarity search is a low-latency random-read workload, much closer to a transactional database than to a dataset dump. Put it on NVMe, give it RAM for its index, and do not let it share spindles with your backup jobs. Where the vector DB physically lives is a real decision: I keep it on the always-on side of the lab, on the mini-PC fleet near the orchestration layer, not on the GPU box — because retrieval should keep working even when the GPU is busy or offline. The retrieval layer behind Project Atlas is built on exactly that separation.\n# the storage split that matters # # Ollama runs natively on the GPU box, not in Docker — its model # directory points straight at fast NVMe, set on the host service: # OLLAMA_MODELS=/mnt/nvme/models # OLLAMA_KEEP_ALIVE=30m (don\u0026#39;t reload weights constantly) # # Everything else is a container; the storage split below is the point: services: qdrant: image: qdrant/qdrant volumes: - /mnt/nvme/qdrant:/qdrant/storage # vector index: low-latency NVMe ingest-staging: image: busybox volumes: - /mnt/nas/datasets:/data # bulk corpora on the NAS The principle: match the medium to the access pattern, not to a generic idea of \u0026ldquo;fast\u0026rdquo;.\nNetworking: east-west, model pulls, and latency Networking for AI is mostly invisible until it is the bottleneck, and then it is the only thing anyone talks about.\nThree flows matter. East-west traffic is the chatter between your own services — the n8n workflow calling the model, the model calling out to retrieval, the embeddings service answering the vector DB. In a single-box setup this is loopback and free. The moment you spread services across hosts, this traffic crosses the wire, and a chatty RAG pipeline can put a surprising amount of east-west load on the network. Keep tightly coupled services close.\nBandwidth for model pulls is the bursty one. Pulling a fresh model is tens of gigabytes at once. On home gigabit that is a few minutes; nobody cares. In a cluster pulling the same model to twenty nodes it is a thundering herd, and you want a local registry so you pull once and distribute internally.\nLatency for interactive use is the one users feel directly. Time-to-first-token is a latency budget, and every hop spends it. This is the strongest technical argument for keeping inference local: a local model answering in tens of milliseconds of network overhead beats a faster cloud model sitting behind a 100ms+ round trip when the interaction is conversational. For batch work latency is irrelevant; for a chat assistant it is the whole experience.\nAt home this is one flat network being slowly segmented into VLANs — trust, IoT, and lab. Segmenting the lab is not just tidiness; it is the network half of governance, which I will come back to.\nGPU placement: bare metal, passthrough, MIG, and containers Here is where I take a firm position, because the options genuinely differ — and because I changed my mind by getting it wrong first.\nThe inference box is bare-metal Ubuntu — AMD Ryzen, an RTX 3090, the NVIDIA driver and Ollama installed straight on the host. The question every local AI builder faces is how the GPU reaches the workload, and there are really three answers.\nflowchart TD A[GPU workload] --\u003e B{Sharing needed} B --\u003e|One workload owns the GPU| C{Need a VM around it} C --\u003e|No, single-purpose box| E[Bare metal: driver + runtime on the host] C --\u003e|Yes, mixed guests| D[PCIe passthrough to one VM] B --\u003e|Many isolated tenants| F{Card supports MIG} F --\u003e|Yes datacentre card| G[MIG partitions] F --\u003e|No consumer card| H[Time-slice or queue] E --\u003e I[Ollama native, containers alongside] D --\u003e J[Containers inside the VM] G --\u003e K[One slice per tenant]Bare metal installs the driver on the host and runs the workload there directly, with nothing between the runtime and the card. Lowest overhead, highest stability, least flexibility. This is what I use, and I arrived at it the hard way. I first tried PCIe passthrough — handing the whole physical card to a single VM to get snapshots and rebuildability for free. It worked, but it was measurably slower and far more fragile across kernel and driver updates than running on the metal, and the snapshots bought me less than the fragility cost. For a single-GPU box that does one job — serve models — the hypervisor earned nothing, so I removed it. The longer version of that story is in the bare-metal box that runs Atlas.\nPCIe passthrough is still the right answer when you genuinely need the card inside a VM — when the same host must also run unrelated guests, or when isolation between tenants matters more than raw simplicity. It hands the whole physical GPU to one VM at near-native speed. The cost is a tower of configuration that breaks in subtle ways every time something underneath it updates, which is exactly the cost that pushed me off it for a single-purpose box.\nMIG — Multi-Instance GPU — slices one physical data-centre card into several hardware-isolated instances, each with its own slice of compute and memory. It is brilliant for multi-tenant serving where you want guaranteed isolation between workloads. It is also a data-centre-card feature. My consumer 3090 cannot do it, so the question is academic at home but central in the enterprise, where MIG is how you stop one noisy tenant starving another.\nOn containers: every portable service in the lab runs in Docker — reproducible, defined in compose, disposable — and that is the right unit for an AI service. The one deliberate exception is Ollama itself, which runs natively on the GPU box so it talks to the driver with nothing in the way. The lazy take is \u0026ldquo;containerise everything\u0026rdquo;; the honest answer is that the model runtime earns its place on bare metal for the same reason the box does — it is faster and simpler when one process owns one card — while everything around it stays in a container. That split is a recurring theme in lessons from building a Docker homelab.\nPower and cooling: the constraint nobody costs This is the section people skip, and it is the one that has changed my thinking most.\nA 3090 under sustained inference pulls real power — comfortably 300+ watts, the host around it adding more. Run that around the clock and it is a continuous load that shows up on the electricity bill and as heat in the room. Heat is not a metaphor; a GPU dumping a few hundred watts into a small space needs that heat moved, and \u0026ldquo;the room got uncomfortably warm\u0026rdquo; is a genuine engineering signal that you are under-provisioned on cooling.\nWhich leads to the metric that actually matters: cost per token, and where you pay it. Cloud inference bundles power, cooling, and amortised hardware into a per-token price. Running locally, you pay for the card once and then pay for every watt-hour forever. At low utilisation the cloud is cheaper because you are not paying to keep idle silicon warm. At high, steady utilisation, owned hardware wins because you have amortised the capital and you are only buying electricity. The crossover point depends entirely on your duty cycle, and almost nobody calculates it before deciding.\nCost-per-token sanity check (illustrative) Local: (card cost / lifetime tokens) + (watts × hours × £/kWh) / tokens Cloud: £ per million tokens, all-in, zero capital Low duty cycle → cloud wins (you stop paying when idle) High duty cycle → local wins (capital amortised, only power left) There is a pleasing symmetry here with my home energy work: the same time-of-use tariff that makes me shift the battery charge into cheap half-hours makes me think about scheduling heavy local inference and fine-tuning into those same windows. Power-aware compute is not a data-centre-only idea. It is just more visible when the meter is yours.\nGovernance: who can call what, with whose data An AI system that anyone can call with any data is not a capability, it is a liability. Governance is infrastructure, not paperwork.\nThree questions decide the design. Data residency — where is the data allowed to be processed? This is frequently the deciding factor between local and cloud, and it is the single biggest reason a regulated customer cannot \u0026ldquo;just call an API\u0026rdquo;. If the data may not leave the tenant, the inference comes to the data. Access — who, or which service, is permitted to call which model, and with what scope? Treat the model endpoint like any other privileged API: authenticated, authorised, rate-limited, not an open port on the lab network. Audit — every call logged with who, when, what prompt, what context was retrieved, and what came back. Without that you cannot answer the questions that always eventually get asked.\nThis is where the enterprise lesson and the home lab meet. The Microsoft 365 health check work taught me that the report is only trusted if you can show exactly what was queried and why. The same discipline applies to my own assistant: the retrieval is logged, the tool calls go through n8n where they are recorded, and the model only sees what it is allowed to see. Governance you build in from the start is a feature. Governance you bolt on after an incident is a remediation project.\nWhat I got wrong, and would not repeat I under-sized storage latency first time round and ran models off a slow disk, then spent weeks blaming the GPU for slow model switches that were entirely an I/O problem. Match the medium to the access pattern early.\nI also under-estimated the KV cache badly. I sized a model to fit comfortably, then watched it OOM the moment I gave it a long context and a couple of concurrent users, because I had budgeted for weights and forgotten that context costs memory too. Now I size for weights plus worst-case cache, always.\nAnd I ignored power for far too long, treating the electricity as free until the bill and the room temperature both disagreed. The home lab is honest in a way the cloud is not — you feel every watt — and that feedback made me a better architect for the day job.\nWhere this goes next The roadmap is concrete. Finish the VLAN segmentation so the lab network is properly isolated from trust and IoT, which closes the network side of governance. Stand up a small local model registry so model pulls are cached internally rather than re-fetched. Add proper GPU metrics into the existing Prometheus and Grafana so VRAM headroom, power draw, and tokens-per-second are graphed, not guessed — you cannot manage a constraint you do not measure. And formalise the cost-per-token model into a real spreadsheet with my actual duty cycle and tariff, so the local-versus-cloud decision for each workload is a number, not a vibe.\nLonger term, a second GPU is tempting, but only if a workload genuinely needs the pooled VRAM. Buying compute I have no use for is the mistake I tell customers not to make.\nClosing thought The cloud did not delete infrastructure. It abstracted it, charged you for it, and let you forget it was there. AI workloads pull it straight back into view, because they are heavy, hungry, latency-sensitive, and bound by where data is allowed to live. The architects who do well in this era are not the ones who learned to call an API. They are the ones who still understand what the API is standing on.\nDesign the substrate deliberately. Buy memory, match storage to access patterns, keep interactive inference close, respect the watts, govern from day one. Do that and the model becomes what it should always have been — the easy part.\n","date":"2026-06-25T00:00:00Z","permalink":"/infrastructure/designing-infrastructure-for-ai-workloads/","title":"Designing Infrastructure for AI Workloads"},{"content":"From Proposal to Production I write proposals for a living, and then I have to live with them. That second half is the part most people in my trade quietly skip, and it has taught me the most. A proposal is a hopeful artefact. It describes a system that does not exist yet, sized against untested assumptions, priced against a scope that will move, and signed by people who will not be in the room when it gets built.\nThe gap between what gets sold and what gets built is where projects go to suffer. I have been on both sides of it — the architect who drew the optimistic diagram in presales, and the architect who had to make that diagram real eighteen months later with a smaller budget and a constraint nobody mentioned. This article is about that arc: the stages a platform passes through from a first qualification call to a live system somebody owns, and the disciplines that stop intent from leaking out along the way.\nThe short version of my thesis: a proposal is a promise about a system, and the only way to keep it is to carry the reasoning forward, not just the design. The diagram travels fine. The why behind every box is what gets lost, and the why is what you need when reality pushes back.\nThe proposal as a hopeful artefact A proposal is written to win. That is its job, and there is nothing dishonest about it — but it shapes the document in ways that come back to bite. It is written before discovery is complete, often before you have seen inside the customer\u0026rsquo;s environment at all. It assumes the network is healthy, the AD is clean, the change process is sane, the application vendor will cooperate, and the budget is real. Every one of those is a small loan against delivery, and delivery pays the interest.\nThe decisions made in presales are the ones that bite hardest, because they are made fastest and documented least. Somebody on a call says \u0026ldquo;we\u0026rsquo;ll use the existing SQL cluster\u0026rdquo; and it goes into the design as a fact. Six months later that cluster turns out to be at end of life, owned by a different team, and not licensed for the new workload. The decision was never wrong on the day — it was made with the information available, under time pressure, to keep the proposal moving. The failure is that the reasoning never travelled with it. Nobody recorded \u0026ldquo;we assumed the SQL cluster because the customer said it had headroom; this needs validating in discovery.\u0026rdquo; So the assumption hardened into a commitment without anyone choosing to commit — and a decision you cannot reconstruct the reasoning for is one you will re-litigate badly, under pressure, at the worst possible moment.\nThe hand-off chasm Here is the structural problem, and it is structural, not personal. In most organisations the people who sell and scope a platform are not the people who build and operate it. Presales qualifies, designs at a high level, and writes the proposal; delivery picks it up after the contract is signed; operations inherits it after go-live. Each hand-off is a cliff edge, and context falls off every one of them.\nThe presales architect knows things that never make it into the document: which requirements were firm and which were the customer \u0026ldquo;thinking out loud\u0026rdquo;, why a product was chosen over the obvious alternative, what the customer is really worried about underneath the stated requirement, which sizing numbers are confident and which are guesses. That tacit knowledge is the most valuable thing in the engagement, and the standard process throws most of it away at the first hand-off.\nflowchart LR Q[Qualify] --\u003e D[Discovery] D --\u003e HLD[High level design] HLD --\u003e P[Proposal and SoW] P -.risky handover.-\u003e DD[Detailed design] DD --\u003e B[Build] B --\u003e T[Test] T --\u003e M[Migrate and cutover] M -.risky handover.-\u003e H[Handover] H --\u003e O[Operate] O -.feedback.-\u003e QThe two dotted lines are where I have watched the most damage happen. The first, proposal to detailed design, is the sales-to-delivery chasm: the architect who understood the intent is reassigned to the next bid, and a delivery engineer inherits a document and a set of commitments without the reasoning behind either. The second, cutover to handover, is the delivery-to-operations chasm: a working system thrown over a wall to a team who were not consulted on any of the decisions they now have to live with at 2am. The dotted feedback line from Operate back to Qualify is, in most organisations, the one that does not exist at all — what operations learns never reaches the people writing the next proposal, so the same hopeful assumptions get made again. Projects do not usually fail at the build; they fail at the joins.\nWhat actually changes at each stage The stages look linear on the diagram. They are not. Each one is where a particular kind of reality arrives.\nQualification is where you decide whether to be here at all. I have written about this discipline in building an AI consultancy toolkit — the most valuable qualification question is often \u0026ldquo;should we walk away?\u0026rdquo; A badly qualified engagement poisons everything downstream: you spend the whole delivery fighting a misalignment you should have caught on the first call.\nDiscovery is where the proposal\u0026rsquo;s assumptions meet the environment, and where you find the undocumented dependency, the firewall rule nobody can explain, the \u0026ldquo;temporary\u0026rdquo; integration that has been load-bearing for four years. Everyone wants to rush it because the customer wants a number, but it is the stage that determines whether the number is real. Every hour cut from discovery is repaid with interest in the build.\nHigh-level design and proposal is where ideal meets budget. The architecture I would draw with no constraints is never the one that gets signed — the customer has a number in their head, a renewal date driving urgency, and an incumbent system that must keep running throughout. The discipline is to record which compromises you made, so that when someone asks in eighteen months \u0026ldquo;why didn\u0026rsquo;t we just do it properly\u0026rdquo;, the answer exists.\nDetailed design is where the hand-wave becomes a buildable specification and the gaps in the proposal become visible. \u0026ldquo;We\u0026rsquo;ll integrate with their identity provider\u0026rdquo; is a line in a proposal; in detailed design it becomes forty decisions about claims, conditional access, fallback authentication, and what happens when the IdP is down. This is the stage that most exposes the cost of a shallow proposal.\nBuild is where the constraints nobody mentioned arrive in person — the change freeze over the customer\u0026rsquo;s financial year-end, the application vendor who needs six weeks\u0026rsquo; notice for anything, the one engineer who knows the legacy system and is on long-term sick. None were in the SoW; all are now your problem.\nTest, migrate, cutover is where \u0026ldquo;it works in the lab\u0026rdquo; meets real users and real data. Migration is almost always harder than the build, because the build is greenfield and clean while migration drags a live, messy, in-use system across without dropping anything. Scope creep concentrates here, because this is where the customer finally sees the thing and realises what they really wanted.\nHandover and operate is where the project ends and the system\u0026rsquo;s real life begins — and it is the stage given the least time and respect. I will come back to it, because \u0026ldquo;production\u0026rdquo; is the word everyone uses and almost nobody defines.\nWhy the reasoning has to travel — ADRs and traceability The single discipline that has changed how I deliver is recording decisions as decisions, with their reasoning, in a durable place that travels across every hand-off — Architecture Decision Records. I went deep on the knowledge side of this in building knowledge instead of documents, but the delivery angle is specific: an ADR is how a decision survives the people who made it.\nA design document tells you what the system is. It does not tell you why, and the why is what you need when reality pushes back. When a delivery engineer hits the SQL cluster problem from earlier, a design document says \u0026ldquo;uses existing SQL cluster\u0026rdquo; and leaves them stranded. An ADR says \u0026ldquo;we chose the existing cluster over a new instance to avoid licence cost and because the customer stated it had headroom; this assumes it is supported through 2027 and licensed for this workload; if either is false, revisit and price a dedicated instance.\u0026rdquo; Now the engineer knows what assumption broke, what the alternative was, and what it costs to change course. The decision is recoverable.\nI keep ADRs as plain Markdown in the project repository, next to the code and the runbooks, version-controlled, so the decision log is part of the deliverable and not a slide deck that dies in someone\u0026rsquo;s inbox. A template I have settled on:\n# ADR-014: Identity provider for the new platform - Status: Accepted - Date: 2026-04-18 - Deciders: K Haynes (architect), customer infra lead - Supersedes: ADR-009 ## Context The platform needs SSO. The customer runs Entra ID for M365 but also has an on-prem AD FS estate still serving two legacy apps. ## Decision Use Entra ID as the sole IdP for the new platform. Do not extend AD FS. Legacy apps stay on AD FS until their own migration. ## Consequences - Positive: one identity plane, Conditional Access applies natively, no AD FS lifecycle to own for the new system. - Negative: two legacy apps remain on AD FS short-term; a second migration is implied and must be tracked as a dependency. - Assumption: customer\u0026#39;s Entra P2 licensing covers all platform users. If false, revisit — Conditional Access design depends on it. The format is unremarkable; the discipline is stating the assumption out loud with the trigger for revisiting it, and keeping the \u0026ldquo;supersedes\u0026rdquo; line. Decisions change, and a log that only shows the current state hides the fact that you considered and rejected something for a reason that might still apply.\nUnderneath the ADRs sits the discipline that ties the whole arc together: traceability. Every requirement should trace to a design decision, every decision to something built, and every built thing back to the requirement that justified it. When a customer asks \u0026ldquo;why is this here?\u0026rdquo; in the operate phase, I want a thread that runs requirement to ADR to build to running config. When scope creep arrives — and it always arrives — that thread lets me say \u0026ldquo;that wasn\u0026rsquo;t in the requirements, here\u0026rsquo;s the one we agreed, let\u0026rsquo;s decide together whether to add it\u0026rdquo; instead of arguing from memory.\nRepeatable patterns so every delivery isn\u0026rsquo;t from scratch If every engagement starts from a blank page, the hand-off chasm gets wider, because there is nothing shared for context to attach to. The antidote is repeatable patterns and automation, so that most of a delivery is assembling known-good parts rather than inventing everything fresh.\nThis is the through-line of building repeatable customer health checks. A health check is a reusable framework that turns discovery from a bespoke investigation into a repeatable assessment — the same queries, posture checks, and prioritised report run against every tenant. It makes discovery faster and more honest, because you are measuring the environment rather than relying on what the customer tells you about it, and it makes the operate phase easier, because the same tooling that assessed the environment before the build can monitor it after.\nThe same logic applies to the designs themselves. I argued in modern Citrix architecture for treating Citrix designs as reference architectures rather than bespoke snowflakes, and Citrix is a good worked example because it has a small number of recurring shapes — control plane in the cloud, VDAs next to the data, MCS image pipeline, Entra ID identity. Once you have built that pattern well twice, the third delivery is assembly, not invention: the proposal references the reference architecture, the detailed design inherits its ADRs, the operate phase inherits its runbooks. That is the real argument for automation too — infrastructure as code and configuration in Git are the one form of documentation that cannot drift from reality, so the build is the system and the next engineer can reproduce it exactly.\nStaying involved — not throwing it over the wall The worst thing an architect can do is design something and disappear. I have done it early in my career, and I have inherited other people\u0026rsquo;s over-the-wall designs, and both are miserable. The design that looks elegant in a proposal often has a flaw that only shows up in the build, and if the architect is gone, nobody with the full context is there to make the call. So the delivery engineer either builds the flawed thing faithfully, or improvises a fix that quietly breaks the design\u0026rsquo;s intent — and now the running system and the design document disagree, a slow-burning disaster of its own.\nI stay involved through delivery. Not hovering over every task, but present at the joins — reviewing the detailed design against the proposal\u0026rsquo;s intent, available when the build hits something the design did not anticipate, in the room for cutover, and signing off the handover personally. This is partly self-interest: if I never see my own designs reach production, I never find out which of my assumptions were wrong, and I keep making them. Watching a design behave in build and operations is the only thing that makes the next proposal less hopeful and more grounded.\nIt is also where the awkward conversations live, and the architect is the right person to have them. When scope creeps, somebody has to draw the line between \u0026ldquo;reasonable clarification\u0026rdquo; and \u0026ldquo;new project\u0026rdquo;. When a presales commitment turns out to be undeliverable as written, somebody has to own it rather than letting the delivery team absorb the blame for a decision they did not make. These conversations are uncomfortable because they are honest, and the architect who handled the original design has the standing to have them well. Avoiding them just defers the problem to handover, where it costs more.\nWhat production actually means Here is the line I care about most. \u0026ldquo;It works in the demo\u0026rdquo; is not production. A demo is a system performing under ideal conditions for a friendly audience. Production is a system that survives indifference, neglect, real users doing unexpected things, and the absence of the people who built it.\nA platform is in production when it is operable — somebody who was not on the project can run it from documentation that matches reality. When it is monitored — its health is visible, alerts fire to someone who can act, and a problem is noticed before the customer reports it. When it is owned — a named team has the access, the knowledge, and the mandate to keep it alive. When it is documented — not a 200-page document nobody reads, but the runbooks, the decision log, and the diagrams that let the next person reason about it. And when it is handed over — the operations team has accepted it knowingly, not had it dropped on them.\nThis is exactly the operational readiness I argue is the real determinant of success in why most AI projects fail. Projects rarely fail because the technology did not work. They fail because it worked in the build and was never made operable, never properly owned, never genuinely handed over. The build is the easy 80%; operability is the hard 20% everyone underfunds, and it is the only part the customer actually experiences day to day.\nA system nobody can operate is not finished. It is abandoned with extra steps.\nWhat I have got wrong I have made every mistake in this article, which is why I can write it. I have written confident sizing numbers that were really guesses and watched them become contractual commitments I had to defend at a capacity review that proved me optimistic. I have made a presales decision to reuse existing infrastructure to keep a price competitive, and handed the delivery team a dependency that was rotten. I learned to write the assumption down next to the decision, so the risk travels with it instead of ambushing someone later.\nI have thrown a design over the wall once and told myself the documentation was good enough. It was not, because no documentation captures the dozen small judgements that did not feel worth recording at the time and turned out to be load-bearing. That is the engagement that taught me ADRs are not bureaucracy — they are the only mechanism I have found that makes a decision outlive the person who made it. And I have under-scoped handover more than once, treating it as a formality rather than a phase with its own work, when the truth is the project is not done at go-live — it is done when someone else can keep the system alive without me.\nWhere this goes next The improvement I am working on is making the whole arc traceable as data rather than as prose scattered across tools. Requirements, ADRs, build artefacts, and monitoring all linked, so that the thread from \u0026ldquo;the customer asked for X\u0026rdquo; to \u0026ldquo;this running config delivers X, and here is the alert that tells us if it stops\u0026rdquo; is a query, not an archaeology project. Some of that is tooling around the same Markdown-and-Git knowledge base that runs the rest of my practice; some is discipline no tool will replace.\nThe bigger shift is cultural, and it is the one I push hardest in the future of technical presales: close the loop between operate and qualify. The architect who writes the proposal should hear what the operations team learned, because that is what makes the next proposal honest. Most organisations are structured to prevent exactly this — the bid team and the run team never speak. The ones who fix it write proposals grounded in what their systems actually do in production, not what the slide hoped they would do.\nClosing thought A proposal is a promise about a system that does not exist yet. Keeping that promise is not a single act of building — it is a chain of hand-offs, each one a chance for intent to leak out, and the architect\u0026rsquo;s real job is to be the thread that runs through all of them. The diagram is the easy part. Carrying the why from the first qualification call to a system somebody owns and operates without you — that is the work, and most people in my trade never see it, because they leave before the system has to survive them.\nThe best compliment I get is not \u0026ldquo;great proposal\u0026rdquo;. It is a customer\u0026rsquo;s operations lead, two years later, telling me the platform just quietly works and they always know why it was built the way it was. That is what production means. Everything before it is a hopeful artefact.\n","date":"2026-06-25T00:00:00Z","permalink":"/posts/from-proposal-to-production/","title":"From Proposal to Production"},{"content":"Lessons from Building a Docker Homelab Every homelab starts the same way. You spin up one container to try something, it works, and you forget about it. Then another. Then a database for that one. Then a second thing that talks to the first, on a port you picked because it was free that afternoon. Six months later you have a server doing useful work and absolutely no idea how to rebuild it if the disk dies.\nThat was me. The lab that now runs my AI infrastructure at home — Ollama, Open WebUI, n8n, Home Assistant, the whole spine — did not arrive as a tidy design. It accreted. The most valuable thing I have done in three years of running it was not adding a service. It was going back and making the mess reproducible.\nThis is the operational counterpart to the more architectural infrastructure for AI workloads piece. That one is about what the hardware should be. This one is about how you actually run forty-odd containers on it without losing your mind, and the specific things I got wrong on the way. None of it is theory. All of it cost me an evening at some point.\nThe sprawl problem nobody admits to The honest failure mode of a homelab is not a dramatic outage. It is entropy. You end up with a pile of docker run commands that live nowhere except your shell history and a vague memory. Ports clash, so you start picking arbitrary high numbers and writing them on a sticky note. Two services both want 8080. You \u0026ldquo;temporarily\u0026rdquo; expose a database to the host to debug something and never close it. A container called app_final_v2 is doing something load-bearing and you genuinely cannot remember what.\nIf you cannot rebuild the host from a fresh OS in an afternoon, you do not have a homelab. You have a pet you are afraid of.\nThe thing that makes this insidious is that it works. Sprawl is functional right up until the moment it isn\u0026rsquo;t — a power cut, a failed SSD, a docker system prune you ran while tired. Then you discover that \u0026ldquo;it works\u0026rdquo; and \u0026ldquo;I can reproduce it\u0026rdquo; are completely different properties, and you only ever invested in the first one.\nThe fix is not a tool. It is a decision: the configuration of every service lives in a file, in Git, and the running state is downstream of that file. Everything else here is a consequence of taking that seriously.\nCompose as code, Git as the source of truth I run everything as docker-compose, and every compose file is in a Git repository. Not a backup of the compose file. The compose file itself, edited in place, committed when it changes. The repo is structured one directory per stack, each with its own docker-compose.yml and a .env that is pointedly not committed.\nhomelab/ ├── caddy/ │ ├── docker-compose.yml │ ├── Caddyfile │ └── .env ├── monitoring/ │ ├── docker-compose.yml │ ├── prometheus.yml │ └── .env ├── ai/ │ ├── docker-compose.yml # open-webui (Ollama runs native on the GPU box) │ └── .env └── .gitignore # *.env, acme.json, data/ People ask where Portainer fits, because I do run it. The answer matters. Portainer is for visibility, not authority. It is a lovely window — container status, logs, resource use, a quick restart when I am on my phone. But the moment you start editing stacks inside Portainer\u0026rsquo;s UI, you have created a second source of truth, and the two will drift. I have been burned by exactly this: a change made in the Portainer web editor that existed nowhere in Git, lost the next time I redeployed from the repo. So the rule is firm. Portainer reads. Git writes. If I want a change to persist, it goes in the file and gets committed, and Portainer simply reflects reality.\nThis is the same instinct that made me move this whole site to plain-text, version-controlled content. Configuration that lives only in a UI is configuration you do not really own. The compose files win every time because they are the only artefact that survives the host.\nDocker networking, properly this time Networking is where most homelabs quietly go wrong, because the defaults are forgiving enough to hide the mistakes. Docker drops every container on a default bridge where everything can talk to everything, and you can publish ports to the host with a single line. Both of those conveniences are traps.\nThe first real lesson was to stop publishing ports I did not need to. Every ports: entry is a hole in the host firewall. A database does not need a published port. The only thing that needs to reach Postgres is the application sitting next to it, and they can find each other over a private network by container name. So I create user-defined bridge networks, one per logical stack, and only publish to the host the handful of things that genuinely must be reachable from outside — really just the reverse proxy.\nnetworks: web: # shared with Caddy, the only externally reachable plane external: true internal: # private, never published external: false User-defined bridges give you something the default bridge does not: automatic DNS. Inside a network, postgres resolves to the Postgres container. No IP addresses, no links, no host ports. The app talks to redis:6379 and postgres:5432 directly, on the internal network, and neither of those ports is exposed to the host at all.\nThe other three-quarters of Docker networking confusion comes from not knowing which driver you actually want:\nbridge — the default and the right answer almost always. NAT behind the host, isolated, DNS by name. Use it. host — the container shares the host\u0026rsquo;s network stack directly. Fast, no isolation, and it ignores ports: entirely. I reach for it rarely — occasionally for something doing service discovery or needing the real client IP — and I treat it as a smell. macvlan — gives a container its own MAC and IP on the physical LAN, as if it were a separate machine on the network. Genuinely useful for something like a Pi-hole or a service that wants to look like real hardware, but it bypasses the host firewall and does not talk to the host easily. Powerful and sharp. I use it deliberately, never casually. The mental model I wish I had started with: the published-port trap is thinking that exposing a port is how containers communicate. It is not. Containers on the same user-defined network already talk to each other freely. Publishing a port is only about reaching the host from outside. Once that clicked, the set of exposed services shrank dramatically, and the database came off the public network where it had no business being.\nArchitecture: how it actually fits together Here is the shape of the network now. One ingress point, TLS terminated once, a public-facing plane shared with the proxy, and private planes that the outside world cannot see at all.\nflowchart TD Internet[Internet] --\u003e Router[Router and Firewall] Router --\u003e|443 only| Caddy[Caddy reverse proxy] subgraph web [web network exposed] Caddy --\u003e WebUI[Open WebUI] Caddy --\u003e N8N[n8n] Caddy --\u003e Grafana[Grafana] Caddy --\u003e Kuma[Uptime Kuma] end subgraph internal [internal network private] N8N --\u003e Postgres[(Postgres)] WebUI --\u003e Ollama[Ollama on GPU box, bare metal] Grafana --\u003e Prometheus[Prometheus] Prometheus --\u003e Cadvisor[cAdvisor] Prometheus --\u003e NodeExp[node-exporter] end Portainer[Portainer] -.read only view.-\u003e web Portainer -.read only view.-\u003e internalThe important property of that picture is the dotted line for the databases. Postgres sits on the internal network only, and nothing from the internet can reach it. Ollama is not in Docker at all — it runs natively on the separate GPU box, reachable only across the lab LAN — so the containers call it over the network rather than hosting it. Caddy is the single front door, and the only thing the router forwards is 443.\nWhy Caddy, and why I left Nginx Proxy Manager I started on Nginx Proxy Manager. It is a fine tool and a kind on-ramp — a clean UI, click to add a proxy host, click to get a Let\u0026rsquo;s Encrypt certificate. For a handful of services it is genuinely pleasant.\nThe problem is the same one as Portainer\u0026rsquo;s stack editor, only worse. NPM\u0026rsquo;s configuration lives in its own database, behind a UI. It is not in Git. Adding a service means a human clicking through a form, which means it is not reproducible and not reviewable. As the lab grew past a dozen services, the gap between \u0026ldquo;what is proxied\u0026rdquo; and \u0026ldquo;what is in my repo\u0026rdquo; became the single biggest piece of undocumented state I owned.\nCaddy solves this by putting the entire routing intent in one plain-text file — the Caddyfile — committed to the same repo as everything else. There is no UI and no database. The file is the source of truth, and Caddy\u0026rsquo;s headline feature is that it obtains and renews Let\u0026rsquo;s Encrypt certificates automatically, with no ACME plumbing to wire up. A service is exposed when it has a block in the Caddyfile, and not before.\n# Caddyfile — the routing intent for the whole lab, in Git chat.lab.example.com { reverse_proxy open-webui:8080 } That block is the whole story for a service: a hostname, and the container and port to send it to. Caddy reaches open-webui by name over the shared web network — the container publishes no host ports of its own — terminates TLS, and keeps the certificate valid without being asked.\nThe service compose, then, carries no proxy configuration at all. It just joins the web network so Caddy can reach it:\nservices: open-webui: image: ghcr.io/open-webui/open-webui:0.5.20 container_name: open-webui restart: unless-stopped environment: - OLLAMA_BASE_URL=http://gpu-box.lab.internal:11434 # native Ollama, over the LAN networks: - web - internal # note: no ports: published. Caddy reaches it over the web network. networks: web: external: true internal: external: true Caddy itself is the only container that publishes ports to the host. It mounts the Caddyfile read-only and persists its certificates in a named volume:\nservices: caddy: image: caddy:2.8 restart: unless-stopped ports: - \u0026#34;80:80\u0026#34; - \u0026#34;443:443\u0026#34; environment: - CF_API_TOKEN=${CF_API_TOKEN} # for the DNS challenge on internal-only hosts volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro - caddy_data:/data # certificates and ACME state - caddy_config:/config networks: - web volumes: caddy_data: caddy_config: The thing that matters most is that nothing is exposed unless I write a block for it. With NPM, everything became reachable the moment I clicked it in; with a Caddyfile, the proxied surface is exactly the set of hostnames in one reviewable file, and a service with no block is simply invisible from outside. That is the only safe default — nothing is published unless I deliberately declare it — and it lives in version control where I can diff it. TLS everywhere, automatically, and the routing intent in a single committed file.\nA note on the Docker socket: the label-based proxies need to mount it so they can watch containers come and go, and even read-only that is a real trust decision, because a container that can read the socket learns a great deal about the host. Caddy reading a static Caddyfile needs no such access — it never touches the socket — which is one fewer privileged mount in the lab and one fewer thing to reason about when I think about blast radius.\nThe .env and secrets discipline The networking and proxy work is wasted if your secrets are sitting in the compose file in Git. They were, for an embarrassingly long time. Database passwords, the Cloudflare API token, n8n\u0026rsquo;s encryption key — all committed in plain text because it was easier.\nThe discipline now is absolute. Every secret lives in a .env file next to its compose file, and .env is in .gitignore before the first commit. The compose file references variables; it never contains values:\nenvironment: - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY} # .env — never committed, backed up separately and encrypted POSTGRES_PASSWORD=... N8N_ENCRYPTION_KEY=... CF_DNS_API_TOKEN=... I keep a committed .env.example with the keys and no values, so the repo documents what a stack needs without leaking anything. The real .env files are backed up encrypted, separately from the code, because losing n8n\u0026rsquo;s encryption key means losing every stored credential it holds. If you take one thing from this section: run git log -p against your old compose files and check what is in your history. Secrets in Git history are still secrets in Git, even after you delete them from the current file. I had to rotate a few.\nMonitoring you actually act on There are two distinct questions monitoring answers, and conflating them is why so many homelab dashboards are beautiful and useless.\nThe first is \u0026ldquo;is it up?\u0026rdquo;. For that I run Uptime Kuma. It pings every service, shows a wall of green, and shouts at me when something goes red. It is simple, it is reliable, and it is the thing I actually look at. The second question is \u0026ldquo;how is it behaving over time?\u0026rdquo; — CPU, memory, container restarts, disk filling up. For that I run Prometheus scraping cAdvisor (per-container metrics) and node-exporter (host metrics), with Grafana on top for dashboards.\nThe honest lesson here is about alerting, not collection. My first instinct was to alert on everything, and I quickly trained myself to ignore the alerts, which is worse than having none. An alert that does not change your behaviour is just noise wearing a uniform. So I cut it down hard, to a handful of things I will genuinely get up and fix: a service down for more than a few minutes, host disk above 85%, memory pressure that will start killing containers, the backup job failing. Everything else is a dashboard I look at when curious, not a notification that interrupts dinner. Fewer, sharper alerts that you act on every time beat a comprehensive system you have learned to dismiss.\nUpdates: where I most changed my mind I started with Watchtower pointed at everything, set to pull and recreate any container with a newer image automatically. It felt responsible. It was not.\nThe problem is that \u0026ldquo;latest\u0026rdquo; moves under you, and not always in the direction you want. One morning a service I depended on had silently jumped a major version overnight, its config schema had changed, and it would not start. Nothing in my repo had changed. I had not touched it. An automatic update had broken a working system while I slept, and I spent the morning working out which of forty containers had quietly changed.\nSo I stopped auto-updating everything, and I would not go back. The policy now:\nPin tags. No service runs :latest. Every image is pinned to a specific version — caddy:2.8, open-webui:0.5.20. The running version is therefore recorded in Git, which means I can see exactly what changed and when. Update deliberately. Updating a service is a commit: I bump the tag in the compose file, redeploy that one stack, watch it come up, and move on. If it breaks, git revert and I am back to a known-good version in seconds. Watchtower still runs, but only in notify mode. It tells me an update is available; it does not apply it. The decision stays mine. Pinned tags and deliberate updates are slightly more work each week and dramatically less work the one week something would have broken. The lab is more boring now. Boring is the goal.\nAlongside that sits plain image hygiene. Pinning versions means old images pile up, so a periodic docker image prune and an occasional docker system prune (carefully — never with --volumes unless I mean it) keeps the disk from filling. Disk full is one of the few things that takes the whole host down, which is exactly why it is on the short list of things I alert on.\nBackups, and the only thing that actually matters Here is the realisation that reorganised how I think about the whole lab. Containers are disposable. Volumes are not. A container is just a running copy of an image I can pull again in thirty seconds. The compose file that defines it is in Git. Neither of those needs backing up in any meaningful sense — they are already reproducible. The thing that is irreplaceable is the data in the volumes: the Postgres databases, n8n\u0026rsquo;s workflows, Home Assistant\u0026rsquo;s history, Grafana\u0026rsquo;s dashboards, Open WebUI\u0026rsquo;s chats.\nSo my backups target volumes, and they follow 3-2-1: three copies of the data, on two different media, with one off-site. In practice that is the live volumes on the server, a nightly snapshot to the NAS, and an encrypted copy pushed off-site. The backup I care about is a pg_dump of the databases and a tarball of the named volumes, run nightly, with the job itself monitored — a failed backup that fails silently is the same as no backup, and I learned that the way everyone does, by needing one that was not there.\nThe test that matters is not \u0026ldquo;does the backup run\u0026rdquo;. It is \u0026ldquo;can I restore it\u0026rdquo;. I have done it from cold deliberately, on a spare box, and proving I could rebuild the lab from Git plus the volume backups was the moment it stopped being a pet I was afraid of.\nWhat I got wrong A short, honest list, because the mistakes taught me more than the successes:\nlatest tags everywhere. Covered above. The root of more 7am debugging than anything else. Secrets committed to Git. Also covered, also rotated in a hurry. Set up .gitignore before the first commit, not after the first leak. The single host as a SPOF. Everything runs on one bare-metal Ubuntu box. Backups mean I can rebuild, but I cannot fail over. That is a deliberate, eyes-open trade-off for a homelab, but I do not pretend it is resilient. The load-bearing container with no documentation. I had a small custom container doing something important and undocumented. When it broke I had to reverse-engineer my own work. Now anything non-obvious gets a comment in the compose file and a line in the repo\u0026rsquo;s README. The glue scripts that hold this together get the same treatment — part of why I argue every infrastructure engineer should learn Python is so that glue is readable and maintainable rather than a black box. Where this goes next The roadmap is concrete rather than aspirational. The first job is finishing the VLAN segmentation I have been slowly doing — trust, IoT and lab on separate networks, so an IoT device\u0026rsquo;s compromise cannot reach the database plane. The flat network is the oldest debt in the lab.\nSecond is removing the single point of failure for the always-on services. The mini-PC fleet is the obvious target: a small Docker Swarm or a lightweight k3s across two or three N100 boxes so the critical services — DNS, the proxy, monitoring — survive one host dying. The AI workloads stay on the GPU box because they are pinned to that hardware anyway.\nThird is treating deployment itself as code: a CI pipeline that, on a push to the homelab repo, validates the compose files and redeploys the changed stacks, so the gap between \u0026ldquo;committed\u0026rdquo; and \u0026ldquo;running\u0026rdquo; closes to zero. That makes the lab a genuine practice ground for the patterns I use professionally — the same reason I treat it as a home lab as a learning platform rather than just a place to run services.\nClosing thought The homelab taught me something I keep carrying into client work: the running system is never the artefact worth protecting. The artefact is the description of the system — the compose files, the labels, the .env.example, the documented mistakes. Get that right and the running system becomes a cheap, regenerable consequence. Get it wrong and you are one dead SSD away from archaeology.\nI did not learn that by reading it. I learned it by getting it wrong, on my own hardware, on a quiet evening when something I could not reproduce stopped working. That is the real value of a homelab: it lets you make every one of these mistakes where the only thing at risk is your weekend, not someone\u0026rsquo;s production tenant.\n","date":"2026-06-25T00:00:00Z","permalink":"/infrastructure/lessons-from-building-a-docker-homelab/","title":"Lessons from Building a Docker Homelab"},{"content":"Microsoft 365 AI Health Check A Microsoft 365 health check, done properly, is a few days of tedious clicking followed by an evening of writing it up. I have done enough of them to resent both halves.\nThe clicking is the same every time. You log into the admin centre, the Entra portal, the Security portal, the compliance portal, the Exchange admin centre, and you copy posture out of a dozen blades into a spreadsheet. The writing-up is where it gets worse, because two engineers looking at the same tenant will produce two different reports, with different priorities, different tone, and different things quietly forgotten. The output quality depends entirely on who happened to pick up the job and how tired they were when they wrote it.\nThis bothered me for a long time before I did anything about it. A health check is, fundamentally, a repeatable thing. The facts come from an API. The judgement is mostly pattern matching against the same set of known-good positions. So I built an assistant that collects the facts deterministically and writes the report consistently, running entirely on infrastructure I own. This article is how it works, what I got wrong, and where it goes next. It is a concrete instance of the broader argument I make in building repeatable customer health checks — that a health check is a product, not a craft.\nThe problem with how we do this The honest version of a manual M365 health check is that it is inconsistent, slow, and unauditable.\nInconsistent, because there is no fixed checklist that survives contact with a real tenant. Everyone has their own mental list. Mine is good. My colleague\u0026rsquo;s is good in different places. Neither is written down in a way that guarantees the same coverage twice. Some engineers obsess over Conditional Access and skip licensing waste entirely. Some produce a beautiful Secure Score narrative and never look at who holds Global Administrator.\nSlow, because the data collection is manual and the data is spread across portals that genuinely do not want to talk to each other. A thorough check is two to three days, most of which is not thinking, it is navigation and copy-paste.\nUnauditable, because the report is prose written from memory and a spreadsheet. If a customer asks \u0026ldquo;how did you conclude our MFA coverage was 71%?\u0026rdquo; three months later, the honest answer is often \u0026ldquo;I counted it in my head at the time.\u0026rdquo; That is not good enough when the report drives remediation spend.\nA finding nobody can reproduce is an opinion wearing a suit.\nI wanted findings that were reproducible, coverage that was guaranteed, and a report whose quality did not depend on my mood. That meant separating the two things a health check actually is: gathering facts, and reasoning about them.\nDesign decisions The central decision, the one everything else hangs off, is the collect-then-reason split. The system collects raw tenant data deterministically, normalises it, runs deterministic rules over it to produce facts, and only then hands those facts to an LLM to turn into a prioritised narrative. The LLM never touches the tenant. It never calls Graph. It never decides what is true. It writes.\nThis matters because of everything I learned writing why most AI projects fail: a language model asked to both gather and judge will confidently invent the gathering. If you let it near the API and ask it \u0026ldquo;is MFA enforced?\u0026rdquo;, it will reason its way to a plausible answer rather than a correct one. The fix is to never give it that job. Facts are computed in code. The model is a writer with a locked source of truth.\nThe second decision was read-only, least-privilege access. I created a dedicated app registration in Entra ID using client-credentials auth — no user, no delegated session, no interactive sign-in. It holds only the application Graph scopes it genuinely needs, all of them read-only: Directory.Read.All, Policy.Read.All, Reports.Read.All, SecurityEvents.Read.All, RoleManagement.Read.Directory, Sites.Read.All. There is no write scope anywhere in the consent. The worst this credential can do, if it leaks, is read configuration. That is a property I can put in writing to a customer\u0026rsquo;s security team, and it is the difference between them saying yes and saying no.\nThe third decision was n8n as the orchestrator rather than a monolithic script. n8n is already the automation spine of my lab — it is the same tool I lean on across nearly everything, for the reasons in why every infrastructure engineer should learn Python (n8n is glue, Python is the muscle inside the glue). Using it here gave me retry handling, credential storage, scheduling, and a visual map of the pipeline for free. The collectors are n8n HTTP nodes with Python in Code nodes where the logic gets gnarly.\nThe fourth decision was the model stays local. Tenant configuration is sensitive. Nothing about this tenant — not user counts, not policy names, not admin lists — goes to a public AI service. The report writer is a local model on Ollama, the same runtime that powers Project Atlas. I will come back to this because the data-handling story is the part customers care about most.\nWhat it actually inspects The deterministic collectors gather a fixed set of areas every run, which is how I guarantee coverage. The list is opinionated and it is the same every time:\nIdentity and Conditional Access — every CA policy, its state (on, off, report-only), its assignments and exclusions. Excluded users are flagged because exclusions are where good policy goes to die. MFA coverage — registration and capability pulled from the authentication methods and registration reports, expressed as a real percentage with the denominator shown. Licensing and assignment waste — assigned versus enabled SKUs, unused premium licences, P2 features being paid for but not used, accounts holding licences that have not signed in for 90 days. Secure Score — current score, the control profiles, and the highest-impact unactioned controls. Sharing posture — Exchange external forwarding and mail flow rules, SharePoint and OneDrive external sharing settings, Teams guest and external access. Privileged roles — who holds the directory roles that matter, how many Global Admins exist, whether they have MFA, and whether PIM is in use or the roles are permanently assigned. Each of these produces structured facts, not prose. \u0026ldquo;9 Global Administrators, 2 without registered MFA, none under PIM\u0026rdquo; is a fact. The model\u0026rsquo;s job is to know that this is bad, explain why, and rank it.\nHow it fits together The architecture is a straight line with one deliberate firebreak in the middle. Data flows from Graph through the collectors into normalised JSON, the rules engine turns that JSON into graded findings, and only the findings — never the raw tenant dump — cross into the LLM.\nsequenceDiagram participant n8n as n8n Orchestrator participant Graph as Microsoft Graph participant Rules as Rules Engine participant LLM as Local LLM via Ollama participant Out as Markdown and PDF n8n-\u003e\u003eGraph: Client credentials token request Graph--\u003e\u003en8n: Access token loop Each inspection area n8n-\u003e\u003eGraph: GET config with paging Graph--\u003e\u003en8n: Page of results end n8n-\u003e\u003en8n: Normalise into canonical JSON n8n-\u003e\u003eRules: Evaluate deterministic checks Rules--\u003e\u003en8n: Graded findings with severity n8n-\u003e\u003eLLM: Findings JSON plus grounding prompt LLM--\u003e\u003en8n: Prioritised narrative report n8n-\u003e\u003eOut: Render template to Markdown and PDFThe firebreak is the line between the rules engine and the LLM. To the left of it, everything is deterministic and reproducible — same tenant, same day, same findings. To the right, the model does language, not facts.\nAuthentication is the dull, important part. The collector starts by exchanging the app credentials for a token:\ncurl -s -X POST \\ \u0026#34;https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token\u0026#34; \\ -d \u0026#34;client_id=${CLIENT_ID}\u0026#34; \\ -d \u0026#34;client_secret=${CLIENT_SECRET}\u0026#34; \\ -d \u0026#34;scope=https://graph.microsoft.com/.default\u0026#34; \\ -d \u0026#34;grant_type=client_credentials\u0026#34; Then the collectors page through Graph. The key detail, and the one that bit me, is that Graph hands you data in pages via @odata.nextLink, and you have to follow the chain or you silently under-report. A naive single call to the CA policies endpoint looks like it works on a small tenant and quietly truncates on a large one. The collector loop is boring on purpose:\n# n8n Code node: page every Graph collection, never trust the first response import requests def collect_all(url, token): items, headers = [], {\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {token}\u0026#34;} while url: r = requests.get(url, headers=headers, timeout=30) if r.status_code == 429: # throttled wait = int(r.headers.get(\u0026#34;Retry-After\u0026#34;, 10)) time.sleep(wait) continue r.raise_for_status() body = r.json() items.extend(body.get(\u0026#34;value\u0026#34;, [])) url = body.get(\u0026#34;@odata.nextLink\u0026#34;) # follow the chain return items ca_policies = collect_all( \u0026#34;https://graph.microsoft.com/v1.0/identity/conditionalAccess/policies\u0026#34;, token, ) Once everything is collected and normalised, the rules engine grades it. A rule is a tiny pure function: it takes the canonical JSON and emits a finding with an id, a severity, the evidence, and the metric it measured. Nothing here is clever. That is the point — clever is where reproducibility goes to die.\n{ \u0026#34;id\u0026#34;: \u0026#34;ROLE-001\u0026#34;, \u0026#34;area\u0026#34;: \u0026#34;Privileged Roles\u0026#34;, \u0026#34;severity\u0026#34;: \u0026#34;high\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;Excessive Global Administrators\u0026#34;, \u0026#34;evidence\u0026#34;: { \u0026#34;global_admins\u0026#34;: 9, \u0026#34;without_mfa\u0026#34;: 2, \u0026#34;pim_managed\u0026#34;: 0 }, \u0026#34;metric\u0026#34;: \u0026#34;9 Global Administrators, 2 without MFA, 0 under PIM\u0026#34;, \u0026#34;recommended_target\u0026#34;: \u0026#34;fewer than 5, all MFA-enforced, all PIM-eligible\u0026#34; } The model receives an array of these objects. It does not receive the tenant. It receives facts that have already been judged true and graded, and its job is to write them up well and in the right order.\nStopping the model from lying This is the part I spent the most time on, because a language model writing a security report is a liability unless you cage it carefully. Left to its own devices it will smooth over gaps, invent plausible-sounding posture to fill a section, and assert things the data never said. Confidently. Every time.\nThe grounding system prompt does three things: it tells the model it is a writer not an investigator, it forbids any claim not present in the findings, and it gives it a fixed template so the structure is never its decision. A sketch of it:\nYou are writing a Microsoft 365 health check report. You will be given a JSON array of FINDINGS. Each finding is already verified and graded. These are your ONLY source of truth. Rules: - Never state a fact that is not present in the findings JSON. - Never invent metrics, counts, policy names, or percentages. - If an area has no findings, write exactly: \u0026#34;No issues detected in this area.\u0026#34; Do not speculate about why. - Order the report by severity: critical, then high, then medium, low. - For each finding: state the issue, why it matters, the evidence verbatim from the JSON, and the recommended action. - Do not soften severities. Do not add reassurance the data does not support. Output Markdown using the section headings provided. Nothing else. The single most effective line in that prompt is the instruction to quote evidence verbatim from the JSON. It forces the model to anchor every sentence to a value it was actually given, and it makes hallucination visibly inconsistent with the rendered numbers, so it shows up immediately in review. The \u0026ldquo;No issues detected\u0026rdquo; escape hatch matters just as much — without an explicit thing to say about an empty area, the model will fill the silence, and what it fills it with is fiction.\nI also learned to stop trusting free-form output and lean on templating. The model writes the prose for each finding, but the document skeleton, the executive summary table, the severity counts and the cover page are rendered from the findings JSON by a deterministic template. The model never decides how many high-severity issues there are. It is told, and the count is computed in code. Templating beats free-form every time you can get away with it.\nWhat I got wrong Graph pagination cost me a real finding once. Early on, on a large tenant, my CA collection truncated at the first page and the report cheerfully concluded the tenant had far fewer policies than it did. Nobody was harmed because I caught it in review, but it taught me to never trust a single Graph response and to assert expected counts where I can. Always follow @odata.nextLink.\nThrottling is not an edge case, it is Tuesday. Run enough collectors in parallel against a big tenant and Graph will start returning 429 with a Retry-After. My first version did not honour it and the run failed halfway. Now every call respects Retry-After and the collectors are deliberately a bit patient. A health check that takes four minutes instead of two but never falls over is the better product.\nConsent and permissions are the genuinely hard part. I expected the engineering to be the work. It was not. The work was getting the scopes exactly right — least privilege means you discover a missing permission at the worst moment, mid-run, as a 403 — and getting a customer\u0026rsquo;s tenant admin to grant admin consent to an app they did not create. That conversation is easier precisely because the app is read-only and I can prove it, but it is still the bottleneck. The technology was the easy half.\nThe model lied to me before I grounded it. My first naive version handed the model a big blob of tenant JSON and asked for a report. It produced something beautiful and partly fictional. It asserted MFA percentages that were not in the data, named a Conditional Access policy that did not exist, and was completely calm about all of it. That failure is the whole reason for the collect-then-reason split and the grounding prompt. It is the single clearest lesson of the project, and it generalises: an LLM must be grounded in real data or it will lie confidently. I would not build any reporting system on a model again without a hard firebreak between facts and prose.\nWhere this goes next The current system runs against one tenant on demand and produces a point-in-time report. Three things are coming.\nScheduled drift detection. Run the same collection on a schedule, store the normalised JSON in Git, and diff it. A health check that runs once is a photograph. A health check that runs weekly and tells you \u0026ldquo;someone added a Global Admin and disabled a CA policy on Tuesday\u0026rdquo; is a smoke alarm. Because the findings are already structured JSON, the diff is nearly free — this is exactly the kind of repeatable, version-controlled output I argue for in building knowledge instead of documents.\nMulti-tenant. As a presales and consultancy tool this needs to fan out across many customer tenants from one orchestrator, each with its own isolated app registration and credential, results kept strictly separate. The collect-then-reason architecture already supports this; the work is credential management and tenant isolation, not new logic.\nBenchmarking. Once there is a corpus of anonymised, structured findings across tenants, a finding stops being absolute and becomes relative — \u0026ldquo;your Secure Score is 47, the median for tenants your size is 63.\u0026rdquo; That comparison is far more persuasive to a customer than a bare number, and it is only possible because the findings were structured and reproducible from day one.\nClosing thought The thing I keep coming back to is that this project did not make the LLM smarter. It made the LLM smaller. The model\u0026rsquo;s job shrank from \u0026ldquo;assess this tenant\u0026rdquo; to \u0026ldquo;write up these graded facts in the right order,\u0026rdquo; and the report got better the more I took away from it.\nThat is the lesson I would hand to anyone building automation around a language model. The intelligence that matters here is not in the model — it is in the deterministic collection, the least-privilege access, and the rules that decide what is true. The model is a competent writer that I refuse to let near the source of truth. Keep the facts in code, keep the model on a short leash, and the result is something a customer can trust and an engineer can reproduce. That is the whole product. The model, as ever, is not.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/microsoft-365-ai-health-check/","title":"Microsoft 365 AI Health Check"},{"content":"Modern Citrix Architecture I design Citrix for a living. Not the marketing version — the version that has to survive a maintenance window at 2am, a licence true-up at renewal, and a sceptical infrastructure lead who has been burned before.\nSo I want to be honest from the first line. Citrix in 2026 is a smaller, sharper proposition than it was five years ago, and a lot of the people still buying it are buying it out of habit. My job, when I sit in front of a customer, is not to sell Citrix. It is to work out whether they need it, and if they do, design something that will not become an operational millstone.\nThis article is the design thinking I bring to that conversation: where Citrix sits now, the trade-offs of moving the control plane to the cloud, how I build hybrid deployments, how identity has changed, and — the bit most vendors skip — when I tell a customer to walk away from Citrix entirely.\nWhere Citrix actually sits in 2026 The platform has been through an ownership change and a consolidation, and the messaging has shifted with it. The product family has been pruned and repackaged into platform bundles, the sales motion is cloud-first, and the language leans hard on \u0026ldquo;the platform\u0026rdquo; rather than the components engineers actually deploy. Some of that is genuine simplification. A lot is repricing dressed up as strategy.\nWhat has not changed is the engineering reality. Underneath the bundles you still have the same moving parts: Delivery Controllers or the cloud-hosted equivalent, VDAs, StoreFront or Workspace, a Gateway, a licensing mechanism, a SQL database or its managed substitute, and a provisioning method. The brochure changes faster than the architecture does. I design the architecture and decode the brochure at procurement.\nThe single most important shift is that Citrix now expects you to run the control plane in their cloud. That is the centre of gravity for every design decision below.\nCVAD versus DaaS — what moving the control plane really costs The classic deployment, Citrix Virtual Apps and Desktops (CVAD), keeps everything on-premises or in your own tenancy: Delivery Controllers, the SQL Server site database, StoreFront, the licensing server, Director, Studio. You own the lot — you patch it, back it up, design its high availability, and when it breaks at 2am it is your runbook.\nDaaS — the Citrix Cloud control plane — lifts the Delivery Controllers, the site database, the consoles, and the licensing brokerage into Citrix\u0026rsquo;s cloud. You keep the VDAs, the machines that actually run the workloads, wherever your users\u0026rsquo; data and applications live, and connect them back through Cloud Connectors. On paper this is attractive: no more controller patching, no SQL HA design, no StoreFront upgrade weekends, evergreen consoles, faster features. Here is the honest trade-off I walk customers through.\nResilience changes shape; it does not disappear. The marketing implies the cloud control plane removes your availability problem. It moves it. Citrix builds in a Local Host Cache so that if the connection to the cloud is lost, the Cloud Connectors keep brokering existing resources from a cached copy of the site data. That is good engineering and it works — but it is a degraded mode: no configuration changes, reduced capabilities, and a reliance on the Connectors and their local SQL Express instance staying healthy. So your resilience story becomes a hard dependency on Citrix\u0026rsquo;s cloud uptime, mitigated by a cache that buys survival, not normality. For most customers that is a sensible trade. For a hospital or a trading floor, I make them stare at that dependency until they are comfortable, because it is now outside their change control.\nCloud Connectors are the new thing you own. People hear \u0026ldquo;DaaS, fully managed\u0026rdquo; and forget the Cloud Connectors are theirs. They are Windows servers, you need at least two per resource location, they need patching, they auto-update on Citrix\u0026rsquo;s schedule, and they are the lifeline between your VDAs and the brain in the cloud. Lightweight, but not nothing. I have seen \u0026ldquo;managed\u0026rdquo; deployments fall over because someone treated the Connectors as fire-and-forget and let them drift.\nLicensing and cost is where the real argument lives. CVAD has historically been perpetual-with-maintenance; DaaS is subscription, per user or per concurrent, billed annually. Over a three-to-five year horizon the subscription frequently costs more in raw licence spend than the perpetual model did. The counter-argument is that you also remove the cost of running and patching the control-plane infrastructure, the SQL licensing, the upgrade labour, and the on-call burden. Sometimes that nets out in DaaS\u0026rsquo;s favour, sometimes not. I model it properly at the proposal stage rather than asserting it, exactly as I describe in from proposal to production — a cost claim you cannot defend at renewal is how trust evaporates.\nWho owns what becomes a contractual question, not just a technical one. With DaaS the line of demarcation moves. Citrix owns the control plane SLA; you own everything from the Cloud Connector inwards. When something breaks, the first ten minutes of the incident go on working out whose side of the line it is on. I make sure that boundary is written down before go-live, because an ambiguous boundary is an outage extender.\nMy default position in 2026: for greenfield and genuinely cloud-committed customers, DaaS is the right starting assumption. For customers with hard data-residency, latency, or air-gap requirements, or a healthy CVAD estate with years of perpetual value left, I do not move them just because the slide says cloud. I have rejected DaaS more than once, and been right to.\nHow I build a hybrid deployment The realistic 2026 design for most mid-to-large customers is hybrid: the control plane in Citrix Cloud, VDAs spread across one or more resource locations — on-premises hypervisor, a public cloud region, sometimes both. A resource location is just a logical grouping of compute plus its pair of Cloud Connectors, and the pattern scales by adding resource locations, not by scaling a central controller. The topology I most commonly draw:\nflowchart TD User[End user device] --\u003e GW[Citrix Gateway service] GW --\u003e WS[Workspace] WS --\u003e CP[Citrix Cloud control plane] CP --\u003e RL1CC[Cloud Connectors RL1] CP --\u003e RL2CC[Cloud Connectors RL2] RL1CC --\u003e VDA1[On prem VDAs] RL2CC --\u003e VDA2[Azure VDAs] VDA1 --\u003e DATA1[On prem file and app data] VDA2 --\u003e DATA2[Azure data services] CP --\u003e IDP[Entra ID] GW --\u003e IDPThe design rule that matters most is keep the VDAs next to the data. A virtual desktop is a frame-buffer renderer that happens to run Windows; the experience is governed by the round trip between the VDA and the data it touches, not by where the control plane lives. So if the applications and file shares are on-premises, the VDAs go on-premises even though the brain is in the cloud. Mid-migration to Azure, you run two resource locations and shift the VDAs as the data follows. Getting this wrong is the most common cause of \u0026ldquo;Citrix is slow\u0026rdquo; tickets, and it is almost never Citrix\u0026rsquo;s fault — it is a topology decision made without thinking about data gravity. It is the same locality thinking I apply to designing infrastructure for AI workloads: put the compute where the data is, or pay for it in latency forever.\nImage management — MCS, PVS, or App Layering How you build and deliver the machine images quietly determines your operational life for the next three years.\nMachine Creation Services (MCS) is my default now. It uses the hypervisor or cloud platform\u0026rsquo;s native cloning and snapshots to spin VDAs from a master image. It is simple, has no extra infrastructure to run, and works natively across on-prem hypervisors and Azure.\nProvisioning Services (PVS) streams a disk image over the network to target devices. It was the right tool when storage was expensive and you wanted to boot hundreds of identical machines from one shared vDisk over the wire. It still has a place at very large scale or with specific streaming requirements, but it brings its own infrastructure — PVS servers, a streaming network design, a store — and that is more to own and more to break. I reach for PVS deliberately and rarely, and I make the customer prove they need it.\nApp Layering composes images from independently maintained layers — OS, platform, apps — so you can update one application without rebuilding the gold image. Elegant in theory, and genuinely useful when you have a sprawling, frequently-changing application estate maintained by different teams. In practice it adds real complexity, and I have watched it become shelfware at customers who adopted it for the demo and never operationalised it. The honest summary: MCS unless you can articulate exactly why not.\nIdentity and access in 2026 This is the area that has changed the most, and for the better. The old world was a Gateway appliance pre-authenticating against on-prem AD with maybe RADIUS for the second factor. The new world is identity-first, and runs through Entra ID.\nI now design almost every new Citrix access path with Entra ID as the identity provider for Workspace, so authentication, Conditional Access, and MFA are governed by the same policies as the rest of the Microsoft estate. A user\u0026rsquo;s access to their virtual desktop becomes subject to the same risk-based Conditional Access and device-compliance checks that gate their email and SharePoint. One identity plane, one set of policies, one audit story. This is exactly the posture I look for when I run a Microsoft 365 health check — Citrix access that bypasses Conditional Access is a finding, not a feature.\nAdaptive authentication lets you vary authentication and the resulting access by context — where the user is, what device they are on, what network they came from. Combined with adaptive access policies you can allow full desktop access from a compliant managed device but restrict clipboard, printing, and downloads from an unmanaged one. That contextual control is genuinely one of the better reasons to still choose Citrix, and I will come back to it.\nThe Gateway service in Citrix Cloud handles secure remote access without you publishing a StoreFront and an appliance to the internet yourself. Hybrid customers who still run on-prem Gateway appliances keep patching them promptly, because Gateway appliances have a long history of being a target. Patch hygiene on internet-facing Citrix components is not optional, and it goes in every design document I write.\nThe decision that matters most — do they even need Citrix? This is the part of my job I take most seriously, and the part that separates an architect from a reseller. The question I ask before any Citrix design is: what problem are we actually solving? Citrix earns its complexity when you have a real, specific need. The genuine reasons are: keeping data off the endpoint for security or regulatory reasons; delivering heavy, awkward, latency-sensitive legacy applications to a wide audience; centralising compute next to a large dataset; supporting unmanaged or BYOD devices with strong contextual control; or graphics-intensive workloads that need a GPU the endpoint does not have. Where one of those is true, Citrix is excellent and I will happily design it.\nBut a great many organisations bought Citrix for \u0026ldquo;remote access to Windows apps\u0026rdquo;, and in 2026 that problem has cheaper answers.\nIf the honest requirement is \u0026ldquo;let people use Office and a line-of-business web app from anywhere\u0026rdquo;, you probably do not need Citrix.\nFor pooled or personal cloud desktops on Azure, Azure Virtual Desktop (AVD) does the core VDI job natively, integrated with Entra ID, often at lower licence cost, with no Citrix layer to operate. Windows 365 goes further and gives each user a persistent Cloud PC as a fixed monthly per-user item — dead simple to buy and to reason about, and the right answer for a lot of knowledge workers who just need a managed Windows machine in the cloud. And in a surprising number of cases the right answer is the boring one: the applications are now SaaS, and the user needs a managed laptop, a browser, and good Conditional Access — no virtual desktop at all.\nI tell customers this even when it costs me a Citrix sale, because the alternative is selling them a platform they will resent paying to operate. Citrix on top of AVD does add value — HDX for tough networks, richer image and app management, the adaptive access controls, a single pane across hybrid estates — but that value has to clear a bar. If the customer\u0026rsquo;s whole world is Microsoft, their apps are SaaS or web, and their users are on managed devices, layering Citrix over the top is complexity they will pay for and rarely use. When NOT to use Citrix is the most valuable slide in my deck, and usually the one the customer remembers.\nThis is the presales discipline I keep coming back to: design for the customer\u0026rsquo;s actual problem, not for the product you happen to represent. It is the through-line of the future of technical presales — the architect who tells you not to buy the thing is the one you trust with the next project.\nLifecycle and operations — the burden people underestimate The proposal is where Citrix looks cheap. Operations is where the real cost lives, and it is the single most underestimated line item in every Citrix project I have seen. Three things drive that.\nLTSR versus Current Release is the first lifecycle decision. The Long Term Service Release gives you a stable, long-supported baseline with cumulative updates and a multi-year support window — boring, predictable, and what most production estates should run. The Current Release track gives you features sooner but a much shorter support life, so you are on a faster upgrade treadmill. My default for production is LTSR, with a deliberate, tested move to each new LTSR baseline rather than chasing every CR. I only put a customer on CR when they have a specific feature need and the operational maturity to keep up with the cadence.\nImage pipelines are where good operations are won or lost. An image you update by hand, at night, by clicking through a console, is one you will eventually break and struggle to roll back. I push customers towards a repeatable build — a documented gold image, updates applied through a controlled process, validated in a staging delivery group, then promoted. Even modest automation pays for itself the first time you need to back out a bad update. Some PowerShell of the kind that lives in that pipeline, rolling a catalogue onto a new master image:\n# Update an MCS catalogue to a new master image snapshot, staged then promoted $catalogue = \u0026#34;Win11-Knowledge-Workers\u0026#34; $newSnapshot = \u0026#34;XDHyp:\\HostingUnits\\AzureUKSouth\\win11-gold-2026-06.snapshot\u0026#34; # Publish the new image to the catalogue\u0026#39;s provisioning scheme Publish-ProvMasterVmImage -ProvisioningSchemeName $catalogue ` -MasterImageVM $newSnapshot # Roll existing machines onto the new image on next reboot, in controlled batches Request-ProvVMUpdate -ProvisioningSchemeName $catalogue ` -StartsNow $true -DurationInMinutes 0 Write-Output \u0026#34;Catalogue $catalogue scheduled to update to $newSnapshot\u0026#34; The point of writing it as code is not cleverness — it is that the next engineer, or the customer\u0026rsquo;s own team, can read exactly what happened and reproduce it. That is the build-knowledge-not-documents instinct I argue for in from proposal to production: the runbook is the deliverable.\nMonitoring and observability. Director (or the Monitor service in DaaS) gives you the Citrix-native view: session launch times, logon duration broken down by phase, machine health, failure reasons. It is good, and customers underuse it. But I never rely on it alone. The questions that cause escalations — is it the broker, the VDA, the profile, the network, the storage, or the application — are answered by correlating Citrix\u0026rsquo;s view with the platform underneath it. So I wire Citrix health into the same monitoring spine the rest of the estate uses rather than leaving it on an island, the instinct behind building repeatable customer health checks. A green Director dashboard next to red storage latency is a story you want in one place.\nCapacity is the slow leak. Citrix estates grow by accretion — a new application group here, a department migrated there — and the sizing that was right at go-live is quietly wrong eighteen months later. Logon storms at 9am, oversubscribed hosts, profile bloat and \u0026ldquo;it\u0026rsquo;s slow on Mondays\u0026rdquo; all trace back to capacity assumptions nobody revisited. I build a capacity review into the operational handover, because the deployment perfectly sized on day one is guaranteed to be mis-sized by year two. And note what DaaS does and does not buy you here: it reduces the control-plane share of the burden, but it does not touch the VDA, image, profile, capacity, and application-management share — which is most of it.\nWhere this goes next The direction of travel is clear and I design towards it. The control plane will keep moving to the cloud; that argument is largely over. Identity will keep consolidating into Entra ID until Citrix is just another relying party with very good contextual controls. As AVD and Windows 365 mature, the space where Citrix adds defensible value narrows to the genuinely hard cases — tough networks, demanding graphics, deep contextual security, and large heterogeneous estates that need one pane of glass. That is a smaller, more defensible Citrix, and I am comfortable designing for it.\nFor my own practice, the improvement I am working on is treating Citrix designs the way I treat everything else on this site: as code and repeatable patterns rather than bespoke snowflakes. Reference architectures for the common shapes, parameterised image pipelines, and health-check tooling that audits a live Citrix estate the way my Microsoft 365 tooling audits a tenant. The value is in the repeatability now, not the novelty.\nClosing thought The best Citrix design I do this year might be the one where I talk a customer out of Citrix. That is not a failure of the product — it is the product finding its proper place. Citrix is no longer the default answer to \u0026ldquo;how do people get to their applications\u0026rdquo;, and pretending otherwise serves nobody.\nWhen it is the right tool it is genuinely excellent, and designing it well — VDAs next to the data, MCS unless proven otherwise, identity through Entra ID, LTSR for stability, operations taken seriously from day one — produces something that quietly works for years. When it is the wrong tool, the most professional thing I can do is say so. Designing for the problem rather than the product is the whole job, the thread that runs from the first conversation through to a live system in proposal to production.\n","date":"2026-06-25T00:00:00Z","permalink":"/citrix/modern-citrix-architecture/","title":"Modern Citrix Architecture"},{"content":"My Journey into Local LLMs The first time I ran a language model on my own hardware, it was slow, the output was mediocre, and I was completely hooked. There is a particular feeling the first time a model answers you with no network involved at all — no API key, no token meter ticking over, no terms of service deciding what you are allowed to ask. The weights are on a disk you own. The inference happens on silicon in a box three feet away. It is yours.\nThat feeling is not a good enough reason on its own, and I want to be honest about that up front. The frontier hosted models are still smarter than anything I can run at home, and for the genuinely hard problems they remain the right tool. But somewhere between \u0026ldquo;this is a toy\u0026rdquo; and \u0026ldquo;this replaces everything\u0026rdquo; there is a large, useful, and rapidly growing space where running models locally is the correct engineering decision — and most people never explore it because they assume it is either trivial or impossible. It is neither.\nThis is the article I wish I had read before I started. It is about why you would bother running LLMs locally at all, the GPU reality that nobody tells you, what quantisation actually trades, how to pick a model for a job instead of a leaderboard, and how Ollama ties it together. It sits underneath a lot of the rest of my work: the assistant in Project Atlas runs on exactly this stack, and the hardware reasoning here is the practical companion to the deeper sizing exercise in designing infrastructure for AI workloads.\nWhy local at all The honest answer is that no single reason justifies it, but a stack of them does.\nPrivacy is the one everyone reaches for first, and it is real. When I run a health-check report or summarise a customer\u0026rsquo;s tenant configuration, that data never leaves my network. There is no clause in a vendor\u0026rsquo;s data-processing agreement to read three times. For anything touching client systems, \u0026ldquo;the inference happened on a machine I control\u0026rdquo; is a sentence that ends a lot of awkward conversations.\nControl is the quieter, more important one. A hosted model can change underneath you without warning — a new version, a different refusal behaviour, a deprecated endpoint, a tightened rate limit on the Tuesday you had a deadline. A local model is a file. It does exactly what it did yesterday because it is byte-for-byte the same thing. When I build automation on top of a model, that determinism is worth a great deal. I describe the broader version of this argument in AI is becoming infrastructure: the moment you depend on something operationally, you want to own its failure modes.\nThen there is cost and latency. Bulk work — classifying a thousand documents, drafting boilerplate, summarising logs overnight — is exactly the kind of high-volume, low-stakes task where per-token pricing adds up and where a model sitting warm on local VRAM answers in milliseconds with no round trip. No rate limits. No 429s at 2am when a batch job runs hot.\nAnd finally, learning. You do not really understand how these systems behave until you have watched one fill your VRAM and slow to a crawl because you asked for one more billion parameters than your card could hold. Running models locally taught me more about how they actually work than any amount of using them through a polished API ever did.\nIf hosted models are the frontier, local models are the foundation. You want to own the foundation.\nWhat local is not is a replacement for the best hosted models on the hardest tasks. When I need genuine reasoning over a thorny architecture problem, the largest frontier models are still ahead, and pretending otherwise is how people end up disappointed. The skill is knowing which jobs fall on which side of the line.\nThe GPU reality: VRAM is king Here is the single most important thing I learned, and it is the thing the benchmark culture obscures: for local inference, VRAM is king. Not clock speed, not the headline TFLOPS, not the marketing tier of the card. The question that decides whether a model runs at all is simply: does it fit in video memory?\nA language model has to load its weights into VRAM to run quickly. If the weights plus the context plus the working memory fit, you get fast inference. If they do not fit, you spill into system RAM over the PCIe bus, and performance falls off a cliff — we are talking an order of magnitude slower, the difference between a conversation and a coffee break.\nThis is why I run a single RTX 3090 with 24GB of VRAM rather than something newer and faster with less memory. A 4070-class card might win a gaming benchmark, but with 12GB it simply cannot hold the models I actually want to run at a quality I am happy with. The 3090, bought second-hand off the back of the gaming market, gives me 24GB for a sane price, and 24GB turns out to be the practical line where local LLMs get genuinely useful. I made the same VRAM-per-pound argument when I built the wider AI infrastructure lab at home, and I would make it again today.\nWhat actually fits on 24GB, in my experience:\n7–8B models run comfortably, fast, with a generous context window. This is the daily-driver class. 13–14B models run fine at a sensible quant — a little slower, noticeably more capable for harder instruct and coding work. 32–34B models are the stretch: doable at Q4, with a tighter context budget and patience. 70B models only at aggressive quantisation, with a small context, and slowly. It works, it is occasionally worth it, but you feel every gigabyte. The lesson that took me longest to internalise: a 14B model that fits entirely in VRAM will beat a 70B model that is half spilling into system RAM, every time, on responsiveness. Fit first. Cleverness second.\nQuantisation, explained properly If you come from an engineering background, quantisation deserves a proper explanation rather than the hand-wave it usually gets.\nA model\u0026rsquo;s weights are originally trained in 16-bit floating point. Quantisation stores those weights at lower precision — 8-bit, 5-bit, 4-bit, sometimes lower — so each parameter takes fewer bytes. The whole file shrinks roughly in proportion. An 8B model in full FP16 is around 16GB; at 4-bit it is closer to 4.5GB. That is the difference between \u0026ldquo;barely fits\u0026rdquo; and \u0026ldquo;fits four times over with room for a big context\u0026rdquo;.\nThe format I live in is GGUF, the packaging used by llama.cpp and therefore by Ollama. Within GGUF you choose a quantisation level, and the naming looks cryptic until you decode it. The one you will see most is Q4_K_M: 4-bit, \u0026ldquo;K-quant\u0026rdquo; method, medium variant. The K schemes are smart — they spend more bits on the weights that matter most and fewer on the rest, which is why a modern Q4_K_M holds up far better than a naive 4-bit quant from a few years ago.\nThe trade is precisely this: lower bits means smaller file, less VRAM, and faster inference, at the cost of some quality. Quality loss is usually measured as perplexity — how surprised the model is by real text — and the curve is the key insight. Going from FP16 down to Q6 or Q5 costs almost nothing measurable. Q4_K_M sits at the sweet spot of the curve, where you get most of the size saving for a quality drop you genuinely struggle to notice on most tasks. Below 4-bit, the curve turns sharp and quality degrades fast.\nSo my pragmatic rule:\nQ4_K_M is my default. Best balance of size, speed, and quality for almost everything. Q5_K_M / Q6_K when the model is small enough that I have VRAM to spare and want the last few percent of quality — coding and structured-output tasks benefit most. Q8 only when I am specifically checking how much quantisation is costing me on a task. Below Q4 only to squeeze a model in that otherwise would not fit at all, and only after testing that it has not gone stupid. The perplexity numbers are a guide, not a verdict. What matters is whether the quant still does your job.\nChoosing a model by the job, not the leaderboard This is the part where I have changed my mind the most. When I started, I chased leaderboards — whatever topped the charts that week got downloaded. It was a waste of bandwidth. The recurring lesson, the one I keep relearning, is that the model is not the product. The product is the thing you build around it, and different models are simply better at different jobs.\nThe model is not the product.\nIn rough terms, here is how I actually allocate work across the models I keep around:\nLlama 3.1 (8B) — my reliable generalist. Good instruction-following, sane defaults, well-behaved in automation. When in doubt, start here. Qwen2.5 — my pick for coding and tight instruct work. The coder variants in particular punch well above their parameter count, and for structured output it is consistently the strongest small model I run. Mistral — fast, lean, good at summarisation and high-volume bulk work where I want throughput. gpt-oss — a genuinely capable open-weight option that I reach for when I want stronger reasoning locally and can spare the VRAM. DeepSeek — the one I pull out for harder reasoning and maths-flavoured problems when I want to see how far local can stretch. None of these is \u0026ldquo;the best\u0026rdquo;. Each is the best at something. A leaderboard collapses that into a single number and throws away the only information I care about. The way I evaluate is to keep a small folder of my own real tasks — a tricky n8n transformation, a Conditional Access summary, a Python refactor — and run candidate models against them. That homemade eval set has been worth more than every public benchmark combined.\nOllama as the runtime Everything above would be academic without a runtime that makes it pleasant, and for me that is Ollama. It wraps llama.cpp, handles model download and storage, manages what is loaded in VRAM, and exposes a clean HTTP API. It is the layer that turned \u0026ldquo;an interesting weekend\u0026rdquo; into \u0026ldquo;a thing I actually use\u0026rdquo;.\nThe day-to-day is unremarkable in the best way:\n# Pull a model at a specific quant and chat with it ollama pull qwen2.5:14b-instruct-q4_K_M ollama run qwen2.5:14b-instruct-q4_K_M # See what is downloaded and what is currently loaded in VRAM ollama list ollama ps Two settings matter more than any other. The first is the context window, num_ctx. Bigger context lets the model see more at once, but — and this is the trap — context costs VRAM, and it costs it on top of the weights. Double the context and you can push a model that fit comfortably into spilling over the edge. I size context to the job, not to the maximum the model supports.\nThe second is keeping models warm. The first request after a model loads is slow because the weights have to stream into VRAM; subsequent requests are fast while it stays resident. Ollama unloads models after an idle timeout to free memory, so for anything latency-sensitive I keep the primary model pinned in memory and accept that it is occupying VRAM full time. That trade — memory for responsiveness — is one you make deliberately.\nThe thing that makes Ollama genuinely useful for building, though, is the Modelfile. It lets me bake a base model, a system prompt, and parameters into a named, version-controlled artefact:\n# Modelfile — a focused assistant for infra summarisation FROM qwen2.5:14b-instruct-q4_K_M PARAMETER num_ctx 8192 PARAMETER temperature 0.3 PARAMETER top_p 0.9 SYSTEM \u0026#34;\u0026#34;\u0026#34; You are an infrastructure assistant. Answer concisely and technically. Prefer British English. When asked for config, return valid, copy-pasteable snippets. If you are unsure, say so rather than inventing detail. \u0026#34;\u0026#34;\u0026#34; ollama create infra-assistant -f ./Modelfile ollama run infra-assistant Now infra-assistant is a reproducible thing I can commit to Git, the same way I treat every other piece of my homelab as code, in the spirit of my Docker homelab lessons. The API is equally plain — a POST to /api/generate or /api/chat — which is exactly how n8n and the rest of Atlas talk to it. The model becomes just another service on the network with an endpoint.\nA router mindset: where local fits versus hosted The mistake is treating \u0026ldquo;local versus hosted\u0026rdquo; as a religious choice. It is not. It is a routing decision you make per task, and once I started thinking of it that way the whole thing clicked.\nCheap, private, bulk, latency-sensitive, or determinism-critical work goes local. Hard reasoning, frontier-level capability, the genuinely novel problem — that goes hosted. The trick is having an explicit rule for which is which, rather than defaulting to whichever you happen to have a tab open for.\nflowchart TD A[Incoming task] --\u003e B{Sensitive or private data} B -- Yes --\u003e L[Run local] B -- No --\u003e C{High volume or bulk} C -- Yes --\u003e L C -- No --\u003e D{Needs frontier reasoning} D -- Yes --\u003e H[Use hosted frontier model] D -- No --\u003e E{Coding or structured output} E -- Yes --\u003e Q[Local Qwen2.5 at Q5] E -- No --\u003e F{General chat or summary} F -- Yes --\u003e G[Local Llama or Mistral at Q4] F -- No --\u003e H L --\u003e M{Fits in 24GB at Q4} M -- Yes --\u003e N[Load and keep warm] M -- No --\u003e O[Drop a size or raise quant pressure]In practice the router lives in n8n. A workflow inspects the task, decides the destination, and only escalates to a hosted model when the local one is genuinely the wrong tool. Most days, most tasks never leave the building.\nLessons learnt Some of these I learned the slow way.\nContext length is not free, and it is the silent VRAM killer. I have spent more time than I would like debugging an out-of-memory error that turned out to be a context window I had bumped up and forgotten about. The weights are the obvious cost; the KV cache for a large context creeps up behind you.\nTokens per second sets the experience, and expectations matter. On the 3090, an 8B model at Q4 runs fast enough to feel conversational. A 70B at aggressive quant produces tokens at reading pace at best. Neither is wrong — but if you expect frontier-API speed from a 70B on a single consumer card you will be perpetually disappointed. Know the number for each model before you build a UX on top of it.\nBenchmarks lie, or at least mislead. A model that tops a public leaderboard can be mediocre at the specific shape of work you do, and a humbler model can be excellent at it. Test on your own tasks. I will say it again because it cost me real time to learn: build a small eval set of your actual work and trust that over any chart.\nThe ecosystem moves weekly. A new model, a better quant, a runtime improvement, a context-length breakthrough — it genuinely changes month to month. This is exhilarating and exhausting in equal measure. I have learned to pin a working stack and resist upgrading mid-project, while keeping a scratch environment for the churn. Treat it like any fast-moving dependency: control when the change lands.\nI got the GPU question wrong at first. My instinct was to optimise for speed, and I nearly bought a faster card with less memory. The 3090 with its 24GB was the right call, and it was VRAM, not flops, that made it right. If I were buying again today I would still start the question with \u0026ldquo;how much VRAM\u0026rdquo; and only then ask \u0026ldquo;how fast\u0026rdquo;.\nWhere this goes next A few things I am actively working towards.\nSmall models keep getting better, faster than I expected. The 7–8B class today is comfortably ahead of where the 13B class sat eighteen months ago, and that trend is the most exciting thing in the space because it directly expands what fits on hardware I already own. More capability per gigabyte is the gift that keeps giving.\nSpeculative decoding is next on my list to set up properly — a small draft model proposing tokens that a larger model verifies, buying real speed without a quality cost. On a single card the gains are meaningful, and it is exactly the kind of optimisation that makes a 14B feel closer to an 8B in responsiveness.\nI want to fine-tune my own small model on my knowledge base and writing, rather than relying on retrieval alone — a model that has genuinely internalised my domain, paired with the retrieval layer that already feeds Project Atlas. And underpinning all of it, a proper, growing eval harness, because the only way to navigate a weekly-moving ecosystem without thrashing is to be able to measure, quickly and honestly, whether a change actually made my real work better.\nClosing thought I did not get into local LLMs because they were better than the hosted frontier. They are not, and I have tried hard not to pretend otherwise. I got into them because owning the whole stack — the weights, the hardware, the runtime, the failure modes — taught me how this technology actually behaves, and made me a far better judge of when to reach for the frontier and when not to bother.\nThe gap between what I can run at home and what the largest hosted models can do will open and close as the field lurches forward. None of that changes the core of it. There is a box three feet away that answers when I ask, on data that never leaves my network, doing exactly the same thing today as it did yesterday. For a growing share of the work I actually do, that is not a compromise. It is the better answer.\n","date":"2026-06-25T00:00:00Z","permalink":"/ai/my-journey-into-local-llms/","title":"My Journey into Local LLMs"},{"content":"Project Atlas Atlas is the assistant I built because I was tired of explaining myself to a stranger every morning.\nEvery time I opened a hosted chatbot I had to re-establish who I was, what I was working on, what my homelab looked like, which acronyms meant what in my world. The conversation started from zero. The model was clever, but it knew nothing about me, and worse, it could not actually do anything. It could write me a paragraph about Microsoft Graph. It could not go and query my tenant.\nAtlas is my attempt to fix both problems at once: an assistant that remembers my world and can act on it, running on hardware I own. It is the recurring brain of the lab I describe in building an AI infrastructure lab at home, and over the past year it has become the thing several other projects quietly plug into.\nThis is the honest account of how it works, what I got wrong, and where it goes next.\nWhy not just use ChatGPT This is the first question anyone sensible asks, and it deserves a real answer rather than a privacy slogan.\nThere are four reasons I run my own, and only one of them is privacy.\nPrivacy is the obvious one. A lot of what I would want an assistant to be useful with is exactly the stuff I should not paste into someone else\u0026rsquo;s service: client tenant details, internal architecture, half-finished thinking about a deal. If the assistant is going to be genuinely useful it has to see genuinely sensitive context, and I would rather that context never leave the network.\nControl is the second. Hosted models change underneath you. The behaviour shifts, the guardrails move, the price moves, the model gets deprecated. When the assistant is part of my daily workflow I do not want it changing personality because a vendor shipped a new system prompt on a Tuesday.\nThe third reason is the one people underrate: it knows my projects. A generic model is brilliant and amnesiac. Atlas has a knowledge base of my actual notes, so when I ask \u0026ldquo;what did I decide about VLAN segmentation,\u0026rdquo; it answers from what I wrote, not from the average of the internet.\nThe fourth, and the one that changed everything, is that it can act. Atlas is wired into n8n, so it can hit Microsoft Graph, poke Home Assistant, or search my files and come back with a real answer. A model that can only talk is a clever notepad. A model that can call tools is an assistant.\nA model on its own is just a text box. The value is everything you build around it.\nNone of this means hosted models are bad. For throwaway questions I still reach for one. But the assistant I depend on is mine.\nThe hardest part is memory If you only take one thing from this article, take this: memory is the hard problem, not the model.\nPeople imagine an assistant remembers things the way a person does. It does not. A language model has no memory at all between calls. Everything it \u0026ldquo;knows\u0026rdquo; in a conversation is just text you stuffed into the context window this time. So the entire design of Atlas is really a design about what text goes into that window, and where the durable stuff lives in between.\nI ended up with three distinct kinds of memory, and keeping them separate was the single best decision I made.\nThe first is short-term chat memory — the running conversation, held in Open WebUI, scoped to the current thread. This is genuinely ephemeral and I treat it as such. It is the working scratchpad, nothing more. When the thread ends, it is gone, and that is correct. You do not want yesterday\u0026rsquo;s tangent silently leaking into today\u0026rsquo;s reasoning.\nThe second is the durable knowledge base — Markdown files in a Git repository. This is the real memory. It is plain text, version-controlled, diffable, and it outlives any model, any tool, any platform. This is the same conviction that drives the whole site and that I set out in building knowledge instead of documents: the durable thing is the knowledge, written down in a format that will still open in twenty years.\nThe third is retrieval — the machinery that, at question time, pulls the relevant slices of that knowledge base into the context window. This is where RAG lives, and where most of the disappointment lives too.\nThe mistake I made early was conflating these three. I tried to make the chat history into the long-term memory, and it became an unmanageable, lossy mess. Separating \u0026ldquo;the conversation\u0026rdquo; from \u0026ldquo;what I actually know\u0026rdquo; from \u0026ldquo;how I fetch what I know\u0026rdquo; is what made Atlas reliable.\nThe knowledge layer, and the truth about RAG My notes are Markdown in Git. That is the second brain I describe in building a second brain, and Atlas reads from the same repository. Nothing duplicated, one source of truth.\nThe retrieval pipeline is conventional. A small ingestion job chunks the Markdown, generates embeddings with a local embedding model, and stores them in a vector database. At question time, Atlas embeds the question, finds the nearest chunks, and prepends them to the prompt with instructions to answer from that context.\nHere is the part the tutorials skip: naive RAG is mediocre, and it is mediocre for boring reasons.\nThe first reason is chunking. If you split documents on a fixed character count you slice sentences in half and orphan the heading from the paragraph that gives it meaning. I switched to chunking on Markdown structure — headings and sections — so a chunk is a coherent idea, not an arbitrary 500 characters. That single change did more for answer quality than any model swap.\nThe second reason is that semantic similarity is not relevance. The nearest vectors to your question are often near because they share vocabulary, not because they answer it. Ask \u0026ldquo;how is the battery charged\u0026rdquo; and you will happily retrieve five chunks that mention batteries and none that mention the charging schedule. Pure vector search has no notion of \u0026ldquo;the bit that actually answers this.\u0026rdquo;\nThe third reason is the killer: RAG quality is mostly data hygiene. If your notes are contradictory, stale, or vague, retrieval faithfully serves up contradictory, stale, vague context, and the model dutifully launders it into a confident wrong answer. The assistant is only ever as good as the knowledge you feed it. I spent weeks tuning retrieval parameters before I accepted that the problem was my notes, not my cosine similarity threshold.\nSo I stopped treating RAG as magic and started treating it as a search problem with an LLM on the end. I added a keyword pass alongside the vector pass — hybrid retrieval — and a re-ranking step so the chunks that survive are the ones that actually look like answers. It is less glamorous than \u0026ldquo;AI that reads your documents\u0026rdquo; and far more useful.\nTooling: how Atlas actually does things Retrieval makes Atlas knowledgeable. Tools make it useful.\nThe mechanism is tool-calling. The model is told, in its system prompt, that it has a set of tools it can invoke by emitting a structured request. Open WebUI passes those requests to n8n, n8n is the spine that I lean on across the whole lab, and each tool is just an n8n workflow with a webhook trigger. When Atlas decides it needs live data, it calls the workflow, n8n does the real work against a real API, and the result comes back into the conversation for the model to reason over.\nThree tools earn their keep daily. One queries Microsoft Graph — the same app-registration-and-client-credentials plumbing behind the Microsoft 365 AI health check — so Atlas can answer questions about a tenant from live config rather than guesswork. One hits Home Assistant, so I can ask whether the battery is charging and get the actual state, which ties straight into the AI battery optimiser. One searches my files, a deliberately separate path from RAG for when I want an exact filename or a grep, not a fuzzy semantic match.\nHere is the shape of a tool definition as Atlas sees it. The model never touches credentials; it only knows the tool exists and what it returns.\n{ \u0026#34;name\u0026#34;: \u0026#34;query_home_assistant\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Get the live state of a Home Assistant entity. Use for real-time values like battery level, solar generation, or whether a device is on. Do not guess these values.\u0026#34;, \u0026#34;parameters\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;entity_id\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;The HA entity, e.g. sensor.battery_level\u0026#34; } }, \u0026#34;required\u0026#34;: [\u0026#34;entity_id\u0026#34;] }, \u0026#34;endpoint\u0026#34;: \u0026#34;https://n8n.lab.internal/webhook/ha-state\u0026#34; } And the n8n side is a small workflow: a webhook node receives the call, an HTTP node queries Home Assistant with a token from the vault, and a function node trims the response to just the fields the model needs. Sending the model less is almost always better than sending it more.\n[Webhook] -\u0026gt; [HTTP Request: GET /api/states/{{entity_id}}] -\u0026gt; [Function: pick(state, attributes.unit)] -\u0026gt; [Respond] The discipline that matters: tools return data, they do not make decisions. n8n fetches the battery state. Atlas decides what to say about it. Keeping the judgement in the model and the actions in audited workflows is what lets me trust the thing.\nPersona, guardrails, and saying \u0026ldquo;I don\u0026rsquo;t know\u0026rdquo; A system prompt is not flavour text. It is the constitution of the assistant, and I rewrote mine more times than any other part of the project.\nAtlas has a deliberately plain persona — a competent, terse engineering colleague, not a chirpy helper. Voice matters less than two behaviours I had to engineer hard: grounding and humility.\nGrounding means it answers from retrieved context and tool output, and flags when it is going beyond them. Humility means it is allowed — encouraged — to say it does not know. The default failure mode of every model is confident fabrication, and in an assistant that can act, a confident wrong answer is worse than no answer. I would rather Atlas say \u0026ldquo;I don\u0026rsquo;t have a note on that\u0026rdquo; than invent one.\nYou are Atlas, a local engineering assistant for Kris\u0026#39;s homelab and work. Rules: - Answer from the provided context and tool results. If they do not contain the answer, say so plainly. Do not guess. - For any live value (battery, tenant config, device state) you MUST call a tool. Never state a current value from memory. - Prefer \u0026#34;I don\u0026#39;t know\u0026#34; or \u0026#34;I have no note on that\u0026#34; over a plausible invention. Being wrong is more expensive than being silent. - Be concise. You are talking to an engineer, not writing marketing copy. - When you use a note, name the source file so it can be checked. That last line — name the source — turned out to be a quiet superpower. Citing the file it drew from makes Atlas auditable. I can click through and check, and when it cites a file that does not actually say what it claimed, that is a signal to go and fix the note. The assistant becomes a test of my own knowledge base.\nHow it fits together The whole thing is the standard lab stack: Ollama for inference, Open WebUI as the front end, n8n as the orchestration spine, all in Docker with compose files in Git as the source of truth.\nflowchart TD User[Me] --\u003e UI[Open WebUI] UI --\u003e LLM[Ollama assistant model] LLM --\u003e|needs knowledge| RET[Retrieval] RET --\u003e VEC[Vector store] VEC --\u003e KB[Git Markdown notes] LLM --\u003e|needs to act| N8N[n8n workflows] N8N --\u003e GRAPH[Microsoft Graph] N8N --\u003e HA[Home Assistant] N8N --\u003e FILES[File search] N8N --\u003e LLM RET --\u003e LLM LLM --\u003e UIThe flow is worth narrating because the loop is the point. I ask a question. Open WebUI sends it to the assistant model on Ollama. The model decides whether it needs knowledge, in which case retrieval pulls note chunks from the vector store backed by the Git repository, or whether it needs to act, in which case it calls an n8n workflow that hits a real system. The results come back, the model reasons over them, and I get an answer that is grounded in either my own notes or live state — and usually told where it came from.\nOn model choice: the assistant role wants instruction-following and reliable tool-calling far more than raw cleverness. I settled on Qwen2.5 at the 7B–14B range, quantised to Q4_K_M, because it follows the system prompt tightly and emits clean tool calls, which a more \u0026ldquo;intelligent\u0026rdquo; but sloppier model would not. This is exactly the lesson from my journey into local LLMs: you pick the model for the job. For Atlas the job is being obedient and grounded, not winning benchmarks. The bigger models I keep for one-off heavy reasoning, not for the assistant that runs all day on the RTX 3090.\nThe model is not the product. The system around it is.\nWhat I got wrong Plenty.\nI built memory before I built retrieval discipline, and ended up with an assistant that confidently remembered things that were never true, because I had let stale chat history leak into its context. Separating the three memory types fixed it, but I should have designed that boundary on day one.\nI over-trusted RAG. I assumed that \u0026ldquo;give the model my notes\u0026rdquo; would just work, and spent weeks tuning retrieval when the real fault was that my notes contradicted each other. Garbage in, confidently-phrased garbage out. The data hygiene is the work; the embeddings are the easy bit.\nI gave it too many tools too early. Every tool you add is another way for the model to misfire — to call the wrong one, or to hallucinate a tool that does not exist. A handful of reliable, well-described tools beats a sprawling toolbox the model cannot navigate.\nAnd I under-invested in guardrails until it embarrassed me by inventing a tenant setting that did not exist. That is when \u0026ldquo;prefer I don\u0026rsquo;t know\u0026rdquo; went to the top of the system prompt and stayed there.\nWhere this goes next Three directions, all concrete.\nThe first is agentic loops. Today Atlas mostly does one retrieval or one tool call per turn. I want it to plan — call a tool, look at the result, decide the next call, iterate towards an answer — within strict limits on how many steps it may take before it has to report back. The ceiling matters; an unbounded agent is a great way to generate a large bill of nonsense.\nThe second is better retrieval. Hybrid search and re-ranking got me a long way, but I want query rewriting, where the model reformulates a vague question into a better search before retrieving, and I want to chunk on meaning rather than just on headings.\nThe third, and the one I have neglected longest, is evals. I have no systematic way to tell whether a change made Atlas better or just different. I am building a small set of golden questions with known good answers, so that when I swap a model or tweak the prompt I can measure it instead of guessing. Until that exists, every \u0026ldquo;improvement\u0026rdquo; is a vibe.\nClosing thought Atlas started as a vanity project — I wanted my own JARVIS. What it actually taught me is that the interesting engineering in an AI assistant is almost never the AI.\nIt is the boundary between three kinds of memory. It is the data hygiene of the notes underneath. It is the discipline of letting tools fetch and the model judge. It is a system prompt that gives the thing permission to be uncertain.\nThe model is the cheap part. You can swap it in an evening. The knowledge base, the retrieval that respects it, the tools that act safely, and the honesty about what the assistant does not know — that is the part that took a year and is still not finished.\nAtlas is only ever as good as the knowledge I feed it. Which means, quietly, the project was never really about building an assistant. It was about getting serious about what I actually know, and writing it down well enough that a machine could use it.\n","date":"2026-06-25T00:00:00Z","permalink":"/projects/project-atlas/","title":"Project Atlas"},{"content":"The Future of Technical Presales I have spent most of my working life in the room where a customer decides whether to spend money. Not the room where the contract gets signed — the one before that, where someone technical from the vendor or partner side has to convince someone technical on the customer side that the proposed thing will actually work. That is presales. On a good day it is solution architecture with a deadline. On a bad day it is reading slides aloud.\nFor years the job had a comfortable shape. You learned a product line deeply, you built a few demos, you knew the answers to the common objections, and you could respond to a request for proposal faster than the next partner. The value you sold was, in large part, knowledge that the customer did not have and could not easily get. They did not know the licensing edge cases. They did not know which features were vapourware. They did not know how the thing failed at scale. You did, and that asymmetry was the job.\nThat asymmetry is collapsing. A customer can now open a chat window and get a competent, confident, mostly-correct explanation of any product\u0026rsquo;s architecture in about fifteen seconds. They can get a feature comparison table, a sizing rule of thumb, and a list of the gotchas — for free, at midnight, without booking a call with a vendor who will try to sell them something. So the obvious, uncomfortable question, the one I think every honest presales person is quietly asking themselves, is this:\nIf the customer can get the product facts from a machine, what exactly is the human in the room for?\nThis article is my answer, written from inside the job rather than from a strategy deck. I think the answer is genuinely good news — but only for some of us.\nThe old model was already rotting Let me be unkind about my own profession for a moment, because the AI disruption did not arrive at a healthy patient.\nA large slice of technical presales had quietly degraded into pattern-matching. There is a recognisable archetype: the person who has memorised the vendor deck, can navigate the partner portal, knows the discount thresholds, and has never actually deployed the product they sell at three in the morning when it breaks. The slideware specialist. The RFP-response factory, where the same boilerplate gets pasted into every proposal with the customer name swapped. The \u0026ldquo;generic demo\u0026rdquo; that is identical for a hospital and a hedge fund because nobody bothered to learn the difference. The box-shifter who treats architecture as a bill of materials.\nThat model worked because information was scarce and the gatekeeping was real. If you wanted to know how a product behaved, you had to ask someone who sold it. The whole edifice rested on the customer knowing less than you.\nAI kicks the leg out from under that. Not because the model is smarter than a good architect — it is not — but because it instantly erases the information asymmetry that the weakest presales relied on. The slide-reader sold facts. Facts are now free. The slide-reader is in trouble, and frankly should be.\nI do not feel sentimental about this. I have sat across the table from box-shifters as a customer and been actively misled by people who did not understand their own product. The commoditisation of product knowledge is going to be brutal for a particular kind of vendor, and I think the industry will be better for it.\nWhat the machine actually takes It helps to be precise about what AI commoditises, because the panic and the hype both come from being vague. In my own workflow — and I automate aggressively, the same way I describe in my AI consultancy toolkit — the boundary is surprisingly clean.\nAI is very good at the boring sixty percent. The first draft of a proposal. The discovery write-up that turns two hours of messy call notes into a structured summary. Configuration generation — give it the parameters and it will produce a plausible Citrix policy set, a Conditional Access baseline, a Terraform skeleton. Demo scripting. Comparison tables. The \u0026ldquo;explain this product\u0026rsquo;s HA model\u0026rdquo; paragraph. The executive summary nobody enjoys writing. All of that is now a generation away, and pretending otherwise to protect billable hours is both dishonest and a losing bet.\nHere is the part the vendors selling \u0026ldquo;AI for sellers\u0026rdquo; tools tend to skip. None of that is the actual job. It is the packaging of the job. It always was the low-value layer — the clerical work that surrounded the thinking, exactly as I argued about manual reviews when I built repeatable customer health checks. The clerical work just got cheap. The thinking did not.\nWhat AI does not do — cannot do, structurally, not just \u0026ldquo;yet\u0026rdquo;:\nIt does not hold trust. A customer who is about to bet their reputation on a migration is buying confidence in a person, and that confidence is built over months of being right and, crucially, being honest when you are not sure. A model cannot be on the hook.\nIt does not exercise judgement under ambiguity. Real architecture is a sequence of trade-offs where the right answer depends on the customer\u0026rsquo;s risk appetite, their politics, their existing skills, the thing the CFO will not say out loud, and the failed project from two years ago that nobody documented. The model will give you a confident answer to a question you have framed wrong.\nIt does not ask the awkward question. The single most valuable thing I do in a discovery session is occasionally say \u0026ldquo;why are you actually doing this?\u0026rdquo; and watch the room go quiet. AI is relentlessly agreeable. It will help you build the wrong thing beautifully.\nIt does not own the risk. When the design is wrong, when the cutover fails, when the assumption nobody validated turns out to be load-bearing — someone has to be accountable, and accountability cannot be delegated to a token predictor. The customer knows this even when the vendor pretends otherwise.\nAnd it does not read the room. Presales is a social act. Knowing when to stop talking, when the technical buyer has already decided, when the silent person at the end of the table is the actual decision-maker — that is not in the training data for your specific room.\nOld flow, new flow The shape of the work changes more than the existence of the work. Here is roughly how a presales engagement used to run, and how it runs for me now.\nflowchart TD subgraph Old A1[Generic deck] --\u003e A2[Generic demo] A2 --\u003e A3[Manual discovery notes] A3 --\u003e A4[Boilerplate RFP response] A4 --\u003e A5[Hope it lands] end subgraph New B1[AI assisted discovery] --\u003e B2[Live health check of their estate] B2 --\u003e B3[Findings drive the conversation] B3 --\u003e B4[Bespoke proposal drafted by AI] B4 --\u003e B5[Human edits judgement and risk] B5 --\u003e B6[Accountable recommendation] endThe difference that matters is not \u0026ldquo;we added AI\u0026rdquo;. It is that the centre of gravity moves from telling to showing. The demo was always a polite fiction — a curated environment where nothing breaks. The new opening move is to look at the customer\u0026rsquo;s actual estate and tell them something true about it that they did not know.\nThis is why I keep arguing that health checks are the new demo. When I can run a tenant assessment or a Citrix posture review and walk in with \u0026ldquo;here are the three things in your environment that will bite you, ranked,\u0026rdquo; I have done something no slide and no chatbot can. I have demonstrated competence on their territory instead of asserting it on mine. The demo says \u0026ldquo;look what this product can do.\u0026rdquo; The health check says \u0026ldquo;look what I found in your house.\u0026rdquo; Only one of those builds trust.\nThe architect who has actually built things This is the whole thesis, so let me state it plainly. The dividing line in the next five years of this profession is not AI-literate versus AI-illiterate. It is people who have built things versus people who only describe them.\nWhen product facts were scarce, you could fake depth. You could read the reference architecture, learn the vocabulary, and perform expertise convincingly enough. The customer had no cheap way to call your bluff. Now they do — they can check your claims against a model in real time, and the model knows the documentation better than you ever will. The only thing you can offer that the model cannot is what actually happened when you ran it.\nThat knowledge does not come from decks. It comes from the lab and from real delivery. It comes from the migration that went sideways at 2am and what you learned rebuilding it. It comes from having run the thing at small scale on your own hardware until you understood its failure modes in your hands, not in a diagram. Half the reason I keep a homelab — the GPU box, the Docker stack, the local LLM work — is that it is the cheapest place in the world to earn the scars that make you credible in front of a customer. I cannot demonstrate judgement I have not paid for.\nThis connects to something larger I have written about: AI is becoming infrastructure. When a capability becomes infrastructure, the value moves from having it to integrating it well. Electricity is infrastructure; nobody pays a premium for access to it, they pay for the engineer who wires the building safely. Product knowledge is becoming infrastructure in exactly this sense. The premium moves to the person who can integrate it into a working, accountable, context-specific design — and that person is, definitionally, someone who has built working things before.\nThe consultant as orchestrator So what does the day actually look like? Less typing, more conductor.\nI increasingly treat myself as the orchestrator of a small toolchain rather than the person who produces every artefact by hand. Discovery notes go into a model that structures them. Findings from a health check — pulled by automation, not by clicking through consoles — feed a workflow that drafts the assessment. A proposal skeleton gets generated, then I spend my time on the twenty percent that is actually architecture: the trade-offs, the risks I am willing to put my name against, the things I am deliberately not recommending and why.\nA trivial but representative piece of the plumbing — the kind of n8n-triggered draft step I lean on:\n# proposal-draft step (n8n http node -\u0026gt; local model) - node: generate_proposal_draft model: qwen2.5:14b-instruct-q4_K_M inputs: discovery_summary: \u0026#34;{{ $json.structured_notes }}\u0026#34; findings: \u0026#34;{{ $json.health_check_ranked }}\u0026#34; constraints: \u0026#34;{{ $json.customer_constraints }}\u0026#34; instruction: \u0026gt; Draft sections 1-4 only. Flag every assumption explicitly. Do not invent figures. Leave risk and commercials blank for the human. Note the last two lines. The machine drafts; it does not decide. \u0026ldquo;Leave risk for the human\u0026rdquo; is not a limitation I am working around — it is the entire division of labour. This is the same discipline that carries an engagement from proposal to production: the AI accelerates the document, but the person owns the commitment, because the person is the one who will be in the room when it ships.\nRunning the models locally matters here too, and not for ideology. Customer discovery notes, tenant configurations, network diagrams — that is exactly the data you should not be pasting into someone else\u0026rsquo;s cloud. The homelab inference box is also a confidentiality posture.\nThe sameness problem, and how you escape it There is a real risk in all of this, and I do not want to wave it away. If every partner uses the same models to draft the same proposals from the same prompts, every proposal starts to sound the same. AI is a powerful homogenising force. The median output of the profession is about to get blander and more interchangeable, even as it gets faster.\nThe escape is not a secret prompt. It is genuine engineering depth and a public, verifiable body of work. When two proposals land on a customer\u0026rsquo;s desk and both are well-written — because both were drafted by competent models — the tiebreaker is whoever can demonstrably show they have done the hard version before. Not claim it. Show it.\nThis site is my answer to that, and I am being completely literal. The reason I keep a public engineering notebook and treat it as a second brain rather than a private wiki is that it is evidence. When I tell a customer I understand how to run AI workloads on constrained hardware, or how to turn a manual review into a repeatable product, there is a paper trail of me actually doing it, with the mistakes left in. A chatbot can generate a confident claim of expertise. It cannot generate three years of consistent, dated, specific, occasionally-wrong working out. The body of work is the moat, precisely because it is expensive and slow and cannot be faked in a generation.\nWhere I would put my time If I were advising someone earlier in this career — or honestly, advising myself — about where to invest as the floor shifts, it would be roughly this, and almost none of it is about prompts.\nBuild things, for real, on your own kit, until you understand failure modes in your hands. Depth is the only durable differentiator left. Learn enough automation and scripting to be the orchestrator and not the typist — Python and a workflow engine go a long way, which is why I think every infrastructure engineer should learn Python. Get good at the human layer that AI cannot touch: discovery, the awkward question, reading intent, owning a recommendation out loud. And publish. Maintain a body of work that proves the depth is real, because in a world of identical drafts, evidence is the differentiator.\nThe skill that is depreciating fastest is \u0026ldquo;I know the product.\u0026rdquo; The skills appreciating are \u0026ldquo;I have built the thing\u0026rdquo; and \u0026ldquo;you can trust my judgement when it goes wrong.\u0026rdquo;\nWhat I actually think happens next I am not worried about AI replacing technical presales. I am quietly optimistic, in the specific way that you are optimistic about a fire that is burning down a building you never liked.\nThe slide-readers, the box-shifters, the RFP-paste-factories — that work is genuinely going, and it should. The asymmetry it lived on is gone and is not coming back. What survives, and I think thrives, is the architect who has actually built things, who uses AI to delete the clerical sixty percent of the job and spends the reclaimed time on the part that was always the point: understanding a customer\u0026rsquo;s real problem, making honest trade-offs, and putting their name against a recommendation.\nThe machine made the facts free. It turns out the facts were never what people were paying for. They were paying for someone in the room who had done it before, would tell them the truth, and would still be there when it broke. That person just got a very powerful set of tools — and a much shorter list of competitors who can do the actual job.\n","date":"2026-06-25T00:00:00Z","permalink":"/posts/the-future-of-technical-presales/","title":"The Future of Technical Presales"},{"content":"The Home Lab as a Learning Platform I have sat through more vendor labs than I can count. The format is always the same. A pre-built environment, a PDF of numbered steps, a green tick at the end. You click through, the thing works, and you walk away with a certificate of attendance and almost nothing else. I have also broken my own home network at eleven at night, with the family asking why the internet is down, and spent two hours finding out that a container I \u0026ldquo;tidied up\u0026rdquo; three weeks earlier was the only thing serving DNS. One of those two experiences taught me something. It was not the lab with the green tick.\nThis is an opinionated piece, so I will state the thesis up front. The home lab is the most effective learning platform in this industry, and it is effective precisely for the reasons that make it uncomfortable. Vendor labs and certifications are on rails. The hard parts are already solved before you arrive. The lab in my spare room solves nothing for me. It is the difference between watching someone parallel park and parking the car yourself, in the dark, with a kerb you cannot see and someone watching.\nI have written a lot here about what I have built — the AI infrastructure lab at home, the Docker homelab that runs it, the reasons every infrastructure engineer should learn Python. This piece is the meta-article behind all of them. It is about why building those things at all is the point, and why the building is worth more than any course I have ever paid for.\nThe problem with how the industry teaches Start with the vendor lab, because most engineers meet it first. A vendor lab is a sanitised reproduction of a product working correctly. Someone has already chosen the hardware, sized it, networked it, patched it, and confirmed the demo path runs. Your job is to follow the path. The trouble is that real engineering almost never happens on the path. It happens in the ditch beside it. The lab deliberately removes the ditch.\nThink about what is missing. You never see the upgrade that bricks the cluster, because the lab is reset to a known-good snapshot every time. You never feel the cost of a bad decision, because there is no decision — the architecture is handed to you. You never debug a failure that nobody documented, because every failure in the lab is anticipated and there is a hint box for it. The whole genre is built to make the product look easy, which is its commercial purpose. Education is a side effect at best.\nCertifications have a different but related failure. A certification tests recall of a vendor\u0026rsquo;s worldview. It rewards you for knowing that their product calls a thing a \u0026ldquo;delivery group\u0026rdquo; and not a \u0026ldquo;pool\u0026rdquo;, that their recommended limit is 250 sessions per host, that their best practice is option B. None of that is engineering judgement. Engineering judgement is knowing when the best practice is wrong for the customer in front of you, and being able to defend the deviation. A multiple-choice exam cannot test that, so it does not. I hold certifications. They got me past HR filters and taught me vocabulary. They did not teach me how to think, and I have interviewed enough heavily-certified people who could not reason their way out of a misconfigured subnet to be sure of it.\nA certificate proves you survived someone else\u0026rsquo;s curriculum. A scar proves you survived your own mistake.\nThe deeper issue is that all of this teaches with the failures pre-removed. In a real outage there is no answer key. Nobody cleaned up the mess before you arrived; you are the person who has to clean it up, and the clock is running. That experience — the 2am failure, the half-understood error message, the sick feeling when you realise the backup you were relying on has been silently failing for a month — is where engineers are actually made. The industry\u0026rsquo;s formal training is designed to spare you exactly that, and in sparing you it withholds the only thing that matters.\nWhat a lab gives you that those cannot A home lab inverts every one of those properties. The defining feature is ownership of the whole stack, end to end, with nobody else to blame. I own the power feeding it — and the electricity bill, which is a real constraint, not a slide. I own the network: the VLANs I am slowly carving out of a flat home LAN into trust, IoT and lab segments, the firewall rules, the DNS that brings everything down when I fat-finger it. I own the storage, the bare-metal hosts, the operating systems, the forty-odd containers, the reverse proxy, the backups, and every failure mode of every one of those layers. When something breaks, the stack of people I can escalate to is zero people tall.\nThat ownership forces real trade-offs against real constraints. Money is finite, so when I sized the GPU box I chose a single RTX 3090 for its 24GB of VRAM-per-pound rather than something faster and dearer — the same reasoning I set out in designing infrastructure for AI workloads. Watts are finite, so an always-on service has to justify its standing draw against the N100 mini PCs I run the cheap stuff on. Time is finite, so I cannot gold-plate everything. And there is a constraint no enterprise architecture review has ever modelled for me: the WAF, the wife-acceptance-factor, the hard limit where \u0026ldquo;the internet has been down for an hour while I learn something\u0026rdquo; stops being acceptable. That is a genuine availability requirement with a genuine stakeholder, and learning to design around it is learning to design around a business.\nThen you break things. Not in a sandbox someone reset for you — in the system your household actually depends on, which raises the stakes enough to make the lesson stick. You run the upgrade that takes a service down. You misconfigure the proxy and lock yourself out. You fill a disk you forgot was shared. And because there is no escape hatch, you learn the one skill that this entire profession is secretly about and that no certification examines: debugging. Reading logs that were not written for you. Forming a hypothesis, testing it, being wrong, narrowing it down. Tracing a failure across the boundary between two systems that each insist the problem is the other one. That is the job. Everything else is vocabulary.\nHow it builds credibility and instinct Here is where the lab stops being a hobby and becomes professional differentiation, which for me — doing technical presales and solutions architecture — is the entire game. When I stand in front of a customer and recommend a design, the value I bring is not that I have read the vendor\u0026rsquo;s reference architecture. Anyone can read that. The value is that I have felt the failure modes I am designing against. I have watched a snapshot chain quietly eat a datastore. I have had a \u0026ldquo;highly available\u0026rdquo; pair fail over and discover the secondary was missing a config change I only made on the primary. I have learned, painfully, that the backup you have never restored is not a backup, it is a hope.\nYou cannot fake that, and customers can tell. The instinct an architect needs — the small voice that says \u0026ldquo;that will be fine on the slide and a disaster at 3pm on go-live day\u0026rdquo; — is built from personal scar tissue, not from coursework. My lab is where I get the scar tissue cheaply, on my own time, on systems where the blast radius is my own evening. By the time a design idea reaches a customer proposal it has often already failed for me at home first. This is the thread running through how I think about the future of technical presales and the journey from proposal to production: the credible architect is the one who has lived in the operational reality, not just the design phase.\nThe lab also forces a breadth that a specialised job never will. In a normal role you are the Citrix person, or the storage person, or the network person, and the layers above and below you belong to someone else. In the lab there is no someone else. To get one local LLM serving traffic through a clean URL I had to be the network engineer (VLANs, DNS, certificates), the sysadmin (the OS, the package versions, the systemd unit that would not start), the SRE (monitoring with Uptime Kuma and Prometheus, the alert that pages me before the family does), the storage admin (where do the models and backups live), and the security person (what is exposed, what is segmented, what secrets are sitting in a .env I had better not commit). You do not get to specialise your way out of understanding the whole. That breadth is exactly what an architect needs and exactly what the org chart denies most engineers.\nRunning a lab as a deliberate learning platform A lab can be a toy. Mine was, for a while — a pile of containers accreting until I could not have rebuilt it if the disk died, which is the honest origin story I told in lessons from building a Docker homelab. The difference between a toy and a learning platform is intent. You have to run it deliberately. There are a few habits that turn the spare-room server into a teacher.\nSet a learning goal, not a feature goal. \u0026ldquo;Stand up Grafana\u0026rdquo; is a feature. \u0026ldquo;Understand how Prometheus actually scrapes and stores metrics, by the time I have Grafana working\u0026rdquo; is a learning goal. Same end state, completely different residue in your head.\nBreak things on purpose. This is the part people skip and it is the most valuable. Pull a disk while a write is happening and watch what your RAID and your filesystem actually do, rather than what the manual claims. Kill a container mid-transaction. Let a certificate expire so you experience the failure signature before you meet it in production. Restore a backup to fresh hardware — not to confirm it works, but to find out all the implicit state you forgot to back up.\nDocument everything, because the lab is also a second brain and the point is to build knowledge instead of documents. An undocumented fix is a lesson you will have to learn again. I keep compose files and notes in Git, in Markdown, plain text that will outlive any platform.\nRun it as production sometimes and as a sandbox other times — and be deliberate about which. The two modes teach different things.\nflowchart TD G[Set learning goal] --\u003e B[Build it yourself] B --\u003e F[Break it on purpose] F --\u003e D[Debug with no answer key] D --\u003e W[Write down what you learned] W --\u003e A[Apply instinct to real designs] A --\u003e GThat loop is the whole method. Notice that the vendor lab only ever lets you do the first two steps, and the certification only tests a sanitised version of the last one. The middle — break it, debug it, write it down — is where every bit of real learning lives, and it is the part formal training cannot give you because it requires genuine failure on a system you care about.\nThe two operating modes are worth being explicit about, because conflating them is how people either learn nothing or break the family internet once too often:\nMode You treat it as You learn Production Real uptime, real backups, change control on yourself Operational discipline, the cost of downtime Sandbox Disposable, spin up and destroy, deliberately fragile Architecture, failure modes, fast iteration The skill is knowing which hat you have on. My DNS, my reverse proxy and the Project Atlas assistant the family now relies on are production: I change them carefully and I back them up properly. A new database engine I am evaluating is a sandbox: I spin it up in a throwaway container, hammer it, and tear it down with its volumes when I am done. Running both modes on the same hardware, and never confusing them, is itself a transferable lesson — it is exactly the discipline a real platform team needs.\nThe limits, honestly I would not be writing in this voice if I were not also willing to say where the argument breaks down. A home lab is not enterprise scale, and pretending otherwise is its own kind of failure. Some lessons only arrive at scale and cannot be reproduced in a spare room. I have never felt the specific pain of a thousand-node fleet where the failure is statistical rather than singular, where a one-in-ten-thousand hardware fault happens daily because you have ten thousand of the thing. I have never managed the blast radius of a change that touches fifty thousand users, or the organisational friction of getting twelve teams to agree on a maintenance window. Those are real skills and the lab cannot teach them. It teaches the engineering; it does not teach the scale, the politics, or the procurement.\nThe lab can also lie to you about robustness. Things that work fine for one user and forty containers fall over at load you will never generate at home. A design that is elegant in the lab can be operationally ruinous when it is multiplied, and I have proposed things that were lovely at home and naive at scale. So I hold lab-derived confidence loosely and pressure-test it against people who have run the big version.\nAnd there is the cost. Time, mostly. A home lab is a rabbit hole with no bottom, and the same property that makes it a great teacher — the lack of guard rails — makes it a great way to lose an entire weekend chasing a problem with zero business value, purely because it annoyed you. There is electricity, there is the WAF, and there is the genuine risk of optimising a system that, being honest, three people use. Learning to walk away from a rabbit hole is itself one of the lessons. I have not fully learned it.\nWhat it is really for Strip it all back and the home lab is not about the hardware, or the services, or even the skills in any narrow sense. It is a machine for manufacturing judgement, by exposing you to consequence. Vendor labs remove the consequence to sell the product. Certifications abstract the consequence into a question with a correct answer. The home lab hands you the consequence directly and unedited, at the merciful scale of your own evening and your own electricity bill, and then asks what you are going to do about it.\nThat is why the differentiation it builds is so durable. Anyone can learn the vocabulary. Anyone can pass the exam. Far fewer people have stood in front of a system that is on fire, that they built, that nobody is coming to fix, and worked the problem until it was out. Every time I do that at home, I get a little harder to replace at work — not because I know more facts, but because I have better instincts about which facts are about to matter and which design is quietly going to hurt someone at 3am.\nSo I will keep breaking my own things on purpose. It remains, by a distance, the best money and the worst-spent weekends I put into this career. If you are early in yours and choosing between another certification and the parts to build something you do not yet know how to run, build the thing. The certificate will tell an employer you can recall. The lab will make you someone worth keeping.\n","date":"2026-06-25T00:00:00Z","permalink":"/infrastructure/the-home-lab-as-a-learning-platform/","title":"The Home Lab as a Learning Platform"},{"content":"What I\u0026rsquo;m Building Now Most of this site is finished thinking. An article gets written when a project has settled enough to have a shape — a problem, a few decisions I can defend, some lessons I got the hard way. This page is the opposite. It is the workbench, not the gallery.\nLast updated: June 2026. That line matters, so let me explain the convention before anything else. This is a living document. I rewrite it every month or two, in place, rather than publishing a new \u0026ldquo;state of the lab\u0026rdquo; post each time. The date above is the contract: if it is stale, treat everything below as a snapshot from then, not gospel. When I update it I bump the date, move anything that shipped down into the changelog at the bottom, and pull something new up from the bench. Future-me only has to keep three lists honest — now, next, and recently done — and the page stays useful without ever becoming an essay.\nThink of this as the hub. Almost everything here has, or will have, a proper write-up elsewhere on the site; I link to those as the canonical version and keep the prose here short and current. If you want the finished argument, follow the link. If you want to know what is actually on my desk this week, stay here.\nHow to read this page It is organised the way I actually think about work: a now/next/later split, plus a deliberate list of things I am not doing. Active work is what I touch in a normal week. The bench is ideas I have committed to memory but not to time. The changelog is recent enough to prove the pattern. And the non-goals are there because, increasingly, I think saying no clearly is the most valuable planning skill I have.\nflowchart LR subgraph Now A[Atlas v2] B[Battery phase 2] C[M365 product] D[VLAN segmentation] E[Homelab as IaC] end subgraph Next F[Self-service portal] G[AI observability] H[Local fine-tune] end subgraph Later I[EV charge coordination] J[Multi-tenant health checks] end Now --\u003e Next --\u003e LaterWhat is active right now Atlas, the next iteration Project Atlas is the recurring brain behind a lot of this, and it is the thing I am most actively pulling apart. The current version works, but it works like a clever search box with a personality. The next iteration is about three things.\nFirst, better retrieval. The naive \u0026ldquo;embed everything, return top-k\u0026rdquo; approach gives confidently wrong answers when the knowledge base gets large, because it retrieves passages that are similar rather than passages that are relevant. I am moving to a hybrid setup — keyword plus vector, with a re-ranking pass — and chunking on document structure instead of fixed token windows. Retrieval is where most of the quality lives. The model is not the product; the context you feed it is.\nSecond, agentic loops. Atlas currently does one-shot tool calls into n8n. I want it to plan, call a tool, look at the result, and decide whether it is done — a proper loop with a stopping condition, not a single hop. The risk is obvious: loops that never terminate, or that burn tokens flailing. So the loop is bounded and every step is logged.\nThird, evals. This is the unglamorous one and the reason the other two are even possible. I am building a small set of golden questions with known-good answers so that when I change the retrieval strategy I can measure whether it got better or just different. Without evals I am tuning by vibes, and vibes are how most AI projects quietly fail.\nThe battery optimiser, phase two The AI battery optimiser has been running in Home Assistant for long enough that I trust its day-ahead charge plan. Phase two is about closing the loop intraday. Right now it commits to a plan each evening based on the solar forecast and the Agile price curve, then largely sticks to it. The next version re-plans through the day as the actual forecast and actual household load diverge from prediction — a cloudy morning should be allowed to change its mind about the afternoon. I am also starting to log forecast-versus-actual properly so I can quantify how much money the optimisation is genuinely saving versus a dumb \u0026ldquo;charge when cheap\u0026rdquo; baseline. If the answer is \u0026ldquo;not much\u0026rdquo;, that is worth knowing too.\nThe M365 health check, becoming a product The Microsoft 365 AI health check started as a script and a good idea. The work now is turning it into something repeatable I can point at any tenant without hand-holding — which is the whole thesis of building repeatable customer health checks. That means parameterising the Graph app registration cleanly, versioning the rule set that decides what counts as a finding, and making the LLM-written report deterministic enough that the same tenant produces a stable report twice in a row. The hard part is not the API calls; it is the judgement layer staying consistent. A health check that grades differently on Tuesday is not a product, it is a mood.\nNetwork segmentation and VLANs The lab still sits on an embarrassingly flat network, and I have written before, in building an AI infrastructure lab at home, that this was a deliberate \u0026ldquo;later\u0026rdquo;. It is now. I am carving the flat network into three VLANs — trust, IoT, and lab — so that a compromised smart plug cannot reach my servers\u0026rsquo; management plane. The work is mostly switch and firewall config, plus the unglamorous job of re-homing every device and fixing the things that quietly depended on everything being on one subnet. mDNS across VLANs alone has cost me an evening.\nMoving the homelab to Git and IaC Too much of my setup still lives as state in running machines rather than as declared intent in a repository. The direction, which falls straight out of lessons from building a Docker homelab, is that the Git repo is the source of truth and the running box is a rebuildable artefact. Compose files already live in Git. Next is the layer below — the host config, the VLAN definitions, the bare-metal host setup — moving toward declarative tooling so I can rebuild a node from scratch without remembering what I clicked eight months ago.\n# the direction of travel: the GPU box as declared intent, not clicked - hosts: gpu-box roles: - nvidia_driver # blacklist nouveau, install and pin the driver - ollama # native service, models on /mnt/nvme - node_exporter # so Prometheus can see it The site itself This very site keeps growing, and not by accident. Building krishaynes.co.uk on Hugo was about owning plain-text, version-controlled knowledge that outlives any platform — the same idea as building a second brain. This now-page is part of that: a deliberate experiment in keeping one document alive instead of letting the site become an archive of frozen posts.\nOn the bench: thinking about, not started These are real ideas with no time committed yet. Writing them down is how I stop them rattling around.\nFine-tuning a small local model. Everything I run locally today is off-the-shelf, picked for the job as I describe in my journey into local LLMs. I am curious whether a small fine-tune — a LoRA on my own corpus of notes and reports — would beat clever prompting for the narrow task of writing in my voice. My suspicion is that it would not be worth it versus better retrieval, but suspicion is not data.\nEV charge coordination. The battery optimiser already reasons about cheap windows. An EV is just a very large, very mobile battery with its own constraints. Coordinating the two against one Agile price curve is an obvious extension, and an obvious way to make a single bad assumption cost real money, so it stays on the bench until phase two of the battery work is solid.\nA self-service health-check portal. The M365 check is currently me running a thing. The natural next step is a front end where a colleague kicks off a run and gets the report, without me in the loop. Worth it only once the engine underneath is genuinely repeatable — otherwise I have just built a nicer way to deliver an inconsistent answer.\nObservability for AI. I have Prometheus and Grafana watching infrastructure. I have almost nothing watching Atlas — token spend, retrieval hit rates, loop lengths, the quality scores from those new evals. As AI becomes infrastructure, it should be monitored like infrastructure. This will probably get promoted to \u0026ldquo;active\u0026rdquo; soon, because every other AI item above is generating telemetry I am currently throwing away.\nRecently shipped The changelog. This is what makes the living-document pattern obvious — things move down here when they leave my desk.\nJun 2026 — Migrated the reverse proxy fully to Caddy with automatic HTTPS; retired the last Nginx Proxy Manager rule. May 2026 — Atlas retrieval moved off fixed-window chunking; first golden-question eval set checked into Git. May 2026 — Battery optimiser day-ahead plan running unattended for a full month, no manual overrides. Apr 2026 — Published the home lab as a learning platform and building a second brain. Apr 2026 — Compose files for all stateful services consolidated into a single Git repo with .env kept out of history. What I am deliberately not doing Saying no is a feature, not an omission, so this list is on purpose.\nI am not running a 70B model as my daily driver. I can, at low quant, but the latency and VRAM cost are not worth it when a well-prompted Qwen2.5 or Llama 3.1 8B handles the actual jobs. The model is not the product.\nI am not building Atlas a slick custom web UI. Open WebUI is good enough, and every hour spent on chrome is an hour not spent on retrieval and evals, which are the parts that actually decide whether it is useful.\nI am not moving anything to a managed cloud LLM for the core knowledge work. The whole point of the local setup is that my notes and tenant data do not leave the building. I will use a frontier model deliberately for a one-off hard task, but the default stays local.\nI am not chasing Kubernetes for the homelab. Docker Compose plus Portainer plus Git is the right amount of complexity for a single-operator lab. k8s would be résumé-driven architecture, and I would spend my evenings operating a control plane instead of building things on top of it.\nAnd I am not publishing a finished version of this page. That is the whole idea.\nThe point of keeping this alive A finished post is a photograph. This is the live feed, and live feeds are honest in a way photographs are not — they show the half-built thing, the idea that has been \u0026ldquo;next\u0026rdquo; for three months, the project I quietly killed. Keeping it current costs me twenty minutes every few weeks and saves me from the slow lie where a site looks busy but nothing on it has moved since last year.\nIf you have landed here from one of the deeper articles, this is the index of where that thread is today. If you have landed here cold, pick any link above; each one is a rabbit hole I have already fallen down so you do not have to. And if the date at the top is more than a couple of months old, send me a nudge — the document being stale is itself a bug, and I would rather hear about it.\n","date":"2026-06-25T00:00:00Z","permalink":"/notes/what-im-building-now/","title":"What I'm Building Now"},{"content":"Why Every Infrastructure Engineer Should Learn Python I have heard the same sentence from good engineers for fifteen years. \u0026ldquo;I\u0026rsquo;m not a programmer.\u0026rdquo; It is usually said with a small note of pride, as if writing code were a separate caste of work that proper infrastructure people are above. I used to half-believe it myself.\nI now think it is the single most expensive belief in the industry.\nThis is not an argument that you should retrain as a software developer. It is an argument that the job has quietly changed underneath us, and that the thing we used to do by hand — clicking through consoles, RDP-ing into boxes, filling in wizards — is increasingly the slow, error-prone, unauditable way to do anything. The fast way is to drive systems by their API. And the most practical language for an infrastructure person to drive APIs with is Python.\nI want to be precise about the claim, because the \u0026ldquo;learn to code\u0026rdquo; crowd has done real damage by overselling it. I am not telling you to learn data structures, design patterns, or how to build a distributed web service. I am telling you to learn enough Python to turn a repetitive, manual, three-hour job into a fifteen-line script you can run again next quarter. That is a much smaller thing, and it is worth more than people expect.\nThe console era is ending For most of my career, the interface to infrastructure was a human sitting in front of a graphical console. You provisioned a VM by clicking through vCenter. You changed a firewall rule in a web UI. You onboarded a user by working down a checklist in three different admin portals. The control plane assumed a person.\nThat assumption is dead, it just has not finished falling over yet. Every serious platform now leads with its API and treats the GUI as a convenience layer on top. Microsoft 365 is Graph with a portal bolted on. AWS is a set of APIs with a console bolted on. VMware, Citrix, your backup product, your monitoring stack — all of them expose the real machinery as an HTTP API or an SDK, and the GUI is just one client of that API.\nThe implication is uncomfortable. If you only know the GUI, you are using the slowest, least repeatable client available, and you are the only client that cannot be version-controlled, scheduled, or handed to a colleague. The engineer who can talk to the API directly is not doing something clever or exotic. They are using the platform the way it was actually designed to be used.\nIf a task is worth doing twice, it is worth not doing by hand the second time.\nThat is the whole thesis. Everything below is detail.\nWhy Python specifically I am not religious about languages. PowerShell is excellent, and if you live entirely inside Microsoft it may be the right first tool. Go is wonderful for shipping a single static binary. But for an infrastructure engineer who wants the broadest possible reach for the least learning, Python wins on a few concrete grounds.\nThe first is ubiquity of SDKs. Almost every vendor ships, or blesses, a Python library. AWS gives you boto3. VMware gives you pyVmomi. Microsoft gives you msgraph-sdk for Graph. And underneath all of them, when no SDK exists, there is requests, which makes any HTTP API reachable in about four lines. You will very rarely hit a system you genuinely cannot talk to from Python.\nThe second is readability. Python code reads close to pseudo-code, which matters enormously when the author is not a full-time developer and the next reader is a tired version of yourself in eight months. You can come back to a Python script after a year and understand it. That is not true of every language.\nThe third, and the one people underrate, is the REPL. You can open a Python prompt, paste in a few lines, and poke at a live API interactively. Print the response. Look at the shape of the JSON. Try one more call. This interactive loop is exactly how an infrastructure person already works — try a thing, look at the result, try the next thing — and it makes learning fast because you are never more than a few seconds from feedback.\n\u0026gt;\u0026gt;\u0026gt; import requests \u0026gt;\u0026gt;\u0026gt; r = requests.get(\u0026#34;https://api.github.com/zen\u0026#34;) \u0026gt;\u0026gt;\u0026gt; r.status_code 200 \u0026gt;\u0026gt;\u0026gt; r.text \u0026#39;Keep it logically awesome.\u0026#39; That is the entire on-ramp. If you can do that, you can read an API.\nScripts are not software, and that is fine Here is the distinction that lets infrastructure people relax, and the one the \u0026ldquo;you must learn to code properly\u0026rdquo; crowd always blurs.\nWriting software is building something that other people depend on, that must handle inputs you have not imagined, that needs tests, packaging, versioning, a support model, and a plan for the day you leave. It is a discipline with real overhead, and that overhead exists for good reasons.\nWriting a script is automating a task you understand, with inputs you control, that you will run yourself, and that fails safely and visibly when its assumptions break. The overhead of \u0026ldquo;proper\u0026rdquo; software engineering mostly does not apply, because the blast radius is small and the operator is you.\nInfrastructure engineers get enormous value from the second thing without ever crossing into the first. A script that pulls a licensing report, a script that checks two hundred mailboxes for a misconfiguration, a script that bulk-updates DNS records from a CSV — none of these need a test suite or a CI pipeline. They need to be correct, readable, re-runnable, and in Git. That is a far lower bar than \u0026ldquo;be a developer\u0026rdquo;, and it captures most of the value.\nThe trap, which I will come back to, is the script that quietly graduates into software without anyone deciding it should. But the line is real, and you are allowed to stay on the infrastructure side of it.\nThe patterns that actually matter When I write automation for real work I am not reaching for clever language features. I am reaching for a small set of patterns that turn a fragile one-off into something I trust against a production tenant. These are the things worth learning properly, because they are what separates a script that works in a demo from one that works at three in the afternoon on a customer\u0026rsquo;s live system.\nIdempotency The most important habit. An idempotent operation produces the same end state whether you run it once or five times. \u0026ldquo;Create this user\u0026rdquo; is not idempotent — run it twice and you get an error or a duplicate. \u0026ldquo;Ensure this user exists with these properties\u0026rdquo; is idempotent — run it as many times as you like and the system converges on the same state. Write your automation as ensure, not as do, and you can re-run it after a failure without fear.\ndef ensure_group_member(client, group_id: str, user_id: str) -\u0026gt; bool: \u0026#34;\u0026#34;\u0026#34;Add user to group only if absent. Returns True if a change was made.\u0026#34;\u0026#34;\u0026#34; members = client.get_group_members(group_id) if user_id in {m[\u0026#34;id\u0026#34;] for m in members}: return False # already correct, do nothing client.add_group_member(group_id, user_id) return True The function checks before it acts, makes no change when none is needed, and tells the caller whether it changed anything. That last detail matters more than it looks — it is the difference between a report that says \u0026ldquo;added 4 users\u0026rdquo; and one that says \u0026ldquo;added 4 users, 196 already correct\u0026rdquo;, which is exactly the kind of evidence you want when you hand the run to someone else.\nRetries and backoff Real APIs throttle you, time out, and occasionally return a 500 for no reason you will ever discover. A script that dies on the first hiccup is useless against anything at scale. The fix is to retry transient failures with exponential backoff, and crucially to honour the API\u0026rsquo;s own Retry-After header when it gives you one.\nimport time import requests def get_with_retry(url: str, headers: dict, max_attempts: int = 5) -\u0026gt; dict: for attempt in range(max_attempts): resp = requests.get(url, headers=headers, timeout=30) if resp.status_code == 429 or resp.status_code \u0026gt;= 500: wait = int(resp.headers.get(\u0026#34;Retry-After\u0026#34;, 2 ** attempt)) time.sleep(wait) continue resp.raise_for_status() return resp.json() raise RuntimeError(f\u0026#34;Gave up on {url} after {max_attempts} attempts\u0026#34;) Pagination This is the one that bites everybody the first time. You call an API, get fifty results, write your report, and ship it — and three weeks later someone notices the report only ever shows the first fifty of four hundred mailboxes. Almost every API pages its results, and Microsoft Graph signals more data with an @odata.nextLink. You have to follow it until it stops. Here is a paginated Graph call with the retry logic folded in, which is close to the shape I actually use when feeding a Microsoft 365 health check:\ndef graph_get_all(path: str, token: str) -\u0026gt; list[dict]: \u0026#34;\u0026#34;\u0026#34;Follow @odata.nextLink and collect every page from a Graph query.\u0026#34;\u0026#34;\u0026#34; headers = {\u0026#34;Authorization\u0026#34;: f\u0026#34;Bearer {token}\u0026#34;} url = f\u0026#34;https://graph.microsoft.com/v1.0/{path}\u0026#34; items: list[dict] = [] while url: page = get_with_retry(url, headers) items.extend(page.get(\u0026#34;value\u0026#34;, [])) url = page.get(\u0026#34;@odata.nextLink\u0026#34;) # None when we run out of pages return items Three patterns — idempotency, retry, pagination — cover most of the bugs I have ever seen in infrastructure scripts. They are not advanced. They are just easy to forget when you are excited that the first call worked.\nSecrets, logging, and Git Three more habits, briefly, because they are the difference between a script you can show a colleague and one you have to apologise for.\nNever put a secret in the code. Read it from the environment or a .env file that is in .gitignore, the same discipline I use for everything in the Docker homelab. If a credential ever lands in a Git history, treat it as compromised and rotate it.\nimport os client_secret = os.environ[\u0026#34;GRAPH_CLIENT_SECRET\u0026#34;] # not a literal in the file Log what you did, to standard output, with timestamps. When a run goes wrong at scale you need to know which of the four hundred operations failed and why, and \u0026ldquo;it errored\u0026rdquo; is not an answer. Python\u0026rsquo;s logging module does this with almost no ceremony and is worth the ten minutes it takes to wire up.\nAnd put the script in Git. Not because you need branching strategies, but because a script that changes a production system and lives only on your laptop is an outage waiting to happen. Version control gives you history, a diff when something breaks, and a way to hand the work to the next person. The same argument I make for building knowledge instead of documents applies to automation: if it is not in version control, it does not really exist.\nWhere this fits together Once you have a handful of these scripts, they stop being isolated chores and start being a toolkit. Most of my automation falls into a few repeating shapes.\nflowchart TD A[Manual console task] --\u003e B{Worth automating} B -- No --\u003e C[Just do it once] B -- Yes --\u003e D[Read the API docs] D --\u003e E[Python script in Git] E --\u003e F[Health checks] E --\u003e G[Bulk changes] E --\u003e H[Reporting] E --\u003e I[Glue between systems] F --\u003e J[Feed data to AI] G --\u003e J H --\u003e J I --\u003e J J --\u003e K[Readable prioritised output]Health checks are the obvious one: pull the current state of a system, compare it against what good looks like, flag the gaps. Bulk changes are next: apply the same change across hundreds of objects, idempotently, with a log of exactly what moved. Reporting turns raw API responses into something a human or a customer can read. And glue is the quiet workhorse — a dozen lines that take the output of one system and feed it into another that was never designed to talk to it.\nThe newest and most interesting category is feeding data to AI. A Python script is the perfect shim between a messy infrastructure API and a language model. It pulls the raw configuration, normalises it into clean JSON or Markdown, and hands that to an LLM to summarise, prioritise, or explain. That pattern is the entire engine behind a repeatable customer health check: Python does the gathering and the model does the narrative. The script is doing the boring, deterministic part, which is exactly where you want determinism, and the model does the part that genuinely benefits from language.\nThis is also where the consultancy work connects to the homelab. The same skill that lets me build an AI consultancy toolkit for the day job is the skill that lets me wire up services at home. It is one capability, not two.\nWhat I got wrong I would be a hypocrite if I made this sound clean. I have made every mistake in the catalogue, and a few are worth naming because they are the ones you will make too.\nThe worst is the load-bearing script. You write a quick fifteen-line thing to solve a problem on a Tuesday. It works. So you run it again. Then someone else runs it. Then it is in a scheduled job. Then a quarterly process depends on it. At no point did anyone decide this thing was production software, but it now is, and it has no tests, no error handling beyond a stack trace, and exactly one person who understands it. This is how a helpful script becomes a liability. The honest fix is to notice the graduation and either harden the thing deliberately or replace it with a proper tool. The dishonest fix, which I have used more than once, is to hope. Hope is not a maintenance strategy.\nThe second mistake is reinventing configuration management. The first time you write a script that loops over a hundred servers ensuring a setting is correct, it feels like genius. By the third such script you have built a worse, untested version of Ansible. There is a real boundary here. If you are managing the desired state of a fleet, use Ansible. If you are provisioning and tracking the lifecycle of cloud or virtual infrastructure, use Terraform. Those tools exist because enough people wrote enough one-off scripts to learn that the problem deserves a dedicated, declarative engine with state tracking. Python is the right tool for orchestration, one-off operations, glue, and anything genuinely bespoke. It is the wrong tool for the jobs that already have a mature declarative answer.\nThe way I now decide is rough but it serves. Is this about the ongoing desired state of many similar things? Reach for Ansible or Terraform. Is this a bespoke flow, a piece of glue, a report, or a thing that has to happen once or on demand? Reach for Python. When in doubt I ask whether I am about to write a state machine, because if I am, someone has already written a better one.\nThe third mistake is over-engineering in the other direction — adding classes, config frameworks, and abstraction layers to a script that runs once a month. The overhead of \u0026ldquo;proper\u0026rdquo; software is overhead you are choosing to carry. For most infrastructure automation, a flat script with good functions and clear logging is the correct level of engineering, and reaching for more is just a different way of avoiding the actual work.\nWhere this goes next For my own work the direction is clear, and it is less about writing more scripts than about treating the ones I have as a real asset. I want the recurring ones in a small internal repository with a sane structure, shared helper functions for the patterns above, and a thin layer of tests around the handful that have genuinely become load-bearing — admitting the graduation rather than hoping it away.\nThe bigger shift is using Python less as the thing that does the work and more as the thing that prepares the work for a model. The collect-normalise-summarise pattern is becoming the default shape of everything I build, because the deterministic gathering belongs in code and the judgement belongs in the model. I expect, over the next year, that more of my scripts shrink to \u0026ldquo;fetch the data, clean it, hand it over\u0026rdquo; and that the interesting logic moves up into the prompt and the workflow. That is not Python becoming less important. It is Python finding its proper place as the reliable plumbing under everything else.\nThe actual point You do not need to become a software developer. You need to stop doing by hand the things a computer should be doing, and the gateway to that is a few hundred lines of unglamorous, readable Python that you keep in Git and are not embarrassed to show a colleague.\nThe \u0026ldquo;I\u0026rsquo;m not a programmer\u0026rdquo; line was never really about programming. It was a way of declaring a job done at the edge of the GUI, when the job had quietly extended past it. The platforms moved to APIs. The work moved with them. The engineers who noticed picked up just enough Python to follow, and the gap between them and the ones who did not is widening every year.\nIt is a small skill. It pays for itself the first afternoon you save. And once you have it, you will wonder how much of your career you spent clicking the same buttons in the same order, getting them subtly wrong, with nothing in version control to show for it.\nLearn enough to drive the API. That is the whole of it.\n","date":"2026-06-25T00:00:00Z","permalink":"/automation/why-every-infrastructure-engineer-should-learn-python/","title":"Why Every Infrastructure Engineer Should Learn Python"},{"content":"Why Most AI Projects Fail I have sat in the room when the demo lands. The screen lights up, the model answers the impossible question, somebody senior says \u0026ldquo;this changes everything\u0026rdquo;, and a budget appears out of nowhere. Six months later I am in a different room, quieter, where the same project is being quietly defunded. Nobody calls it a failure. It just stops being mentioned.\nI have seen this cycle enough times now to be unsentimental about it. Most AI projects fail. Not because the model was bad — the model is almost never the problem — but because the organisation around the model was never built to carry it. The uncomfortable truth is that AI projects fail for the same boring reasons every IT project has always failed: unclear ownership, poor data, no operational discipline, security treated as an afterthought, expectations set by a salesperson rather than an engineer. AI just amplifies all of it, because it fails confidently and it fails in prose.\nThis is a flagship opinion piece, so I am going to take positions rather than hedge. If you want the constructive version of the argument — what AI actually is once the hype burns off — read it alongside AI is becoming infrastructure, which is the spine of everything I believe about this subject. This article is the autopsy. That one is the blueprint.\nThe demo-to-production cliff The single most expensive misunderstanding in this field is the belief that a working demo is a nearly-finished product. It is not. A demo and a production system are not the same thing at different stages of completion. They are different things entirely, and the gap between them is where most projects die.\nA proof of concept is built to succeed. You pick the friendly question, the clean document, the happy path, the one example that makes the room gasp. That is fine — that is what a PoC is for. The dishonesty creeps in when the PoC\u0026rsquo;s success rate gets quietly extrapolated into a production promise. The slick demo answered ten curated questions perfectly. Production has to answer ten thousand questions it has never seen, from users actively trying to break it, against data that changes hourly, while staying within budget, latency, and compliance limits nobody mentioned in the demo.\nI think of it as a cliff rather than a slope. The work does not get gradually harder. It falls off the edge.\nflowchart TD A[Slick PoC] --\u003e B{Cross the cliff?} B --\u003e|Data is messy| C[Garbage answers] B --\u003e|No owner| D[Pilot rots] B --\u003e|No governance| E[Compliance block] B --\u003e|No security| F[Data leak] B --\u003e|No ops| G[Silent failure] C --\u003e H[Quietly defunded] D --\u003e H E --\u003e H F --\u003e H G --\u003e H B --\u003e|All handled| I[Boring reliable system]The PoC lives on the left. Production lives on the right. Everything in the middle is the work that does not demo well, gets no applause, and decides whether the thing survives. When I qualify an AI opportunity now — using the same discipline I describe in the AI consultancy toolkit — my first question is never \u0026ldquo;can we build a demo?\u0026rdquo; It is \u0026ldquo;who is going to operate this at 3am in March?\u0026rdquo; If there is no answer, the project is already failing; it just does not know it yet.\nGarbage in, confident garbage out The oldest law in computing did not retire when the transformers arrived. It got a promotion. \u0026ldquo;Garbage in, garbage out\u0026rdquo; used to produce obviously broken output — a null where a name should be, a report that clearly did not add up. You could see the garbage. An LLM does something far more dangerous: it takes your garbage data and returns it as fluent, confident, well-formatted prose. Garbage in, authoritative garbage out.\nThis is the failure mode that terrifies me most, because it is invisible to exactly the people making decisions on it. If your CRM has three conflicting records for the same customer, a traditional report shows three rows and somebody notices. Ask an LLM and it will smoothly synthesise a single confident answer — and you will never know it averaged three contradictory facts into one plausible lie. The model does not flag uncertainty unless you force it to. It is a fluency engine, not a truth engine, and fluency is precisely what disarms a reader\u0026rsquo;s scepticism.\nMost organisations dramatically underestimate how bad their data is, because they have never had to confront it head-on. The dashboards papered over it. Retrieval rips the paper off. The first time you point a serious retrieval pipeline at a real SharePoint estate, you find duplicate policies, three versions of the \u0026ldquo;current\u0026rdquo; pricing sheet, a 2019 document that contradicts the 2024 one, and an org chart describing people who left. The model dutifully grounds itself in all of it. This is why I am so insistent about doing grounding properly — retrieval is not a magic trick that fixes bad data, it is a magnifying glass held over it.\nThe hard position: if you would not trust a junior analyst to write a report from your data unsupervised, you cannot trust an LLM to do it either, and the LLM is a far more convincing liar.\nMagical thinking and the vendor hype machine A large share of AI failures are baked in before a single line of code is written, in the gap between what the vendor demo promised and what the technology can actually do. The market is awash with magical thinking, and the people fuelling it are not engineers — they are marketing departments with a quarterly number to hit.\nI work in presales. I understand the gravitational pull of a good demo, and I have a low tolerance for the version of it that sets a customer up to fail. When a slide promises \u0026ldquo;AI that understands your entire business\u0026rdquo;, what is actually being sold is a probabilistic text generator with a context window and a retrieval index. Those are genuinely useful. They are not magic. The damage is done when a senior stakeholder, primed by twelve months of breathless coverage, sets a success criterion that no system could meet — \u0026ldquo;it should just know the answer to anything\u0026rdquo; — and then judges a perfectly good tool a failure for not being omniscient.\nIf you cannot say in one sentence what the AI is allowed to be wrong about, you have not scoped the project. You have placed a bet.\nThe cure is unglamorous and it is the presales engineer\u0026rsquo;s actual job: translate the magical expectation into a measurable one before the contract is signed. Not \u0026ldquo;summarise our knowledge\u0026rdquo; but \u0026ldquo;draft a first-pass answer to tier-one support questions, correctly cited, that a human approves before it reaches a customer\u0026rdquo;. The second framing can succeed. The first can only disappoint. This is the same expectation-management discipline that decides whether a deal survives the journey from proposal to production.\nNobody owns it Here is the failure that hides in plain sight. The pilot works. Everyone is pleased. And then the question nobody wants to answer arrives: whose job is this now?\nAn astonishing number of AI initiatives are orphans. They are championed by an enthusiast — often someone brilliant and slightly bored in their actual role — who builds something genuinely clever in their evenings. Then that person gets promoted, or leaves, or simply runs out of evenings, and the system has no product owner, no budget line, no roadmap, and no one accountable when it breaks. It does not get switched off. It just decays. Models drift, the index goes stale, an API key expires, and one day it is quietly wrong about everything and nobody is watching.\nWorse is the shadow-IT version, which I am seeing constantly now. A team, frustrated with the official backlog, wires up a copilot or a third-party AI tool against company data without telling anyone. It works. It spreads. Now there is a production system processing sensitive data that security has never reviewed, that has no owner of record, and that nobody can confidently switch off because three departments depend on it. The convenience that made it spread is exactly what makes it dangerous.\nOwnership is not a nice-to-have. It is the difference between a system and a science project. A real owner means a named human accountable for the outcome, a budget that survives the next reorganisation, and a roadmap that treats the thing as a product with a lifecycle rather than a clever demo frozen in time. I treat this as a qualification gate: no owner, no project. I would rather kill an AI initiative at the proposal stage than let it become an unowned liability eighteen months later.\nThe model is a new attack surface Security is where I see the most dangerous combination: high stakes and low awareness. People are bolting LLMs onto their most sensitive systems while reasoning about security as though it were a normal web app. It is not. An LLM connected to your data and your tools is a new and genuinely strange attack surface, and the old playbook does not fully cover it.\nStart with prompt injection, which is not a theoretical worry — it is the defining vulnerability class of this technology. If your model reads untrusted content — a web page, an inbound email, a document a user uploaded, a calendar invite — that content can contain instructions, and the model has no reliable way to distinguish data it should read from commands it should obey. A support assistant that ingests customer emails can be told, by a customer email, to ignore its rules and exfiltrate the last ten tickets. We have spent decades learning to separate code from data; LLMs cheerfully blend them back together.\nThen there is oversharing through copilots, which is quietly the most common real-world incident. Deploy an enterprise copilot over a document estate where permissions were always \u0026ldquo;a bit loose but nobody could find anything anyway\u0026rdquo;, and you have just built a search engine that finds everything. The salary spreadsheet that was technically shared with the whole company but buried in a forgotten folder is now one polite question away. The AI did not breach anything. It just made existing bad permissions usable, and that is enough to cause a disaster.\nThe defensive posture has to assume the model can be manipulated and that its outputs and tool-calls are untrusted by default:\n# The model is untrusted by default. Constrain it like one. ai_assistant: data_access: enforce_user_permissions: true # the model sees only what the asking user can see no_service_account_shortcuts: true # never run as an all-seeing identity tools: allowlist: [search_kb, create_ticket] deny: [delete, send_external_email, run_shell] human_in_the_loop: [send_external_email] inputs: treat_retrieved_content_as_untrusted: true strip_instructions_from_documents: true logging: prompts: true tool_calls: true retention_days: 90 The principle is old and it still holds: least privilege, defence in depth, log everything, trust nothing. The novelty is that one of the things you must now distrust is the AI itself. I build the underlying platform with this assumption baked in, which is part of why I care so much about getting the infrastructure for AI workloads right — security at the model layer is worthless if the box underneath is wide open.\nGovernance and the questions nobody asked Closely related, and just as fatal, is governance — the set of questions that are boring to ask and catastrophic to skip. Who is allowed to use this? On what data? Where does that data physically live, and does sending it to a US-hosted API breach a contract or a regulation? Can we produce an audit trail of what the model was asked and what it answered? If a regulator asks why a decision was made, can we explain it, or does the answer disappear into a 70-billion-parameter shrug?\nThese questions kill more projects than any technical limitation — usually late, expensively, just before go-live, when legal or compliance finally looks at what has been built and says no. I have watched a genuinely excellent pilot get vetoed at the final gate because nobody had asked, on day one, whether the data was allowed to leave the tenant. It was not. Months of work, dead, over a question that takes an afternoon to answer at the start.\nData residency, model risk, access control, auditability, retention — these are not blockers invented to slow you down. They are the difference between a system you are allowed to run and a system that gets you fined. The same logic that makes me build repeatable health checks for customer environments applies here: governance is not a one-off sign-off, it is a posture you maintain. Bolt it on at the end and it becomes the thing that says no. Build it in from the start and it becomes the thing that lets you say yes safely.\nNo one is on call The last failure mode is the one that separates people who have run production systems from people who have only built demos: operational readiness. Most AI projects have none.\nAsk a team how they will know the model has started giving worse answers and you usually get a blank look. There is no evaluation harness, so quality is measured by vibes and the occasional complaint. There is no monitoring of answer quality, latency, cost, or refusal rate, so a model update that quietly degrades output goes unnoticed for weeks. There is no rollback plan, so when a new prompt or model makes things worse, the only option is to thrash in production. And there is no one on call, because everyone assumed an AI system, unlike every other production system in history, would somehow look after itself.\nIt will not. An AI feature in production is a production system, and production systems need the unglamorous scaffolding: a set of evals you can run on every change, monitoring that watches quality and cost, alerting when something drifts, and a human who is accountable when the alert fires. None of this is novel. It is the same operational maturity we expect of a database or a payment gateway. The mistake is exempting AI from it because it feels new and magical. Magic does not page someone at 3am. Engineering does.\nWhat success actually requires If the failures are mostly old IT failures wearing a new costume, then so are the cures, and I find that genuinely encouraging. We are not short of knowledge about how to run reliable systems. We are just refusing to apply it to AI because the hype told us this time was different. It is not different. It is the same, amplified.\nA project that survives the cliff has, in my experience, the same handful of things in place from the start. There is a named owner who is accountable for the outcome and has a budget that outlives the excitement. The expectations were set by an engineer, not a slide, and written down as something measurable the system is actually allowed to fail at. The data was treated as the real project, because it is — retrieval is grounded properly and the bad data was confronted, not papered over. Governance and security were designed in on day one, with least privilege, audit trails, and a clear answer to where the data lives. And the thing is operated like production: evals on every change, monitoring on quality and cost, a rollback that works, and a human on call.\nNotice what is not on that list. The choice of model. The cleverness of the prompt. The frontier benchmark scores. Those are real, but they are the part everyone already obsesses over, and they are almost never why a project dies. The model is not the product — the system around it is, and the system is built from ownership, data, governance, security, and operations.\nSo when someone shows me a dazzling demo and asks how fast we can ship, I have learned to be the person who asks the unwelcome questions. Who owns this? How bad is the data, really? Where is it allowed to go? What happens when it is wrong, and who finds out? They are not exciting questions. They will not make the room gasp. But they are the entire difference between a system that is still running, quietly and reliably, in three years — and one more orphaned pilot that nobody quite remembers switching off.\n","date":"2026-06-25T00:00:00Z","permalink":"/ai/why-most-ai-projects-fail/","title":"Why Most AI Projects Fail"},{"content":"The Discovery Call: The Questions That Decide a Project Most failed projects were lost in the first conversation, not the last one. By the time a deployment is struggling — the architecture fighting the requirements, the stakeholders disappointed, the scope quietly mutated into something nobody agreed to — the cause is almost always a discovery call that did not dig deep enough months earlier. The technical work usually was not the problem. The understanding underneath it was, and understanding is the entire job of discovery.\nI have come to treat the discovery call as the highest-leverage hour in any engagement. Get it right and the rest of the project is largely execution. Get it wrong and no amount of clever engineering downstream will save you, because you will be building the right thing\u0026rsquo;s evil twin with great precision.\nThe expensive mistakes are not made in the build. They are made in the first conversation, where the wrong assumption gets baked in so quietly that nobody notices until it has set like concrete.\nDiscovery is not requirements gathering The first reframe is that discovery is not a form-filling exercise where you collect a list of requirements and read them back. A requirements list tells you what people think they want. Discovery is about understanding why they want it, what problem it is really meant to solve, and what is actually true about their situation underneath the stated ask. Those are different things, and the gap between them is where projects die.\nPeople are very good at describing solutions and very bad at describing problems. They will tell you they need a specific technology, a particular feature, a named product — because they have already, in their heads, leapt from their problem to a solution, and what they hand you is the solution. The job of discovery is to walk that backwards: past the solution they have proposed, to the problem they are actually trying to solve, because very often the proposed solution is wrong even though the problem is real. If you build what they asked for without understanding why, you will deliver exactly what was specified and entirely miss what was needed.\nThe questions that earn their place A handful of questions do most of the work, and they are deliberately about why and what really, not what technology.\n\u0026ldquo;What does success look like, specifically?\u0026rdquo; Not \u0026ldquo;what do you want to deploy\u0026rdquo; but \u0026ldquo;how will you know this worked?\u0026rdquo; The answers are revealing. Vague success criteria — \u0026ldquo;make it better\u0026rdquo;, \u0026ldquo;modernise the estate\u0026rdquo; — are a warning that nobody has agreed what they are actually buying, and a project with no shared definition of done cannot be finished, only abandoned. Pin this down early or inherit the argument later.\n\u0026ldquo;What happens if we do nothing?\u0026rdquo; This finds the real driver. A genuine, urgent pain produces a sharp answer — a cost, a risk, a deadline, a fire. A waffly answer means the project is a nice-to-have wearing a priority\u0026rsquo;s clothing, and it will lose its funding and its stakeholders\u0026rsquo; attention the moment something genuinely urgent appears. Knowing which one you are in changes how you scope and how hard you push.\n\u0026ldquo;Who is actually affected, and have you spoken to them?\u0026rdquo; The stakeholders in the room are rarely the people who will live with the result. The decision is often made by people who will never use the thing, for people who were never asked. Surfacing the affected-but-absent early prevents the late, expensive discovery that the design everyone signed off makes the actual users\u0026rsquo; jobs worse.\n\u0026ldquo;What have you already tried?\u0026rdquo; This is the question that respects their intelligence and saves you from proposing what already failed. There is almost always history — a previous attempt, a tool they bought and abandoned, an approach that did not take. Understanding what did not work, and why, is worth more than any greenfield assumption, and walking in as though the problem is brand new tells them you have not listened.\nflowchart TD A[Stated ask: a solution] --\u003e B[Why do you want it?] B --\u003e C[What problem is it really solving?] C --\u003e D[What does success look like?] D --\u003e E[What happens if we do nothing?] E --\u003e F[Who is affected? What have you tried?] F --\u003e G[The real project, which is oftennot the one they first described]Listening for what is not said The other half of discovery is not in the questions at all. It is in hearing the things people do not say directly — the hesitation when you raise budget, the stakeholder who goes quiet when a particular system is mentioned, the constraint that gets mentioned once and waved away. These are the signals that the official story and the real story differ, and the real story is the one that determines the project\u0026rsquo;s fate.\nThe discipline is resisting the urge to fill silences and to start solving. The moment you begin proposing, you stop learning, and discovery is for learning. I have trained myself to ask the question, then actually stop talking, because the most important thing the other person says is frequently the thing they add after the pause, once they realise I am genuinely waiting rather than queuing up my pitch.\nWhy this connects to everything else Discovery is where I learn whether a project is the kind of thing I can build well — and increasingly, whether the knowledge and tools I already have make it something I can assemble from validated pieces rather than invent from scratch. A good discovery call does not just scope the work; it tells me which of the problems I have already solved this one resembles, and that recognition is half the value I bring. The compounding I care about across the whole of my work starts here, in understanding the problem well enough to know what it is like.\nIt also sets the honesty of the whole engagement. If discovery establishes a shared, specific understanding of the problem and what success means, then every later conversation has a foundation to stand on. If it does not, every later conversation is really an argument about the thing nobody pinned down at the start, conducted in the more expensive currency of build time.\nThe hour that pays for itself I have stopped thinking of the discovery call as the preamble to the real work and started treating it as the real work, or at least the part that decides whether the real work succeeds. An hour of genuine understanding at the start saves weeks of building the wrong thing well. The questions are not clever and the technique is not complicated. It is mostly the discipline to ask why instead of what, to walk a proposed solution back to its actual problem, and to stay quiet long enough to hear the thing that was not in the brief.\nEvery project I have seen go wrong, I can trace back to a question I did not ask at the beginning. So I ask them now, deliberately, every time — because the cheapest place to fix a misunderstanding is in the first conversation, and the most expensive place is everywhere else.\n","date":"2026-06-18T00:00:00Z","permalink":"/posts/the-discovery-call/","title":"The Discovery Call: The Questions That Decide a Project"},{"content":"Profile Containers and the IOPS Ceiling Nobody Plans For There is a sizing mistake in Citrix design so common that I now assume it is present until proven otherwise. A team plans a session-based deployment, sizes the hosts on CPU and RAM, builds it, tests it with a handful of users, sees plenty of headroom, and ships it. Then real load arrives and the whole thing falls over at a user count well below what the CPU graphs said was possible. The processors are barely warm. The deployment is on its knees anyway. Almost every time, the culprit is the same, and it is not CPU. It is storage, and specifically the IOPS demanded by profile containers.\nThis is the gotcha I keep as canon in my own notes, because I have watched it derail enough projects to treat it as a law rather than a risk.\nCitrix session density is bounded by the profile container IOPS ceiling long before CPU saturates. Size on the storage and the CPU will look after itself; size on the CPU and the storage will ambush you.\nWhy CPU is the wrong thing to size on The instinct to size on CPU is understandable. It is the number everyone quotes, the resource we are trained to think about, and the one the capacity tools put front and centre. For a session host running a dozen users\u0026rsquo; worth of office applications, the CPU genuinely does have room — modern processors chew through that workload easily. So the CPU-based sizing says \u0026ldquo;this box can hold forty users\u0026rdquo;, and on CPU alone, it can.\nBut session density is not bounded by the resource with the most headroom. It is bounded by whichever resource hits its ceiling first, and in a profile-container deployment that resource is almost always storage I/O. The CPU having spare capacity is irrelevant if the disk subsystem is already saturated, the same way a car\u0026rsquo;s top speed is irrelevant in a traffic jam. You sized the engine and got stopped by the road.\nWhat profile containers actually do to your storage The mechanism is worth understanding because it explains why the demand is so much higher than people expect. A profile container — FSLogix-style — holds each user\u0026rsquo;s entire profile in a virtual disk that is mounted when they log on and stays mounted, live, for their whole session. Every profile read and write the user generates, and there are far more of those than people imagine, becomes I/O against that container on shared storage. Multiply by every concurrent user on the host, and again by every host hitting the same storage, and the aggregate IOPS demand climbs steeply.\nLogon is the brutal part. When users log on in bursts — nine o\u0026rsquo;clock on a Monday, the entire shift arriving at once — every one of those container mounts and profile loads lands on the storage simultaneously. This is the dreaded logon storm, and it is a storage event, not a CPU event. The disks are asked for far more IOPS in a few minutes than steady-state operation ever requires, and if the storage cannot deliver them, logons crawl, sessions hang, and users who were promised a fast desktop are staring at a spinner. The host CPU, throughout, sits there with nothing to do.\nflowchart TD A[9am: the whole shift logs on at once] --\u003e B[Every profile container mounts] B --\u003e C[Simultaneous profile reads/writes] C --\u003e D{Can the storage deliver the IOPS?} D --\u003e|no| E[Logons crawl, sessions hang] D --\u003e|yes| F[Fast logons, happy users] G[CPU sits idle throughout] -.-\u003e EHow I size instead I size from a starting assumption about heavy users per CPU pair, and then I treat that number as a hypothesis to validate under real load, not a final answer — because the synthetic numbers lie. A light-touch test with five users tells you almost nothing, because five users do not produce a logon storm and do not stack enough concurrent profile I/O to reach the ceiling. The number that matters is what happens when a realistic population logs on the way they actually do: in a burst, all hitting the same storage.\nSo the storage design comes first. I plan for the IOPS that a full logon storm will demand, not the gentle average of steady state, because the deployment lives or dies on its worst few minutes of the morning. I make sure the profile container storage can deliver that burst, and only then do I trust the CPU-based density numbers — because once storage is not the bottleneck, the CPU headroom is finally real rather than theoretical. Get the order right, storage then compute, and the deployment holds. Get it backwards and the CPU graphs will reassure you right up until the morning everything stops.\nThe reusable rule The reason this is worth writing down once and reusing forever is that it is the same lesson on every engagement, and re-deriving it from scratch each time is exactly the waste a knowledge base exists to prevent. The validated note in my own library says it plainly: single-server session density is bounded by the profile container IOPS ceiling long before CPU saturates; plan to a starting density per CPU pair, then validate under real load because the synthetic numbers lie. When I write the next Citrix design, I do not re-discover this. I link to it, and the sizing starts from a corrected, battle-tested foundation rather than from the CPU-first instinct that keeps catching people out.\nIt pairs directly with the broader thinking in modern Citrix architecture — the principle that the resource you instinctively reach for is rarely the one that constrains you, and good design comes from finding the real ceiling rather than the obvious one.\nThe thing to take away If you remember one sentence about sizing a Citrix session deployment, make it this: the CPU is not your constraint, the profile container IOPS are, and the moment that decides everything is the morning logon storm. Size the storage for that burst first. Validate with a realistic population logging on the way they really do, not a tidy handful in a test window. And write the result down as a durable rule, because you will need it again on the next project, and the version of you doing that design deserves to start from what this one learned the hard way.\n","date":"2026-06-11T00:00:00Z","permalink":"/citrix/profile-containers-iops-ceiling/","title":"Profile Containers and the IOPS Ceiling Nobody Plans For"},{"content":"Idempotency: The Word That Changed How I Write Scripts For years I wrote automation the way most people do: as a sequence of steps that assumed a clean starting point and ran straight through to the end. Create this, configure that, deploy the other. It worked beautifully the first time and was a small disaster every time after, because the real world is never a clean starting point. The script would hit something that already existed, error out halfway, and leave the system in a state that was neither the old one nor the new one — a mess I then had to untangle by hand. The fix was not better error handling. It was a single idea with an awkward name.\nIdempotency means an operation produces the same result whether you run it once or a hundred times. Designing for it, instead of for one perfect pass, quietly fixed more of my reliability problems than any other change I have made to how I write code.\nA script you can only safely run once is not automation. It is a manual procedure that happens to be typed out, and the first time it fails halfway you find out the difference.\nThe brittle way and why it fails The brittle script is a list of imperative actions: do this, then do this, then do this. It assumes each step starts from the state the previous step left behind, and it assumes every step succeeds. Both assumptions are false often enough to matter. Run it on a system where step one\u0026rsquo;s work already exists and it errors. Run it after a previous attempt died at step three and it either refuses to start or, worse, redoes steps one and two in a way that conflicts with what is already there.\nThe deeper problem is that a one-shot script has no concept of the current state. It only knows the actions it wants to take, not the world it is taking them in. So it cannot tell the difference between \u0026ldquo;this needs doing\u0026rdquo; and \u0026ldquo;this is already done\u0026rdquo;, and that single blindness is the source of almost all the fragility. Every \u0026ldquo;it failed halfway and now everything is broken\u0026rdquo; story is really a story about a script that did not know where it was.\nThe idempotent way: declare the destination The shift is from describing actions to describing the desired end state, and then making each step check reality before changing it. Instead of \u0026ldquo;create the directory\u0026rdquo;, you write \u0026ldquo;ensure the directory exists\u0026rdquo; — which does nothing if it already does, and creates it if it does not. Instead of \u0026ldquo;add the line to the config\u0026rdquo;, you write \u0026ldquo;ensure the config contains the line.\u0026rdquo; Each step asks is the world already how I want it? and acts only on the gap.\n# Brittle: assumes a clean start, explodes on the second run os.mkdir(\u0026#34;/srv/app/data\u0026#34;) # FileExistsError if it exists add_line(\u0026#34;/etc/app.conf\u0026#34;, setting) # duplicate on every run # Idempotent: describes the destination, safe to run forever os.makedirs(\u0026#34;/srv/app/data\u0026#34;, exist_ok=True) # no-op if present ensure_line(\u0026#34;/etc/app.conf\u0026#34;, setting) # adds only if missing Written this way, the script can be run twice, ten times, or resumed after a failure, and it always converges on the same correct end state. A step that is already done is a no-op. A step that failed last time gets completed this time. The script stops being a fragile sequence and becomes a description of where you want to end up, with the path to get there computed against reality each run.\nflowchart LR A[Run the script] --\u003e B{Is each thingalready as desired?} B --\u003e|yes| C[Leave it alone] B --\u003e|no| D[Make just that change] C --\u003e E[Same end state every time] D --\u003e E E --\u003e|safe to run again| AWhy this changes everything downstream Once scripts are idempotent, a whole category of operational anxiety disappears. You can re-run after a failure without fear, because re-running is safe by construction — that is the defensive instinct I apply to n8n flows, pushed down into the scripts themselves. You can run the same automation across many machines in different states and trust them all to converge. And you can run it routinely, as a way of asserting \u0026ldquo;the world should look like this\u0026rdquo;, catching and correcting drift every time rather than only when something visibly breaks.\nThis is the foundation under infrastructure-as-code and under repeatable customer health checks, and it is no accident that the same idea sits beneath both. A health check that grades a tenant has to produce the same result run twice in a row, or it is not a check, it is a mood. Idempotency is what makes \u0026ldquo;run it again and trust the result\u0026rdquo; a reasonable thing to say. Without it, every re-run is a gamble.\nWhat it costs The honest cost is that idempotent code is more work to write than the brittle version, because every step has to check the current state before acting rather than barging ahead. \u0026ldquo;Ensure it exists\u0026rdquo; is genuinely more thinking than \u0026ldquo;create it.\u0026rdquo; You have to consider what already-done looks like for each operation, and handle the partial-completion cases the one-shot script ignored. It is more upfront effort for code that, on its successful first run, looks identical in its effect.\nBut that cost is paid once, at writing time, and it buys you reliability forever. The brittle script saves you effort exactly once — the first time you run it — and charges interest on every run after. I have never regretted making a script idempotent. I have repeatedly regretted not bothering.\nThe word that stuck I keep coming back to how much changed from internalising one awkward word. Before, I wrote automation as a hopeful sequence and braced for the cleanup when it failed partway. After, I write it as a description of the destination, safe to run as many times as I like, converging on the same state no matter where it starts. The scripts got more boring and far more trustworthy, which is exactly the trade I want from anything I am going to rely on.\nIf you take one habit from this, make it the reflex of asking, before every step you automate: what happens if this runs again? If the answer is \u0026ldquo;it breaks\u0026rdquo; or \u0026ldquo;it duplicates\u0026rdquo;, you have written a manual procedure in disguise. Make it a no-op when the work is already done, and you have written automation you can actually trust — which is the only kind worth having.\n","date":"2026-06-04T00:00:00Z","permalink":"/automation/idempotency-changed-how-i-write-scripts/","title":"Idempotency: The Word That Changed How I Write Scripts"},{"content":"n8n as the Nervous System of My Homelab Individually, the pieces of my homelab are useful. A notes repository. An AI assistant. A monitoring stack. A pile of scripts. But the value was never in the pieces — it was always going to be in the connections between them, and for a long time those connections did not exist. Things sat in their own boxes, and getting one to trigger another meant me, manually, being the integration layer. n8n is what replaced me in that role, and it turned a collection of separate tools into something that behaves like a single system.\nIf the notes are the memory and Atlas is the brain, then n8n is the nervous system — the thing that carries signals between the parts and lets an event in one place cause an action in another.\nA homelab full of tools that cannot talk to each other is just a more expensive way to do everything by hand. The wiring between them is where the leverage hides.\nWhat n8n actually is n8n is a workflow automation tool you self-host. You build flows visually as a chain of nodes — a trigger, then a series of steps — where each node does one thing and passes its output to the next. A trigger fires (a webhook arrives, a schedule ticks, a file appears), data flows through the nodes, and things happen: an API gets called, a message gets sent, a record gets written, another workflow gets kicked off.\nThe reason I reached for it rather than writing scripts is integration breadth and visibility. n8n speaks to hundreds of services out of the box, and crucially it lets me see the flow — what triggered, what data moved, where it stopped. A pile of cron-driven scripts does the same work but as an opaque tangle; n8n does it as something I can look at, reason about, and debug without grepping logs across five machines. For glue work — connecting things that were never designed to connect — that visibility is worth a great deal.\nflowchart TD A[Trigger: webhook / schedule / event] --\u003e B[Fetch or transform data] B --\u003e C{Decision} C --\u003e|condition A| D[Call an API] C --\u003e|condition B| E[Write a note / send a message] D --\u003e F[Kick off another workflow] E --\u003e FWhat it does for me The flows that earn their keep are the ones that close loops I used to close by hand. A webhook from Atlas that files a new note straight into my knowledge base from a chat message. A scheduled flow that pulls together the inputs for a repeatable health check so the data is staged before I even start. A monitoring alert that does not just notify me but triggers a first diagnostic step automatically. None of these are individually dramatic. Collectively they mean the system does things while I am not watching, which is the entire point of automation.\nThe pattern underneath all of them is the same: an event in one tool becomes an action in another, without me being the wire between them. That is the leverage. Every flow I build is one more thing I no longer have to remember to do.\nThe payload that vanished I will not pretend it has all been smooth, because the bug that cost me the most was the kind that gives you no error to chase — and writing it down is exactly the discipline I keep preaching. A webhook node was receiving data and silently swallowing a malformed payload. Not rejecting it, not erroring, not logging a complaint. The workflow ran, reported success, and did nothing useful, because the data it was supposed to act on had quietly evaporated at the first node.\nIt took me an embarrassingly long time to find, precisely because everything looked fine. The execution showed green. The logs were clean. The downstream nodes had simply received empty input and done their job correctly on nothing. The lesson was one I have since applied everywhere: a workflow that succeeds is not the same as a workflow that did what you wanted. Success is the absence of an error, and the absence of an error is not the presence of the right result.\nI now build flows defensively. Validate the shape of incoming data at the first node and fail loudly if it is wrong. Treat an empty payload as an error, not a quiet pass-through. Add an explicit check that the thing I expected to happen actually happened, rather than trusting a green tick. The happy path lies to you; the only honest workflow is one that distrusts its own inputs.\nWhy this fits the bigger picture n8n matters to me because it is the layer that makes owning the whole stack pay off. Owning every piece is only worth the effort if the pieces can be wired together in ways a vendor would never build — my specific notes feeding my specific assistant triggering my specific automation. That wiring is exactly what no SaaS will ever do for me, because it is worth nothing to anyone but me. Self-hosting the connective tissue is what turns a set of owned tools into an owned system.\nAnd it is self-hosted for the same reason everything central is: the workflows carry my data between my services, and that traffic stays on my network. The nervous system of the lab is not something I am willing to route through someone else\u0026rsquo;s cloud.\nThe honest caveat The risk with a tool this capable is building flows you do not understand and cannot maintain — a sprawl of half-remembered automations that fire mysteriously and break silently. I keep mine deliberately legible: each flow does one comprehensible job, names its nodes clearly, and validates its inputs. A workflow I cannot read at a glance six months later is a liability, not an asset, and I would rather have ten small flows I trust than one baroque one I am afraid to touch.\nBuilt that way, n8n stops being a collection of automations and becomes infrastructure — the quiet layer that carries signals around the lab so that an event here reliably causes the right action there, while I get on with something else. That is what a nervous system is for.\n","date":"2026-05-28T00:00:00Z","permalink":"/automation/n8n-nervous-system/","title":"n8n as the Nervous System of My Homelab"},{"content":"Embeddings in Plain English Embeddings are the quiet engine under every \u0026ldquo;ask your documents a question\u0026rdquo; system, and they are surrounded by more unnecessary mystique than almost anything else in applied AI. Strip the mystique away and the idea is genuinely simple, genuinely powerful, and worth understanding properly — because if you grasp embeddings you grasp why retrieval works when it works and fails when it fails, which is most of what decides whether a system like Atlas is useful or just confidently wrong.\nSo here is the whole idea with no hand-waving: an embedding turns a piece of text into a list of numbers that captures its meaning, in such a way that texts which mean similar things end up with similar numbers. That is it. Everything else is consequences.\nKeyword search matches the words you typed. Embeddings match the thing you meant. The gap between those two is where semantic search lives.\nMeaning as a location The useful mental model is geographic. Imagine every possible piece of text placed somewhere in a vast space, positioned so that things with similar meaning sit close together and things with unrelated meaning sit far apart. A note about \u0026ldquo;reverse proxy certificates\u0026rdquo; and a note about \u0026ldquo;TLS not issuing\u0026rdquo; land near each other, despite sharing almost no words, because they mean nearly the same thing. A note about \u0026ldquo;battery charge scheduling\u0026rdquo; lands a long way off.\nAn embedding is just the coordinates of a piece of text in that space. The model that produces it has read enough language to have learned where things belong — to place \u0026ldquo;the cat sat on the mat\u0026rdquo; near \u0026ldquo;a feline rested on the rug\u0026rdquo; and far from \u0026ldquo;the quarterly revenue forecast.\u0026rdquo; The coordinates are a long list of numbers, hundreds of them, but conceptually they are nothing more exotic than a position. Meaning becomes geometry, and geometry is something a computer can measure.\nflowchart LR A[Text chunk] --\u003e B[Embedding model] B --\u003e C[A position in meaning-spacea list of numbers] D[Your question] --\u003e B B --\u003e E[A position for the question] C --\u003e F[How close are they?] E --\u003e F F --\u003e G[Closest chunks = most relevant]Why this is so useful Once meaning is a position, \u0026ldquo;find me things related to this\u0026rdquo; becomes \u0026ldquo;find me things that sit nearby\u0026rdquo;, and nearness is something you can calculate. That is the entire trick behind semantic search. You embed every chunk of your knowledge base once, storing each one\u0026rsquo;s position. When a question arrives, you embed the question, find the stored chunks closest to it, and hand those back. The system finds relevant material even when it shares not a single word with the query, because it is matching on location in meaning-space rather than on overlapping text.\nThis is why retrieval can do things keyword search simply cannot. Ask \u0026ldquo;how did I fix the login loop?\u0026rdquo; and a keyword system needs your notes to contain those exact words. An embedding system finds the note titled \u0026ldquo;StoreFront authentication redirect bug\u0026rdquo; because it sits right next to your question in meaning-space, words be damned. The same mechanism powers related-article suggestions, duplicate detection, and the whole retrieval layer that makes a pile of notes queryable.\nWhy retrieval lives or dies here Here is the part that matters for anyone actually building something. The quality of an entire retrieval system is largely decided at the embedding step, before the language model ever gets involved. If the embeddings place things badly — if your question lands far from the chunk that actually answers it — then retrieval hands the model the wrong context, and the model produces a confident, fluent, wrong answer built on irrelevant material. The model is not the failure. The retrieval is, and the retrieval is the embeddings.\nTwo things mostly determine whether the embeddings serve you well. The first is the embedding model itself — different models place text with different fidelity, and a good one for your kind of content makes everything downstream better. The second, and the one people neglect, is chunking: how you cut your documents into pieces before embedding them. Embed a whole 2,000-word note as one position and you get a blurry average of everything it says, useless for a question about one paragraph. Chunk too finely and you lose the context that gave each piece meaning. Getting chunking right — splitting on real structure, at a sensible size — does more for retrieval quality than almost any other single change.\nThe honest limitations Embeddings are not magic and pretending otherwise sets you up to be disappointed. They capture similarity of meaning, which is usually what you want and occasionally is not — two texts can be semantically close while one is right and one is wrong, and the embedding cannot tell you which. They are only as good as the model that produced them, and a model trained on general text may place your niche technical content clumsily. And they reflect meaning, not truth: a confidently mistaken note embeds right next to a correct one on the same topic, so retrieval will happily surface your errors alongside your insights.\nThis is why I keep saying retrieval is only as good as the underlying notes. Embeddings find the relevant chunk brilliantly. They have no opinion on whether that chunk is correct. That judgement stays with me, and any system that pretends otherwise — that treats a retrieved passage as true because it was relevant — is building confident fiction with a citation attached.\nWhy I built mine simply When I built the knowledge engine for my own publishing tool, I implemented the vector maths in plain, dependency-light code, because at personal scale the operation is genuinely simple: store each chunk\u0026rsquo;s position, and when a query arrives, measure which stored positions are nearest. That is a handful of arithmetic over a few hundred items, fast enough without any heavyweight machinery. Understanding embeddings as \u0026ldquo;meaning is a position, relevance is nearness\u0026rdquo; is exactly what made it obvious that the maths did not need to be intimidating to be correct.\nThe plain-English summary An embedding turns text into a position in a space where nearness means similar meaning. Store the positions, and finding relevant material becomes finding nearby points — which is how a system answers a question using documents that never contained your words. The whole quality of that system is set at the embedding step, by the model you choose and the way you chunk your text, long before the language model speaks. Get the embeddings right and everything downstream has a chance. Get them wrong and no model, however large, can rescue an answer built on the wrong context.\nIt is one of those ideas that sounds abstract until it clicks, and then it quietly explains half of applied AI. Meaning as geometry. Relevance as distance. Everything else is detail.\n","date":"2026-05-21T00:00:00Z","permalink":"/ai/embeddings-in-plain-english/","title":"Embeddings in Plain English"},{"content":"Picking a Local Model: Quant, VRAM and the 8B Sweet Spot When I started running models locally I made the mistake everyone makes: I reached for the biggest one that would physically fit in my GPU\u0026rsquo;s memory, ran it once, watched it crawl, and concluded that local AI was not ready. The model was not the problem. My selection criteria were. \u0026ldquo;Biggest that fits\u0026rdquo; is the single worst way to choose a local model, and unlearning it was the thing that made the whole setup actually useful.\nThere are three dials that matter — parameter count, quantisation, and the VRAM you have to spend — and choosing well is about balancing them against the job, not maximising any one of them.\nThe biggest model your card can hold is almost never the right model to run on it. The right one is the smallest that does the job well, run fast enough that you actually use it.\nThe three dials Parameter count is the headline number — 8B, 14B, 70B. More parameters generally means more capability, but with steeply diminishing returns for most real tasks and a steeply rising cost in memory and speed. The jump from a 3B to an 8B model is enormous in practice. The jump from a 14B to a 70B is real but far smaller, and it is rarely worth what it costs you in latency for everyday work.\nQuantisation is the dial people understand least and that matters most. A model\u0026rsquo;s weights are originally high-precision numbers; quantisation compresses them to fewer bits — 8-bit, 5-bit, 4-bit — trading a little quality for a lot of memory and speed. The crucial, counterintuitive fact is that a larger model at a lower quant usually beats a smaller model at full precision, for the same memory budget. A 14B model squeezed to 4-bit often outperforms an 8B model at 8-bit, while using similar VRAM.\nVRAM is the hard ceiling everything else has to fit under. It is non-negotiable in a way the other two are not: exceed it and the model either refuses to load or spills into system memory and slows to a crawl. Every choice is really a negotiation against this fixed budget.\nflowchart LR A[The job] --\u003e B{How much VRAM?} B --\u003e C[Pick the largest paramsthat fit at a sane quant] C --\u003e D[Q4 or Q5 usuallybeats full-precision smaller] D --\u003e E[Measure speed on YOUR box] E --\u003e F[Fast enough to actually use?] F --\u003e|no| C F --\u003e|yes| G[Ship it]Why 8B is the sweet spot for most jobs For the bulk of what I do locally — drafting, summarising, classifying, the retrieval-augmented work behind Atlas — an 8B model at a 4- or 5-bit quant is the sweet spot, and I keep being surprised by how rarely I need more. It fits comfortably in a single consumer GPU\u0026rsquo;s memory with room left for context. It runs fast enough that the assistant feels responsive rather than something I set going and walk away from. And it is more than capable enough for tasks that are really about applying knowledge I have retrieved, rather than dredging obscure facts out of the weights.\nThat last point is the key. Most of my local tasks are not tests of how much the model knows. They are tests of how well it can work with the context I hand it. And for \u0026ldquo;read this and do something sensible with it\u0026rdquo;, an 8B model is plenty. The capability I would gain from a 70B model is mostly capability I do not need, bought at a latency I would resent.\nWhen bigger is actually worth it This is not an argument that small always wins. There are jobs where the larger model earns its keep, and refusing to use one out of minimalist principle is the same mistake as always reaching for the biggest. Genuinely hard reasoning — multi-step problems where a smaller model loses the thread — benefits from more parameters. Tasks that lean on broad world knowledge baked into the weights, rather than knowledge you retrieve and supply, reward size. And anything where quality matters far more than speed, and you are happy to wait, can justify the big model.\nThe discipline is matching the model to the task instead of running one model for everything. I keep a small one for fast interactive work and reach for a larger one deliberately, for the specific jobs that need it. The mistake is treating model choice as a single permanent decision rather than a per-task one.\nMeasure on your own box The numbers people quote online are from someone else\u0026rsquo;s hardware, with someone else\u0026rsquo;s context length, doing someone else\u0026rsquo;s task. The only benchmark that matters is tokens per second on your card, with your typical prompt. So I measure. Pull the model, run a representative prompt, watch the actual throughput, and decide whether it is fast enough to be a tool I reach for rather than a demo I ran once.\n# Pull a sensible default and see how it actually performs here ollama pull llama3.1:8b ollama run llama3.1:8b --verbose \u0026#34;Summarise the following in three bullets: ...\u0026#34; # watch the eval rate — tokens/sec on THIS box is the only number that counts \u0026ldquo;Fast enough to actually use\u0026rdquo; is a real threshold and it is personal. Below it, a model is a thing you avoid, and an AI tool you avoid is worthless no matter how clever it is. Above it, the model disappears into the workflow and you stop thinking about it, which is the whole goal.\nThe thing I keep relearning The model is not the product. The context you feed it is, and the speed at which it responds decides whether you use it at all. I spent my first weeks of local AI optimising the wrong variable — chasing parameter count as if it were a score — and the setup only became genuinely useful once I started choosing the smallest model that did each job well and running it fast.\nA well-chosen 8B model at a 4-bit quant, fed good retrieved context and responding quickly, beats a 70B model you are too impatient to wait for every single day. Pick for the job, quantise without fear, measure on your own hardware, and let go of the instinct to maximise the headline number. The headline number was never the point.\n","date":"2026-05-14T00:00:00Z","permalink":"/ai/picking-a-local-model/","title":"Picking a Local Model: Quant, VRAM and the 8B Sweet Spot"},{"content":"Bare Metal, the GPU and the Box That Runs Atlas Every local AI setup eventually comes down to one unglamorous physical fact: a graphics card has to be doing the work, and something has to be feeding it. In my homelab that something is a plain Ubuntu box running Ollama directly on the metal, and the single most consequential decision I made about it was the one I nearly got wrong — whether to virtualise the GPU or hand it the whole machine. Get the driver stack right and you have a fast, rebuildable box that runs models on hardware you own. Get it wrong and you have a host that boots to a black screen and a long evening ahead.\nThis is the story of that box, why bare metal rather than a virtual machine, and the specific pieces of pain that stand between an idea and a working nvidia-smi on the host.\nThe model gets all the attention. The thing that actually decides whether you can run it locally is whether the operating system, the driver and the runtime all agree on how to talk to one PCI device — and that turns out to be the hard part.\nWhy bare metal, not a VM I did not start here. My instinct was to virtualise the GPU box the way I virtualise everything else — put a hypervisor underneath, pass the card through to a guest, and get snapshots and rebuildability for free. So I tried it. I spent a couple of evenings on GPU passthrough, fighting the card into its own isolation group and stopping the host claiming it before the guest could, and I got it working.\nAnd then I ran the models on it, and the honest result was that it was worse. The passthrough layer was fragile in a way that bit me on every kernel and driver update, the performance was measurably down on native, and the whole arrangement felt like a tower of configuration that existed to buy me snapshots I rarely used. So I wiped it, installed Ubuntu straight onto the metal, put the NVIDIA driver and Ollama on the host, and it was simply faster and more stable. Bare metal won, and it was not close.\nThe freedom I thought I needed a VM for — being able to blow the box away and rebuild it — I get a different way. The install is scripted and the working configuration is in my notes, so the host is a rebuildable artefact even without a hypervisor: a documented sequence of steps that takes a fresh Ubuntu install to a working inference box. Models live on NVMe and are re-pullable. The card does the work, the operating system gets out of its way, and nothing sits between Ollama and the silicon.\nThe driver stack that has to line up Running a GPU on bare-metal Ubuntu sounds like it should be a one-line apt install, and the reason it is frustrating is that several separate things all have to be simultaneously correct, and getting most of them right produces exactly the same symptom as getting none of them right: nvidia-smi fails, and nothing tells you which layer is wrong.\nFirst, the open-source nouveau driver has to be out of the way — it grabs the card on boot, and the proprietary NVIDIA driver cannot bind to a card something else already owns, so nouveau has to be blacklisted. Second, if the machine has Secure Boot enabled, the NVIDIA kernel module is unsigned and the kernel will quietly refuse to load it, which looks identical to the driver not being installed at all. Third, the driver version and the CUDA runtime have to match — a driver too old for the CUDA libraries Ollama ships against gives you a card that nvidia-smi can see but that the runtime cannot use. Fourth, Ollama itself has to be able to find the CUDA libraries, or it silently falls back to the CPU and you wonder why a 3090 is generating tokens at the speed of a laptop.\n# Blacklist nouveau so the NVIDIA driver can claim the card echo \u0026#34;blacklist nouveau\u0026#34; | sudo tee /etc/modprobe.d/blacklist-nouveau.conf sudo update-initramfs -u # Install the driver, then confirm the host can see the GPU sudo ubuntu-drivers install nvidia-smi # must list the card before anything else will work # Ollama runs as a native systemd service on the host, on the GPU systemctl status ollama journalctl -u ollama | grep -i cuda # confirm it found the GPU, not the CPU When all four are true, the host boots cleanly, nvidia-smi lists the card, and Ollama loads models straight onto it. When any one is false, you get a black screen, a module that will not load, or a runtime that runs but never touches the GPU.\nflowchart TD A[nouveau blacklisted] --\u003e B[Secure Boot: module signed or disabled] B --\u003e C[NVIDIA driver loads, nvidia-smi works] C --\u003e D[CUDA runtime version matches the driver] D --\u003e E[Ollama finds CUDA on the host] E --\u003e F[Models run on the GPU, natively]What bit me The failure I remember most came after a routine kernel update. Everything had been working for weeks — and then, after apt upgrade and a reboot, nvidia-smi reported it could not communicate with the driver. Nothing I had touched on purpose had changed. The cause was that the new kernel had booted but the NVIDIA module had not been rebuilt against it, so the running kernel had no driver to load. The fix was making sure DKMS rebuilt the module for the new kernel and that the rebuild actually completed, rather than failing silently in the upgrade noise. Obvious in hindsight. Invisible at the time, because the symptom — \u0026ldquo;the GPU has vanished\u0026rdquo; — is identical regardless of which layer of the stack broke.\nThe general lesson is the one a bare-metal driver stack teaches everyone eventually: the diagnostic skill is not fixing the failure, it is localising it. Several things have to be true and the system will not tell you which one is false, so you learn to test each independently — is the module loaded, does nvidia-smi work, does the CUDA version match, does the runtime see the card — rather than changing three things at once and hoping. I keep the working configuration in my notes now, the same way I keep the Caddy reference config, because re-deriving it from scratch is exactly the wasted effort a knowledge base exists to prevent.\nThe payoff On the other side of all that fiddliness is a genuinely good arrangement. The GPU box is a plain Ubuntu machine running Ollama on the bare metal, and it serves the local models behind my AI work — the writing, the embeddings, the retrieval — on hardware that I own and that never sends a token off my network. When a driver update goes wrong, I have the recovery steps written down and I am running again quickly. When I want to rebuild the host entirely, the install is scripted and the configuration is in notes, so the rebuild is mechanical rather than archaeological.\nThat combination — native GPU performance with a box I can rebuild from notes — is exactly the own-the-centre bargain I keep making. The card does the work. Bare metal keeps it fast and simple. And the evenings I lost — first to passthrough, then to a kernel update that ate the driver — bought me a box I have trusted ever since.\nIf you are about to attempt this Do it on a host you can afford to have down for an evening, not on the day you need it working. Resist the urge to virtualise the GPU unless you have a concrete reason to — for a single-card inference box, bare metal is faster, simpler, and far less fragile across updates. Change one thing at a time. Verify each layer of the driver stack independently before assuming the next one is the problem. And write down the working configuration the moment it works, because the version of you that needs it again will have completely forgotten how you got here — and that future stranger deserves better than a black screen and a vague memory.\n","date":"2026-05-07T00:00:00Z","permalink":"/infrastructure/bare-metal-gpu-box/","title":"Bare Metal, the GPU and the Box That Runs Atlas"},{"content":"Why Everything Runs Behind Caddy Now For a long time my homelab had a reverse proxy the way some houses have wiring added one socket at a time over decades — functional, but a tangle nobody fully understood, including the person who built it. Every new service meant another hand-edited rule, another certificate to remember to renew, another chance to typo a hostname and spend an evening wondering why one container was unreachable while everything else was fine. It worked. It did not scale, and more importantly it did not explain itself.\nMoving everything behind Caddy fixed that, and the reason it fixed it is the same reason I like every tool I keep: the configuration lives in one declared place, and the right behaviour is the default.\nA reverse proxy you have to remember to update is a reverse proxy that is one forgotten step away from a broken service. The fix is to stop remembering and start declaring.\nThe problem with the old way The previous setup was a separate proxy with its own management UI, holding its own list of rules in its own database, entirely disconnected from the services it routed to. When I added a container, I added it in two places: the compose file that defined the service, and the proxy that exposed it. Those two places had no knowledge of each other, so they drifted. A renamed service here, a changed port there, and the proxy would happily keep routing to something that no longer existed.\nCertificates were worse. They were issued and renewed as a manual-ish process I half-trusted, and the failure mode was silent — the cert would lapse and I would find out when a browser threw a warning, which is to say when it was already a problem. The whole arrangement violated the principle I care about most: there should be one source of truth, and everything else should be a view onto it. Here there were two sources of truth that quietly disagreed.\nWhat Caddy changes Caddy collapses all of that into one file. Instead of a UI holding routes in a database, the entire routing intent for the lab lives in a single Caddyfile, committed to Git alongside everything else. The file is the configuration — readable top to bottom, diffable, reviewable — and there is no second place for it to drift from. And the feature that made me switch: Caddy does HTTPS automatically. It obtains and renews Let\u0026rsquo;s Encrypt certificates on its own, by default, with no ACME plumbing to wire up. You declare a hostname and you get a valid certificate, full stop.\n# Caddyfile — the whole routing intent, in Git cosmos.example.co.uk { reverse_proxy cosmos:8000 } chat.example.co.uk { reverse_proxy open-webui:8080 } That is the entire thing. Each block is a hostname and where to send its traffic; Caddy reaches the upstreams by container name over the Docker network, terminates TLS, and keeps the certificates valid without being asked. Add a block and commit it and the service is routed and served over HTTPS. Delete the block and the route is gone. The drift that plagued the old setup is structurally impossible, because there is only one list and it is the one in Git.\nflowchart LR A[Browser] --\u003e|HTTPS| B[Caddy] B --\u003e|reads Caddyfile| C[Routing rules in Git] B --\u003e D[cosmos container] B --\u003e E[open-webui container] B --\u003e|ACME, automatic| F[Let's Encrypt] F -.auto-renewed certs.-\u003e BThe DNS-challenge gotcha that cost me an evening I will not pretend it was frictionless, because the part that bit me is the part that bites everyone moving services that are not publicly reachable, and writing it down is exactly the second-brain discipline I keep banging on about. Caddy would not issue a certificate for an internal-only service, and the logs were unhelpful in that specific way that makes you doubt your own competence.\nThe cause was the ACME challenge. Caddy\u0026rsquo;s automatic HTTPS proves you control a domain by answering a challenge over the public internet — and for a service that is not reachable from outside, that challenge can never succeed, so no certificate is ever issued. The fix is to switch that host to the DNS challenge, which proves control by writing a TXT record instead of needing an inbound connection. But the default Caddy binary does not include any DNS-provider modules, so the DNS challenge silently does nothing until you run a build of Caddy with your provider\u0026rsquo;s plugin compiled in and give it an API token.\n{ # global option: prove control via DNS, for internal-only hosts acme_dns cloudflare {env.CF_API_TOKEN} } internal.example.co.uk { reverse_proxy some-internal-service:9000 } The fix was a different Caddy image and one global directive. The lesson was larger: automatic HTTPS is genuinely automatic right up until the assumption underneath it — \u0026ldquo;the world can reach this host\u0026rdquo; — stops holding, and then the skill is knowing which assumption broke rather than assuming the tool is misbehaving. Caddy was behaving perfectly; it simply could not answer a challenge for a door the internet cannot knock on. I now keep a working Caddyfile and the custom-build notes in my second brain precisely so future-me has a known-good version to diff against instead of re-deriving this at midnight.\nWhy this fits everything else The deeper reason Caddy stuck is that it matches how I want the whole homelab to work. Configuration as declared intent, living in Git. The running system as a view onto that intent rather than a separate state that has to be kept in sync by hand. The right behaviour — valid certs, correct routing — as the automatic consequence of declaring what you want, not a manual chore you have to remember.\nIt is the same philosophy as keeping the running machine rebuildable and treating plain text as the source of truth. A new service is now genuinely a few lines in the Caddyfile and a compose up, and I have stopped thinking about the proxy at all, which is the highest compliment I can pay a piece of infrastructure. The best plumbing is the kind you forget you have.\nWhat I would tell past-me If I could send one note back to the version of me hand-editing proxy rules, it would be this: the effort of migrating is real, and it is a fraction of the cumulative effort of maintaining the old way for another year. Put the routing intent in one file, commit it to Git, and let the certificates issue and renew themselves. Then check, once, that any host the public internet cannot reach is set up for the DNS challenge with a Caddy build that actually includes the provider — because it probably is not, and that one piece of plumbing is the difference between an afternoon of triumph and an evening of doubt.\n","date":"2026-04-30T00:00:00Z","permalink":"/infrastructure/why-everything-runs-behind-caddy/","title":"Why Everything Runs Behind Caddy Now"},{"content":"Backups: The Boring Discipline That Saves Your Homelab Everyone agrees backups matter and almost nobody tests them. That gap — between believing you are protected and actually being protected — is where homelabs go to die. I know, because I lived on the wrong side of it for longer than I would like to admit, running nightly backups I had never once restored and feeling safe for reasons that turned out to be entirely imaginary.\nA backup you have never restored is not a backup. It is a hope with a cron job attached. The day you find out whether it works should not be the day you need it.\nNobody wants backups. What people want is restores. The backup is just the unpleasant subscription you pay to have one available.\nThe night it mattered The lesson landed properly when a routine update to a stateful container ate its own database. Not corrupted-but-recoverable — gone, the volume mangled in a way that no amount of staring at it was going to fix. The old me would have lost days of work and a chunk of dignity. The slightly-less-old me, who had finally started testing restores, pulled the previous night\u0026rsquo;s snapshot, restored it into a fresh volume, and was back up in under twenty minutes.\nThe difference between those two outcomes was not the backup. Both versions of me had a backup. The difference was that one version had proven the restore worked and the other was running on faith. That night converted me permanently. Backups stopped being a box I ticked and became a discipline I actually practise.\nThe rule that survives contact with reality The 3-2-1 rule is old, unfashionable, and correct. Three copies of anything you care about. On two different kinds of media or storage. With one copy off-site. It is not clever and that is precisely why it works — it survives the failure modes that actually happen rather than the ones that sound dramatic.\nflowchart TD A[Live data on the host] --\u003e B[Local snapshot on the NAS] A --\u003e C[Encrypted copy off-site] B --\u003e D{Monthly restore test} C --\u003e D D --\u003e|passes| E[Trust earned] D --\u003e|fails| F[Fix it now, not during a disaster]Three copies, because one is no copies and two is one bad coincidence away from none. Two media, because a single failure — a dead disk, a bad filesystem, a fat-fingered rm — should not be able to take every copy at once. One off-site, because fire, theft and flood do not care how good your RAID array is. The whole rule is just a refusal to let any single event reach all of your copies.\nWhat I actually run The implementation is deliberately boring, because boring survives. Stateful data — databases, container volumes, the Markdown that is the real asset — gets backed up nightly to a local NAS, then replicated encrypted to off-site storage. I use restic because it is a single binary, does deduplication and encryption properly, and its snapshots are content-addressed so I can keep a deep history without the size exploding.\n# Nightly: snapshot the stateful volumes, deduplicated and encrypted restic backup /srv/docker/data \\ --exclude-file /etc/restic/excludes.txt \\ --tag nightly # Keep a sensible history without hoarding forever restic forget --prune \\ --keep-daily 7 --keep-weekly 4 --keep-monthly 6 # The line that actually matters — prove a restore works restic restore latest --target /tmp/restore-test --include /srv/docker/data/atlas That last command is the whole point. The first two lines are what most people stop at. The third line — restoring into a throwaway directory and checking the data is real — is the line that converts a hope into a backup.\nStateless versus stateful, and why it changes everything The thing that made backups tractable was separating my world into two kinds of thing. Stateless services hold no unique data — they are defined entirely by their configuration, and that configuration lives in Git. If a stateless container vanishes, I rebuild it from the compose file and lose nothing. I do not back these up at all; I back up the recipe, which is already in version control.\nStateful services hold data that exists nowhere else — the database, the uploaded files, the index. These are the only things that genuinely need backing up, and ruthlessly separating them from the stateless majority shrank my backup problem to a fraction of its original size. Most of my homelab does not need a backup. It needs a git clone and a docker compose up. Only the small, precious core of actual state needs the 3-2-1 treatment, and once you see the line clearly, the whole thing stops being overwhelming.\nThis is the same instinct as keeping configuration declarative across the homelab: the running machine is a rebuildable artefact, and the only irreplaceable thing is the data it holds.\nWhat it costs, honestly The cost of backups is not the storage, which is cheap. It is the discipline of testing, which is tedious and easy to skip because nothing bad happens when you skip it — right up until everything bad happens at once. I put a monthly restore test on the calendar precisely because I knew that \u0026ldquo;I\u0026rsquo;ll test it when I get round to it\u0026rdquo; means never. Twenty minutes a month, restoring a random snapshot into a scratch directory and confirming it is real. It is the most boring recurring task I have and the one I would defend most fiercely.\nThe other honest cost is that backups feel like pure overhead until the exact moment they do not, and that moment is unpredictable. You pay every night for an insurance policy you hope never pays out. The temptation to let it lapse is constant. Resisting that temptation is the entire discipline.\nThe boring conclusion There is no clever ending here, because backups are not a clever problem. They are a discipline problem wearing a technical costume. Three copies, two media, one off-site, and a restore you have actually performed with your own hands. Do that and your homelab moves from fragile to recoverable — from a thing that one bad update can destroy to a thing that shrugs off disasters because every disaster is just an excuse to prove the restore still works.\nI learned this the way most people do, which is nearly the hard way. I got the warning shot instead of the fatal one, and only because I had started testing restores a few weeks before I needed one. Do not wait for your warning shot. Test the restore this week, while it is boring, so that it is boring the day you actually need it.\n","date":"2026-04-23T00:00:00Z","permalink":"/infrastructure/backups-the-boring-discipline/","title":"Backups: The Boring Discipline That Saves Your Homelab"},{"content":"On Owning Your Tools: Self-Hosting as a Practice There is a moment that arrives, sooner or later, with every service you depend on but do not control. The pricing changes. The feature you built your workflow around gets deprecated. The company gets acquired and the product quietly rots. Or it simply shuts down and gives you sixty days to export your data into a format that never quite round-trips. I have lived that moment enough times that I stopped treating it as bad luck and started treating it as a design flaw in how I was working.\nThis is not a manifesto about leaving the cloud. I use plenty of hosted services and I will keep using them. It is a narrower, more practical argument: for the handful of tools that sit at the centre of how you think and work, ownership is worth the cost. Not for ideology. For control.\nIf the thing you rely on most can be taken away by someone else\u0026rsquo;s roadmap, you do not have a tool. You have a dependency that has not failed yet.\nWhat \u0026ldquo;owning\u0026rdquo; actually means Owning a tool does not mean writing it yourself, and it does not mean hosting everything on a server in your spare room out of stubbornness. It means three concrete things. You hold your own data, in a format you can read without the application. You can run the thing on hardware you control, even if you usually do not. And nobody can change the terms underneath you without your say-so.\nMost \u0026ldquo;self-hosting\u0026rdquo; conversations get stuck on the second point — the server, the Docker containers, the reverse proxy — because that is the visible, fiddly, fun part. But the first and third points are the ones that actually matter. I can lose the server. I cannot lose the data, and I cannot tolerate someone else holding the keys to it.\nflowchart LR subgraph Rent[Renting a tool] A[Your data in their format] --\u003e B[Their pricing] B --\u003e C[Their roadmap] C --\u003e D[Their decision to exist] end subgraph Own[Owning a tool] E[Your data in plain text] --\u003e F[Your hardware] F --\u003e G[Your terms] G --\u003e H[Replaceable parts on top] endThe left column is a chain where every link is held by someone whose incentives are not yours. The right column is a chain where you hold every link that matters and treat everything else as replaceable.\nThe practice, not the purity The reason I call this a practice rather than a principle is that it is something you do repeatedly and imperfectly, not a line you draw once and defend forever. Drawing the line absolutely — never use anything hosted, never trust any vendor — is its own failure mode, the homelab equivalent of the document-graveyard absolutism I argue against elsewhere. Ideology makes you worse at the job. The practice is judgement applied case by case.\nSo I ask the same small set of questions every time I adopt something new. If this vendor vanished overnight, what would I lose, and how fast could I recover? Is my data trapped inside this, or could I walk away with it in a form I can actually use? Is this tool at the centre of how I work, or at the edge? The answers decide how much I invest in owning it. A scheduling app at the edge can stay rented. My notes, my knowledge base, my AI assistant — those live at the centre, and the centre I own.\nThis is exactly the line I draw across the homelab and the AI work: the things that hold my thinking run on hardware I control, and the things that are merely convenient can be someone else\u0026rsquo;s problem.\nWhy it is worth the cost Owning your tools is not free, and pretending otherwise is how people end up resenting their own homelab. You pay in time, in the occasional 11pm outage that is entirely your own fault, and in the unglamorous maintenance that a hosted service would have done for you invisibly. I have spent evenings I will not get back chasing a certificate renewal or a container that would not come back up.\nBut the return is real and it compounds. When I control the stack, I can wire pieces together in ways a vendor would never prioritise — my notes feeding my AI assistant, my assistant triggering my automation, my automation filing back into my notes. Nobody is going to build that loop for me, because it is specific to how I work and worth nothing to a mass market. The whole reason Atlas exists, and the reason it can actually do things rather than just chat, is that I own every layer it touches.\nAnd there is a quieter return that is harder to put a number on. When you own your tools, you understand them. The act of standing the thing up, keeping it running, and fixing it when it breaks teaches you how it actually works, which makes you better at the broader craft. A rented tool asks nothing of you and teaches you nothing. An owned one charges tuition in the form of maintenance and pays it back in understanding.\nWhere it goes wrong The failure mode I see most — and have lived — is owning things that should have been rented. Self-hosting your own email is the classic example: enormous effort, real risk, and the payoff is a worse version of something you could have had for a few pounds a month. Hosting a chat server for three people. Running a wiki nobody else will ever edit. The practice is not \u0026ldquo;host everything.\u0026rdquo; It is \u0026ldquo;own the centre, rent the edge, and be honest about which is which.\u0026rdquo;\nThe other failure is treating the server as the asset. The server is the least important part. I can rebuild a server in an afternoon if my configuration lives in Git and my data lives in a format I can restore. What I cannot rebuild is the data and the accumulated understanding. Get the priorities backwards — lavish attention on the hardware, neglect the backups and the plain-text discipline — and you have built a hobby, not a foundation.\nThe real point Self-hosting, stripped of the hobbyist romance, is just refusing to let the tools that hold your most important work be controlled by someone whose interests are not yours. You do it selectively, you do it for the things that sit at the centre, and you accept the maintenance cost as the price of not waking up one day to an email announcing that the thing you built your work on is sunsetting in sixty days.\nI have had that email more than once. I am building, slowly and deliberately, a working life where I never have to read it again about anything that matters. That is the whole practice. Everything else — the containers, the reverse proxy, the GPU box humming in the corner — is just how it gets done.\n","date":"2026-04-16T00:00:00Z","permalink":"/posts/on-owning-your-tools/","title":"On Owning Your Tools: Self-Hosting as a Practice"}]