letsautom8
← Lab notes

Reach and recover: the four slots that keep my lab in my pocket and in one piece

Part 1 gave the lab a record and a protocol. Part 2 gave it a judge, hands and a clock. With those, work starts, gets reviewed and lands without me at the desk.

That raises two new problems. How does a lab that runs without me reach me, and how do I answer it when I'm not at a keyboard? And because the record is now the most important data I have: what happens on the day something breaks? This part is the last four slots. Three are about reach, and one is about getting it all back.

6. Calendar and notifications

Slot 6: why it exists

A system that works on its own has to tell you things. If it tells you everything, you stop reading, and the one message that mattered goes with the rest.

Every channel needs a rule for what it's not for.

The job: show me what's coming, tell me when something happened, and stay quiet the rest of the time.

Ours: three channels, each with one job.

Three channels from the issue tracker. The calendar shows what's coming: a sync every 30 minutes puts each open issue's due date on a separate calendar as an all-day event that doesn't mark me busy, leaves out confidential issues and removes the event when the issue closes. Pushes say something happened: started, due but not started, your review is ready, the inbox has items, a server check found problems. Nothing to report means no push. Decisions are never pushed: I pull them by asking voice what's waiting on me.
The calendar shows what's coming, a push says something happened, and decisions wait until I ask.

The calendar shows what's coming. Part 2 said the board and the calendar share one date. A sync puts every open issue with a due date on a calendar of its own, every 30 minutes. The events don't mark me busy, and each one says "edit the issue, not this event". When an issue closes or loses its date, its event goes. Confidential issues stay off it, because the calendar lives outside the lab.

Pushes say something happened. The lab's own pushes go to a private topic on my phone, and the list of what may push is short: a session started a scheduled issue, an issue is due but couldn't start, the morning review is ready, the inbox sweep found something, a server check found a problem. Only the server check is high priority. (The content channels have their own pushes, with their own rules.) Here are real lines from the logs:

2026-10-18 21:03:40 ntfy   Started <project>#95
2026-10-29 07:30    ntfy   Daily review not written yet
2026-10-31 07:30    ntfy   Your review is ready

The rule that keeps it quiet: nothing to report means no push. No "all good" pings from the lab, and the review pushes at most once a day for each kind.

Decisions are never pushed. An issue that needs me doesn't buzz. I ask "what's waiting on me" when I have a minute, and get the list.

Two of these exist because something got missed. The inbox sweep was added after a security notice from a provider went unseen. It only reads, and when something needs action it pushes a count and one line per item. The review push used to say only that the review was ready. At 07:30 the Mac is awake but I'm not at it, so now it says what to do from the phone: open Claude, start voice, and say "do my review".

Lesson: a push should tell you what to do next from where you are when it arrives, not from where the system is.

7. A voice bridge

Slot 7: why it exists

The questions come when I'm away from the desk: in the car, between things, on the phone. A terminal on a phone is useless and an issue board is slow. In the car, I can only talk.

If you can only reach your system at a desk, you're the bottleneck again.

The job: let me ask what's happening and decide things by voice, from anywhere.

Ours: a small remote MCP server, added as a connector in Claude's phone app. Voice mode then has the lab's tools: read the map, list what's waiting on me, read an issue, check the sites, see which sessions are busy, file something, add a note, decide, hand off, ask the chief of staff, and do the morning review.

I speak to Claude's phone app in voice mode. It calls the connector, a remote MCP server on Cloudflare, which signs me in with OAuth and a passphrase. When a tool runs, the connector calls a small lab API on the home server, or the Mac's API for sessions, each through a tunnel that machine opens outwards, so no port is open, and every request needs an access token and that API's own key. A decision is written to the issue as a comment. Only if I say go, the go label is added by a separate decisions account, whose token lives only on the home server. The session poller on the Mac sees go, checks who added it, and starts the session. If the server can't be reached, voice answers from the last data it saw and says so.
Voice never talks to a session. It writes to the issue, and the issue starts the work.

The connector reads live, and it changes things only after a read-back. That rule is in the server's own instructions to the model: say what you're about to change, and wait for my yes. Then a decision becomes a comment on the issue, "Decided (Marlon, by voice): …". A decision on its own starts nothing, because "park it" and "no" are decisions too. Only when I say go does a go label get added, by a separate account whose token lives only on the home server. That's the label part 1's check trusts, so a voice go is as good as one from my keyboard, and no better.

Nothing in the lab answers the internet directly. The home server and the Mac each open a tunnel outwards, the server for lab data and the Mac for sessions, and every call through either needs two credentials. If the server can't be reached, voice says so and answers from the last data it saw, with when it last changed.

What broke:

Lesson: give the voice the same door as everything else. If voice can do something the issue doesn't record, you've built a second, unreviewed way in.

8. Always reachable

Slot 8: why it exists

Voice covers questions and decisions. Sometimes I need the board itself, a service's page or a log, from the phone, on mobile data, away from home.

The lab should be one private network that comes with me, not a place I have to be.

The job: reach every lab service from the phone or laptop wherever I am, without opening it to the internet.

Ours: Tailscale. The phone, the laptop and the home server are on one private network. The lab's internal names resolve on the phone anywhere, through a second DNS server that listens only on that network and refuses every other lookup, so it's never an open resolver.

Two zones. On the private network: the phone, the laptop and the home server, with the issue board, the lab's services, the container registry and SSH to the production server, reachable only over it. Internal names resolve through a DNS server that answers only on the private network. On the public internet there is one thing: the voice connector, which reaches the lab only through two outbound tunnels, one from the home server and one from the Mac, each needing two credentials. Below, the trap: on the Mac, local network access is allowed per app that started the process, so tools started by a background job got 'no route to host' to the home network, while private-network addresses are never blocked.
One door faces the internet, and it's the voice connector. Everything else is on the private network.

The line between public and private is short enough to list. On the private network only: the issue board, the services, the container registry and SSH to the production server. On the internet: the voice connector, which reaches in only through the two tunnels the home server and the Mac open outwards. Nothing else.

What broke:

Lesson: "reachable" depends on where you are and which app is asking. Test from the phone on mobile data, and from the process that will really run the command.

9. Underneath

Slot 9: why it exists

Nobody picks an assistant for its backups. But slot 1 is now the most important data you have: every task, decision and rule lives there.

This is the slot you only notice when it's missing.

The job: keep the rest running, and bring it back when it doesn't.

Ours: three pieces, each boring on purpose.

Three pieces. Pipelines: three CI runners on the home server; deploys go through a pipeline, SSH is only for restarts, logs and live debugging. Backups: every night restic encrypts on the home server and sends it off-site, where the storage provider only sees ciphertext; it covers the issue tracker and its secrets, databases, configs, what's pulled from the servers, and the Mac's secrets, encrypted on the Mac first; 14 daily, 8 weekly, 24 monthly; integrity checked every night, with 10 percent of the data read back on Sundays. Secrets: in files on the machine that needs them, never in a repo; only the ones that can't be reissued also go in a password manager; an index lists where each lives, never the value. Below, the restore test: a throwaway copy of the issue tracker restored from backup matched the live one, 32 projects, 84 issues, 4 users, 3 groups.
A backup counts once it has been restored. This one was, into a throwaway copy, and every count matched.

Pipelines. Three CI runners on the home server. Anything that deploys goes through a pipeline, and SSH is for restarts, logs and live debugging. The runners can't reach the server's container engine, because that would make every job root on it.

Backups. Every night, restic encrypts on the home server and sends the result off-site, so the storage provider only ever holds ciphertext. It takes the issue tracker and the secrets file it can't be restored without, the databases, configs, what's pulled from the servers, and the Mac's own secrets, encrypted on the Mac before they leave. The servers hold no backup credentials: the home server pulls from them, with a key that can only run one command.

Secrets. In files on the machine that needs them, never in a repo. Only the ones that can't be reissued also go in a password manager. An index lists where each one lives, never the value.

A backup you haven't restored is a hope. The full restore test rebuilt the issue tracker from backup in a throwaway instance and compared it with the live one: 32 projects, 84 issues, 4 users and 3 groups, all matching, and every stored secret readable.

What broke:

Lesson: check this slot the way you'd test anything else: restore it, run it and break it on purpose. A rule nobody has exercised is a hope with a filename.

The whole series in one line each

  1. A single source of truth: every task is an issue.
  2. Labels as the protocol: a few labels say what happens next.
  3. A chief of staff: an agent makes the routine calls, and I review them each morning.
  4. A session per project: one worker per project, one issue at a time.
  5. The on-the-date nudger: a plain script starts work on the day.
  6. Calendar and notifications: three channels, each with a rule for what it isn't for.
  7. A voice bridge: decisions by voice, written to the issue like any other.
  8. Always reachable: one private network, and one door to the internet.
  9. Underneath: pipelines, backups and secrets, restored at least once.

Each slot names a job, not a product, so any of them can be swapped without touching the rest. If you build one, start with slot 1 and add the others as the work asks for them. That's how this one grew.

← All lab notesQuestions? Say hi →
Say hi hello@
letsautom8.dev

Write to me about a post, an experiment, or something you'd like me to try in the lab. If you'd like me on your project, testing or automation, that goes through my company, so you get proper terms and a proper invoice.

Work with me through Shiloah →