Why this exists
Remote-desktop infrastructure produces a particular kind of support ticket. A user cannot connect. A session dies after ninety seconds. Authentication works for one directory group and fails silently for another. Attached to the case is a pile of logs from the gateway, the session daemon, the broker, the hypervisor, the servlet container, and the per-connection RDP, VNC or SSH trace.
The answer is almost always in there. Finding it means knowing which line matters, which configuration key produces it, which layer of the stack actually failed, and whether anyone has seen this before. A first-line engineer spends an hour reconstructing context that a senior engineer would recognise in ninety seconds.
That is precisely the shape of problem a language model is good at. It is also precisely the data an enterprise cannot paste into a hosted chatbot. Support logs are dense with hostnames, usernames, internal address ranges, session identifiers and, often enough to matter, credentials that were never meant to be written down anywhere.
For most organizations that single fact is where the conversation about AI-assisted support ends — not because the technology does not work, but because the first step of every vendor demonstration is an unacceptable data transfer. So we inverted the usual order and started from the constraint rather than adding a privacy story afterwards: the model runs on your hardware, or there is no product.
The failure mode that makes most LLM-on-logs systems unusable
A language model is fluent whether or not it is correct. Ask one how confident it is, and it returns a number that tracks how confident its prose sounds. It is not a probability, and trusting it is how a team ships a system that is wrong in a tone of complete certainty.
In a support queue that is worse than useless. A wrong answer delivered with authority sends an engineer down a three-hour dead end, and it erodes trust in every subsequent answer the system produces. One confidently wrong diagnosis costs more credibility than ten correct ones earn.
The second failure mode is subtler. The interesting faults in a virtual-desktop stack are cross-layer. A user reports "I cannot connect". The gateway logs a TLS error. The real fault is a dead guest agent two layers below, and the TLS error is an unrelated internet scanner hitting the public port. An agent that reads only the gateway's logs will confidently blame the gateway, and it will be able to cite genuine log lines while doing so.
What the design had to hold to
- No data may leave the perimeter. Not to a model provider, not to a telemetry endpoint, not on a demonstration path, not as a fallback when the local model is unavailable.
- The evidence bundle is untrusted input. It is an archive assembled on a machine we do not control, and it must be treated as hostile until proven otherwise.
- It must run without a GPU budget or a procurement cycle. A system that requires new hardware approval will not be piloted.
- It may not act. No service restarts, no configuration changes, no ticket closures, nothing customer-facing.
- Everything it did must be inspectable. A black box that reads your logs is not an improvement on the problem it was brought in to solve.
An investigator, three ways of knowing, and a gate the model does not control
A collector runs against the environment and produces an evidence bundle: logs, container states, configuration and health output from every layer of the stack. The bundle lands on a support case. An engineer clicks Analyze. From there the agent opens the bundle safely, redacts it, investigates it, corroborates what it found, scores the result, and either posts a drafted diagnosis or hands the case to a human and says why.
The investigator
Most "AI log analysis" is a single prompt with a log file pasted into it. That fails for two reasons: the bundle is far larger than any sensible context window, and the model has no way to follow a thread. avocadoctor instead gives the model a bounded set of read-only tools and lets it run its own investigation across multiple turns.
| Tool | What it does |
|---|---|
list_files | Enumerate the bundle |
read_file | Read a bounded window of a file |
search_files | Search across the bundle |
get_container_states | What was actually running, and what had died |
get_config | The effective configuration, and its delta from a known-good baseline |
lookup_code_error | Trace an error string back to the source that emits it, across all three platforms |
Every tool is read-only, reads redacted content only, and returns a size-bounded result. The model has a turn budget and a byte budget on its accumulated history, so a long investigation cannot silently outgrow its context window. If it exhausts that budget it returns a no-evidence result that the gate rejects. It never returns a guess.
Three ways of knowing, and it must agree with itself
A code index. The source of every platform in the stack is indexed. When a log line carries an error string, the agent traces it to the code path that emits it and the configuration key that triggers it — more than ten thousand indexed error strings across Guacamole, OpenUDS and Proxmox. The symptom stops being a mystery and becomes a location.
A correlated log timeline. Every plane of the bundle is decoded with its own grammar, timezone-normalised, and merged onto a single timeline. A deterministic correlator then reduces that to a layer verdict: which plane is culpable, and which signals are red herrings. This happens before the model speaks, and it is arithmetic rather than inference.
A case base. Prior resolved cases and public knowledge, held as a hybrid vector and graph memory — pgvector for similarity, Apache AGE for structure. Similarity proposes candidates; the graph narrows them by how they actually relate.
The pipeline, and the boundary around it
The egress boundary
Every model call in the system passes through one enforcement point, and it does two things. It fences the endpoint: default-deny, with only loopback and private-range addresses permitted at all, and every cloud provider refused before a byte leaves the process. And it applies a residual-risk backstop, re-scanning everything outbound so that anything resembling a secret which survived the redaction ruleset is withheld from the model, with the withholding recorded in the audit trail.
Critically, in the investigator both checks run on every single turn, not once at the start. A model that opens a new file on turn five is fenced on turn five.
Redaction that fails closed
The evidence bundle is scanned for archive-level attacks before anything is read — path traversal, symlinks, zip bombs, declared-versus-actual size — and every file is digested. A versioned ruleset then masks secrets, tokens, keys, PEM material, password hashes and connection credentials across every plane. What the model receives is not the log file. It is the redacted projection of it.
INPUT
10:41:22.318 [http-nio-8080-exec-7] WARN o.a.g.a.LocalAuthenticationProvider -
Authentication attempt from 10.21.4.17 for user "[email protected]" failed.
guacd[7213]: ERROR: Unable to connect to RDP server rdp-host-04.corp.example.com:
certificate validation failed
REDACTED — EXACTLY WHAT THE WORKFLOW RECEIVES
10:41:22.318 [http-nio-8080-exec-7] WARN o.a.g.a.LocalAuthenticationProvider -
Authentication attempt from <IP_1> for user <USERNAME_1> failed.
guacd[7213]: ERROR: Unable to connect to RDP server <FQDN_1>: certificate validation failedThe diagnostic content survives. The identifiers do not. Operators can run the ruleset against their own text in the console before trusting it with anything.
Why the model does not get a vote on its own confidence
This is the core of the system and the part we would defend hardest. The model's self-reported confidence controls nothing. It is recorded for diagnostics and ignored for decisions.
The commit decision is a deterministic score computed from the evidence actually retrieved: how close the nearest prior case really was, how much corroborating evidence was retrieved, and how far independent sources agree. Absent features contribute exactly zero, and there is no model output anywhere in the expression.
score = w_sim · similarity + w_ev · evidence + w_agr · agreementOn top of it sits the grounding rule: no evidence, no commit. An answer with no resolvable citations scores zero, regardless of its content and regardless of its tone. And citations must actually resolve — a claim's cited text is the real stored log line retrieved from the artifact store, not the model's paraphrase of it. A citation that does not resolve is dropped, and dropping enough of them collapses the score.
Three independent vetoes
Scoring is necessary but not sufficient. A wrong answer can still be well-evidenced if the model cites genuine lines in support of a false conclusion — a failure we hit repeatedly in development. So three deterministic guards can veto the model outright.
- Component corroboration. The answer blames a component the evidence contradicts. Killed.
- Layer corroboration. The answer blames a layer the deterministic correlator has already ruled out with high confidence. Killed.
- Red-herring corroboration. The answer's mechanism is a signal the parser has already flagged as noise. Killed.
When a guard fires, the findings are withheld, the score collapses to zero, and the agent retries. If it still cannot ground an answer, it hands off to a human and says so plainly, in writing, on the ticket.
A committed diagnosis, and a refusal
When the evidence is there, the agent commits and the diagnosis lands on the ticket in a fixed structure: what it thinks is wrong, the reasoning, a suggested fix explicitly marked not applied, and the arithmetic that authorised it.
The refusal matters more
A live case on our own stack. Users could not connect to a Windows 11 desktop, and the logs were dominated by TLS handshake failures at the edge. The agent made nineteen tool calls across two attempts: it checked what was running, checked the configuration for drift, and then walked down the stack — gateway, broker, hypervisor, and finally into the guest itself.
It concluded:
The SSL handshake failures in nginx are caused by an external internet scan targeting the
public HTTPS port, not a configuration or product issue. The root cause is
a Windows guest VM that is running but has a non-responsive QEMU guest agent, preventing the
UDS broker from assigning a valid IP address. The UDS actor reports ip:unknown,
and the upstream connection is never established.
Three things are worth drawing out. It looked past the loudest signal: the TLS errors were the most visible thing in the bundle by an order of magnitude, and they were noise. It crossed layers: the symptom appeared at the edge and the fault was in the guest, three layers down. And it rejected a plausible memory — the case base offered a prior Guacamole handshake bug at 0.72 similarity, a superficially excellent match and exactly the kind of thing a retrieval-augmented system pastes back with confidence. The corroboration guard rejected it because the evidence in this bundle contradicted it.
And then the part that matters most. Having produced a coherent, specific, cross-layer answer, the agent withheld it and handed the case to a human, because it could not resolve its citations back to stored evidence. The gate does not care how sure the model sounds. It posted its ranked options to the ticket, each marked as a reviewed suggestion not applied, told the engineer plainly why it would not commit, and got out of the way.
An agent that would have committed there is an agent that will eventually commit something wrong with exactly the same fluency.
Everything it did is inspectable
Orchestration is Temporal, so every analysis is a durable workflow. A crash mid-investigation resumes rather than restarts, and the full event history of every run is replayable after the fact.
Alongside these sit the investigation timeline, the redaction report and tester, the evidence trail that resolves each claim back to the line that produced it, the gate view with its read-only what-if panel, a hash-chained audit trail of every decision and every external write, and OpenTelemetry spans across the investigation, each turn and each tool call.
Where it runs
The whole pipeline — retrieval, embeddings, reasoning and the operator console — fits on a single desktop-class machine. That is a deliberate target rather than a limitation. It means the answer to "where does our support data go?" is "the box in the corner of your server room", and it means a pilot needs no GPU budget and no procurement cycle.
| Hardware | AMD Ryzen AI Max+ 395, 128 GB unified memory |
| Reasoning | Qwen3-30B-A3B, a 30B mixture-of-experts model, served locally over an OpenAI-compatible API |
| Embeddings | BGE-M3, served locally |
| Data | PostgreSQL with pgvector and Apache AGE |
| Orchestration | Temporal — durable, replayable workflows |
| Console | React |
| Live stack | Proxmox 8.4.19 · OpenUDS 4.0.0 · Apache Guacamole 1.5.2 · Windows 11 guests · nginx |
| Ticketing | Salesforce, behind an adapter — the platform is not built around one vendor |
What it does not do
avocadoctor runs today against our own virtual-desktop estate, diagnosing real tickets from a real support queue with Salesforce as the ticketing front end. It is a pilot on our own infrastructure rather than a system with paying customers behind it, and we will say otherwise the day that changes.
- It never remediates. No service restarts, no configuration changes, no ticket closures. It diagnoses and drafts; a person acts.
- It never publishes anything customer-facing. The write-back is an internal, agent-only comment plus structured fields for the support team.
- It defers when it cannot prove an answer — by design, and readily. That is the trade described above, and we would make it again.
- It is not built around one ticketing vendor. Salesforce is an adapter, and so is the first-party portal. Vendor differences live at the boundary, never in the reasoning core.
Lessons
The engineering that distinguishes a usable AI system from a demonstration has very little to do with the model. It is the boundary drawn around it. Every consequential decision in avocadoctor — the single egress point, the fail-closed redaction, the deterministic score, the three vetoes, the requirement that a citation resolve to a stored line — is a decision about what the model is permitted to influence, and none of them would be improved by a better model.
The second lesson is that the refusal is the product. It is straightforward to build an agent that always answers. What is difficult, and what makes such a system safe to put in front of an engineer, is building one that can recognise when it has not proved its case and say so. That capability cannot be added later, because it has to be structural: the model must not be able to talk its way past it.
The third is that constraints stated first produce better architectures than constraints retrofitted. Had we built the agent and then attempted to add a privacy story, we would have arrived at a perimeter with exceptions in it. Starting from "the model runs on your hardware or there is no product" forced every downstream decision — which model, how much context, how to make a modest local model behave — and the result is a system whose central claim is structural rather than promised.
The reasoning core, the redaction boundary, the deterministic gate and the audit trail are platform-independent. Supporting another stack means writing adapters — a log grammar, a code index, a case corpus — rather than building a new product.