11 — Diagnostics and Field Triage#

Who this is for. Anyone standing in front of a station that is behaving strangely, with limited time and no appetite to re-read sixty thousand words of subsystem documentation to find the one paragraph that explains it.

What you'll learn. Nothing new. Every mechanism named below is documented in full elsewhere in this book — this chapter is an index by symptom into chapters that are organised by subsystem. Find what you are seeing, jump to the section named, and come back here for the next symptom. Every symptom/cause pair below was checked against the current source before it was written; this repository's own history is that roughly two-thirds of unverified doc claims turn out to be phantom, and a troubleshooting chapter that guesses is worse than none.

The operator-facing version of this table is docs/manual/operator/07-troubleshooting.md — read that first if the person in front of the station is an operator, not an engineer. This chapter goes one level deeper: the mechanism, the file, and the line.

11.1 Symptom index#

Symptom	First thing to check	Mechanism, explained in	Escalation
Screens frozen, values not updating, but the link badge is green	Is one screen frozen or all of them? One screen: check whether its own `DispatcherTimer` is still ticking (a swallowed exception can silently stop it). All screens, UI unresponsive to clicks: this is a UI-thread hang, not a PLC problem — see §11.2	ch. 02 §2.5 (the UI-hang sentinel); ch. 03 §3.4 (freshness gating)	Setup → Diagnostic Console → Export Bundle; if the UI is truly unresponsive, a support bundle cannot be taken from inside the app — collect `%ProgramData%\Ring\logs\application.log` and a screenshot instead
A "stale" chip or dimmed value appears on many tank cards at once, not just one	The heartbeat state, not the freshness window — one heartbeat problem starves every heavy poller at once, so it looks like a blanket outage rather than one broken tag	ch. 03 §3.6 (`Starved` vs `Disconnected`); §11.3	PLC Health drawer for the exact state; if `Starved`, this is very likely `MainTask InhibitTask := Yes` on the controller pre-cutover
The link badge is flapping between colours, or reads a state you don't recognise	Which of the five states it actually is, and for how long	ch. 03 §3.6 — the state machine, its four numeric thresholds, and why `Starved` is a separate diagnosis from `Disconnected`	If it settles within ~12 s, this is expected transient behaviour, not a fault
An operator presses a button and nothing on the controller changes, with no error shown	What the write path returns is not uniform — several rails report `true`/success even when the write was suppressed	ch. 04 §4.1, "What a blocked write returns is not uniform"; §11.4	Grep the application log for `SUPPRESSED` around the time of the click (ch. 07 §7.13) before assuming the write failed at the wire
Hold cannot be resumed from any station	`PC_Write_Integer[30].1` is a sealed coil — level-sensitive, not edge-triggered — so a stuck TRUE re-asserts Hold every scan	ch. 04 §4.6 (the F013 exemption and the stuck-bit table); §11.5	`PcWriteInteger30StrandedBitMonitor`'s advisory on the PLC Health drawer names the stuck bit; if it does not clear, the bit must be cleared from Studio 5000
Ring appears not to launch — no window, or a window flashes and closes	A second instance may already be running, or classified as still starting	ch. 01 §1.7; §11.6	Task Manager → check for an existing `Ring.exe`; if genuinely absent, check `%ProgramData%\Ring\logs\application.log` for a startup error
PLC comms degrade while a capture/probe script is running	EtherNet/IP session exhaustion — a CompactLogix caps at roughly 4–8 sessions per source IP, and Ring's own seven pollers already use several	ch. 01 §1.0 Vocabulary, "EIP session"; ch. 03 §3.9	Stop the extra script; the condition self-heals in roughly 30 s once the extra sessions close
PLC IP address, `ReadOnlyMode`, SMTP credentials or cost rates revert after a rebuild	`CopyAppSettingsToOutput` unconditionally overwrites the output `Config\appsettings.json` from the source tree after every build	ch. 02 §2.3, "Configuration is written back into the build output"; ch. 09 §9.1	Re-apply the local overlay, or keep it in `Config/appsettings.local.json` where the Debug-only copy step (and the Release keep-and-warn behaviour) actually preserves it
Database writes seem blocked; export, backup or factory reset seem to hang or silently do nothing	The maintenance latch is held — a restore or reset is in progress, or ended abnormally	ch. 05 §5.5; §11.7	Confirm no restore/reset dialog is actually still open behind another window before assuming a hang; the latch is process-wide, not per-window
Thousands of test failures appear from nowhere on a machine that was fine yesterday	Free disk space, checked first, not last — parallel build lanes filling `C:` have produced exactly this failure mode before	ch. 08 §8.1	`Get-PSDrive C`, or the Windows equivalent, before opening a single test's stack trace
A red banner appears under the top of the window at startup and stays for the session	"Degraded" has one precise meaning: database initialization failed, and `PlcWriteGuard.ForceReadOnlyForSession()` was pulled — a one-way latch for the rest of the session	ch. 02 §2.4; §11.8	Read the banner text exactly — it names the reason. The recovery is a restart, not a retry from inside the running session
A supervisor edited a config value through the app and it does not seem to be sticking, or a build immediately reverted it	Two different mechanisms depending on which: the atomic-write helper failing silently is rare and would log an error; the far more common cause is the build-output copy above	ch. 02 §2.3 (writer classes); same row as the `CopyAppSettingsToOutput` symptom above	Check `%ProgramData%\Ring\logs\application.log` for a write error first; if none, it is almost certainly the build-output revert
A write-path PR looks correct but you are not sure what it owes	Not a live-station symptom, but the single most common "something feels wrong" moment in review	ch. 04 §4.12 and the quick reference at the top of ch. 04	—

11.2 Screens frozen, but the link badge is fine#

This is two different failure modes wearing the same description, and the fix is to tell them apart before doing anything else.

If it is one screen, and every other screen is fine: the heartbeat and the pollers are healthy, so this is local to that screen's own DispatcherTimer. Every PLC-facing screen owns a DispatcherTimer (typically 1000 ms) that reads a static snapshot holder and paints named controls (ch. 02 §2.5). An unhandled exception inside a tick handler can, depending on where it is thrown, stop that timer without crashing the app. Navigating away and back re-creates the screen (or re-seeds a cached one via its Loaded handler) and is the fastest recovery; if it recurs on the same screen, that screen's own tick handler is the place to look, not the PLC link.

If the whole app is unresponsive to clicks, not just showing stale numbers: that is a UI-thread hang, and it is a different mechanism from freshness gating entirely. MainWindow runs a background sentinel that pings the dispatcher and expects a "pong" every 5 s (UiHangPingIntervalMs, MainWindow.xaml.cs:122), feeding Ring/Services/UiHangDetector.cs, which classifies the UI thread's state from that single tick. A station in this state cannot take its own diagnostic bundle — Setup → Diagnostic Console needs a responding UI — so the fallback is to collect %ProgramData%\Ring\logs\application.log directly and note the exact time of the hang.

Neither of these is what freshness gating (ch. 03 §3.4) is for. Freshness gating governs whether a value that arrived is painted as current or refused as stale; it says nothing about whether the screen's own timer is still running at all.

11.3 "Stale" everywhere at once, vs. stale in one place#

A dimmed value or a "stale" caption on one card, next to fresh values on its neighbours, is ordinary freshness gating doing its job — see ch. 03 §3.4.

A "stale" indication on every tank card, every group screen, at the same time is a different signal: it means PlcHeartbeatConnectionTracker.AllowHeavyPlcPolling() has gone false, which every heavy poller gates on (ch. 03 §3.2). Check the PLC Health drawer for the exact state:

Stale (4–12 s without a heartbeat change) does not stop heavy polling — this state alone should not blank every screen.
Starved (12 s+ without a heartbeat change, while reads still succeed) does. This is the diagnosis operators need: the network is fine, Ring is fine, and the controller is not advancing the heartbeat word. The usual pre-cutover cause is MainTask InhibitTask := Yes on the controller.
Disconnected (12 s of no successful read, or three consecutive failures) is the more familiar full comms-loss case.

Starved and Disconnected look similar from the operator's chair — "the plant stopped updating" — but they are different diagnoses with different next steps, which is exactly why the state machine keeps them separate (ch. 03 §3.6).

11.4 A write silently does nothing#

Before assuming the write reached the controller and was ignored, or never reached it at all, read ch. 04 §4.1, "What a blocked write returns is not uniform". Some rails return true on a suppressed write specifically so the operator flow continues and the operator is told separately that nothing was sent — true is not proof of a landed write anywhere in this codebase.

The fastest way to tell suppression from silence is the log, not the screen: every suppressed write goes through PlcWriteGuard.LogBlocked, which emits a line containing the literal word SUPPRESSED (ch. 07 §7.13 has two real examples of the exact line shape). If nothing in the log around the time of the click contains SUPPRESSED, the write was not blocked by the read-only gate — check the per-feature Enable* flag (ch. 04 §4.2), the applicable commissioning hold (ch. 04 §4.3), and the endpoint and heartbeat gates in that order, per the verified per-rail table in ch. 04 §4.7.

11.5 Hold cannot be resumed#

PC_Write_Integer[30].1 (Hold) drives a sealed coil — the rung is level-sensitive, not edge-triggered, so a bit that is stuck TRUE re-asserts Hold on every scan and Resume cannot break the seal from any station, including the PanelView and the hardwired inputs (ch. 04 §4.6). This is precisely why F013 exists: the clear (PlcTagWriter.WriteClear) is exempted from the heartbeat gate specifically so a degraded link cannot prevent Ring from at least attempting to clear the bit.

PcWriteInteger30StrandedBitMonitor is the read-only sentinel that watches for exactly this — it raises a throttled advisory on the PLC Health drawer naming the stuck bit, with operator-facing text along the lines of "PC HOLD request bit is stuck: press Hold, then Resume. Until it clears the plant re-enters Hold every scan and cannot be resumed from any station." It only detects and advises; it never clears the bit itself. If pressing Hold then Resume does not clear it, the bit has to be cleared from Studio 5000 directly — there is no software recovery from inside Ring for a truly stuck sealed coil.

11.6 Ring appears not to launch#

Ring is single-instance, but "not launching" has more than one cause. App.xaml.cs creates a named mutex ("RS360-RingwoodApp-SingleInstance", no Global\ prefix — see ch. 02, "The single-writer problem, stated honestly" for why that prefix's absence matters for remote-access seats specifically). On a second launch, Ring classifies the existing instance rather than simply assuming it is safe to kill:

Already starting — the first instance's own boot marker is still held, which means it is genuinely mid-OnStartup (possibly running its own integrity check and database backup). The second launch exits quietly without disturbing it. Do not kill the first instance in Task Manager in this state — that is the case this classification exists to prevent.
A responsive window was found — it is focused, and the second launch exits.
No window at all, or the window failed two liveness probes five seconds apart — the second launch treats the first as genuinely stuck and may tell the operator it is safe to end it.

If Task Manager shows no Ring.exe at all and nothing appears, check %ProgramData%\Ring\logs\application.log for a startup-time error — a broken appsettings.json produces a message box before the main window, which can look like "nothing happened" if the box is behind another window.

11.7 Database writes seem blocked#

DatabaseMaintenanceLatch is a process-wide "the database file is being swapped or deleted right now" flag, held by DatabaseWriterQuiesce.StopAll() during a restore or a factory reset and released only by ResumeAll() (ch. 05 §5.5). It exists specifically because a modal message box around the swap runs a nested dispatcher pump, and two things can otherwise undo the quiesce mid-swap (the PLC auto-reprobe, and the dashboard's own refresh ticks) — so the latch is a belt-and-braces stop on top of the quiesce, not a redundant one.

If exports, backups or writes seem stuck, first confirm there genuinely is no restore or reset dialog open behind another window — the latch is never released on a path that ends in application shutdown, so an operator who closed Ring mid-restore rather than letting the dialog finish can leave a stale expectation about what state the database is in, even though the latch itself cannot outlive the process.

"Degraded" has exactly one meaning in this codebase: database initialization failed, so batch history, the audit trail and the alarm log are unreliable for the session (ch. 02 §2.4). The response is not just a banner — StartupDegradedState.ApplyDatabaseInitOutcome(false, …) also pulls PlcWriteGuard.ForceReadOnlyForSession(), the one-way latch: once pulled, nothing in the running session can re-arm writes short of a restart, because Configure computes _readOnly = readOnly || _forcedReadOnly and nothing clears _forcedReadOnly in production code (that reset method exists only for test cleanup — see ch. 08 §8.1 and the tripwire test in ch. 08 §8.6).

Read the banner text exactly — the class records the reason first-detail-wins, so the wording tells you which of the startup failures actually happened. The recovery is a restart, not a retry from inside the session: a successful database init on the next boot changes nothing else, and the configured ReadOnlyMode resumes deciding normally.

A milder, related degradation exists for the tank roster specifically: if the roster file fails to load, startup falls back to the legacy 4-tank default, warns, and latches RosterWriteIndex.StorageWritesLockedByRosterLoadFailure even though the substituted roster looks perfectly valid — see ch. 02 §2.4 and ch. 04 §4.3, "The roster locks".

11.9 The first five minutes#

In roughly the order that costs the least time:

Read the banner or dialog text exactly, if one is showing. Both the degraded-startup banner and the countdown warning dialog name their own cause; do not paraphrase from memory before checking the literal text.
Open the PLC Health drawer. It carries the live heartbeat state, the stranded-bit advisory (if any), and the clock-skew warning — three different sentinels in one place. (ch. 03 §3.6, §11.5 above, ch. 03 §3.7.)
Check the comm log (Setup → DisplayCommunicationForm) for the specific tag and time in question — it shows suppressed writes as well as real ones, marked "ReadOnly suppressed". (ch. 03 §3.7.)
Grep %ProgramData%\Ring\logs\application.log for SUPPRESSED (a blocked write), ERROR (most failures worth escalating), or the specific tag name. Remember the log's own retention limit at Debug level is short (ch. 07 §7.13) — the further back you need to look, the less likely it is still there.
For a write that may have reached the controller, check the write evidence journal (%ProgramData%\Ringwood\Ring\plc-write-evidence.jsonl) — but remember its scope: only writes dispatched through PlcSetpointWriteQueue are recorded there. A batch-start write, a PC_Write_Integer[30] pulse, a formula-bank rewrite or an inventory write leaves no line in this file; its record is the application log, the comm log, and its own typed outcome (ch. 04 §4.9).
For a build or test-suite anomaly, check free disk space before opening a single failing test — see ch. 08 §8.1.
If none of the above explains it, scripts/Invoke-ProductionGate.ps1's own summary (artifacts/production-gate/production-gate-summary.json) is the authoritative statement of what currently passes; do not trust a number quoted from memory over it.

Back to: book index.

Verified against#

Every symptom/mechanism pair in this chapter was checked against these files on 2026-09-01, in addition to the chapters it cross-links:

Ring/Services/PLC/PcReadIntegerSnapshot.cs, PlcHeartbeatConnectionTracker.cs, PlcWriteGuard.cs, PcWriteInteger30StrandedBitMonitor.cs, MomentaryPulse.cs, PlcCommunicationLogService.cs, PlcWriteEvidenceJournal.cs, FormulaBankWriteInterlock.cs · Ring/Services/DatabaseMaintenanceLatch.cs, DatabaseWriterQuiesce.cs · Ring/Services/StartupDegradedState.cs · Ring/Services/UiHangDetector.cs · Ring/Services/Logger.cs · Ring/Services/SecondLaunchAdvice.cs · Ring/Views/App.xaml.cs · Ring/Views/MainWindow.xaml.cs · Ring/Ring.csproj (CopyAppSettingsToOutput target) · docs/PLC_FACTS.md · docs/manual/operator/07-troubleshooting.md · CLAUDE.md · CONTRIBUTING.md