smoido
a4afa55580
Acts on docs/incident-mode-switch-corrupts-live-sessions.md, which is added here as the record of why. The report identifies a consequence that was not modelled. A failed call is recoverable; a *successful* one may not be. If a running session takes even one completion from the provider being switched to - which happens when that mode matches the base URL it already had cached - that provider's message-id format lands in its transcript. OpenRouter issues `gen-<epoch>-<rand>` where Anthropic issues `msg_...`, and native Anthropic then refuses to resume the session at all, with a 400 naming previous_message_id. The only way back is to truncate the transcript, losing every turn after the cut. That happened here, and was fixed by hand. Two changes follow. Sessions are now settled before the write, not reported after it. A switch with anything running stops, names the sessions, explains what is about to happen to them, and offers restart (the only answer that ends with everything on the mode the bar now claims), close, proceed anyway, or abort - defaulting to abort. Non-interactively it refuses outright unless given --yes. The old after-the-fact reporter is deleted rather than left as a second, contradictory account. `claude-mode repair-session` replaces the hand surgery: it finds a project's transcripts, reports which are resumable, and on --apply backs the file up and truncates to the last Anthropic-issued message. Verified against the real corrupted transcript from the incident - it reproduces the manual cut exactly, 1921 lines to 1813, dropping the two `gen-` completions and the error placeholders after them, leaving a transcript that ends on a genuine msg_ id. It refuses a transcript written to in the last 90 seconds, since that one belongs to a session still running. The panel passes --yes, having already asked in its own card, and that card now names the transcript risk rather than only the inconvenient one. Requirement 4 of the report - documenting the mechanism - landed in 9c301e1; the README now carries the unrecoverable half as well.
143 lines
7.7 KiB
Markdown
143 lines
7.7 KiB
Markdown
# Incident: switching modes mid-session corrupts and can permanently break that session
|
|
|
|
**Date:** 2026-08-30/31
|
|
**Affected session:** `1fffec9e-e2e3-4255-828c-a9ccfd8f631d` (project `/home/smoido/Work`)
|
|
**Severity:** High — silent, mid-task corruption with no warning, and one failure path is not recoverable by normal means (requires manual transcript surgery).
|
|
|
|
## Summary
|
|
|
|
The user had a live, running Claude Code session open in `~/Work`. While that
|
|
session was mid-task, `claude-mode` was used to switch the active provider
|
|
away from `anthropic` and later back to `anthropic`. The running session was
|
|
not restarted in between. This corrupted the session's credentials, and — because
|
|
the session had picked up a different provider's message-ID format in the
|
|
meantime — it became permanently unable to resume under native Anthropic auth,
|
|
throwing a hard API error on every resume attempt.
|
|
|
|
This must not be possible to trigger silently. Whatever ships next needs to
|
|
either prevent it, warn loudly before it happens, or make recovery automatic.
|
|
|
|
## Root cause / mechanism
|
|
|
|
`claude-mode` writes provider config into `~/.claude/settings.json`
|
|
(`env` block: base URL, model IDs, etc., plus `apiKeyHelper` pointing at
|
|
`claude-key-helper.sh`). A **running** `claude` process only reads part of
|
|
this at startup:
|
|
|
|
- **Static, cached at startup:** `ANTHROPIC_BASE_URL`, the model ID env vars,
|
|
the rest of the `env` block. A running session keeps whatever it started
|
|
with here — switching modes does *not* change these for an
|
|
already-running process.
|
|
- **Not cached — re-fetched on a timer:** the credential. Claude Code
|
|
re-invokes `apiKeyHelper` periodically (`CLAUDE_CODE_API_KEY_HELPER_TTL_MS`
|
|
is present in the 2.1.251 binary), and `claude-key-helper.sh` answers based
|
|
on whatever `~/.claude-mode/state.json` says *at the moment it's called* —
|
|
not at session startup.
|
|
|
|
So a mode switch reaches into a live session through the one part that was
|
|
never cached:
|
|
|
|
| switching the global mode to... | what the *running* session gets on next credential refresh |
|
|
|---|---|
|
|
| `anthropic` | the helper returns nothing (native anthropic auth is expected to need no helper) → session has no credential at all |
|
|
| any other provider/preset | the new key, but the session is still pointed at the **old** base URL (cached at startup) → that endpoint rejects the new key |
|
|
|
|
Either way, the session starts failing API calls from the moment the TTL
|
|
next expires — mid-turn as easily as between turns. The one case that does
|
|
*not* break: switching between two presets of the *same* provider that share
|
|
a `keyRef` (same key, same endpoint survives).
|
|
|
|
`claude-mode status` and `claude-mode health` only ever report the *global*
|
|
config's current state. Neither one has any way to know a specific already-running
|
|
session exists, let alone that it's about to be (or has been) knocked over.
|
|
|
|
## What happened to this specific session (concrete failure chain)
|
|
|
|
1. Session `1fffec9e` was live in `~/Work`, working on `claude-code-switcher`
|
|
itself.
|
|
2. User ran a switch to the `openrouter` "default" preset (via `claude-mode`)
|
|
to test something, while that session kept running.
|
|
3. The running session's next few turns actually succeeded against
|
|
openrouter (its cached base URL now matched, since the switch happened to
|
|
land on the provider its env pointed at) — those turns got real
|
|
completions back, but with **openrouter's message-ID format**
|
|
(`gen-<timestamp>-<id>`), not Anthropic's (`msg_<id>`).
|
|
4. A later call failed outright and Claude Code recorded a synthetic
|
|
client-side placeholder turn (`"model":"<synthetic>"`,
|
|
`"isApiErrorMessage":true`, `"error":"unknown"`) as the last message in
|
|
the transcript — this is the visible "session disconnected mid-task"
|
|
symptom.
|
|
5. User switched the global mode back to `anthropic`.
|
|
6. Resuming session `1fffec9e` (`claude -r 1fffec9e-... -p "..."`) now fails
|
|
unconditionally with:
|
|
```
|
|
API Error: 400 diagnostics.previous_message_id: must be the `id` from a
|
|
prior /v1/messages response (starts with `msg_`)
|
|
```
|
|
because native Anthropic's API requires `previous_message_id` to be a
|
|
real Anthropic-issued ID, and the last message(s) in this session's
|
|
transcript are not (`gen-...` or the synthetic placeholder ID).
|
|
7. **There is no supported way to resume past this.** The only fix found was
|
|
manual surgery on the session's `.jsonl` transcript file: locate the last
|
|
message that still has a genuine `msg_...` id (in this case, several
|
|
turns earlier, at a clean `end_turn` boundary well before the switch was
|
|
even tested), back up the original file, and truncate the transcript to
|
|
that point. That rolls the session back to its last-known-good state and
|
|
makes it resumable again — at the cost of permanently losing every turn
|
|
after that point (the actual code changes from that later work were
|
|
separately safe in git, but the chat narrative was not recoverable).
|
|
|
|
## Impact
|
|
|
|
- Silent corruption: nothing warns the user before or during the switch that
|
|
a live session exists and is about to break.
|
|
- Depending on timing, the break can be "just" an auth failure (annoying,
|
|
session still resumable once you're back on the mode it started with) **or**
|
|
a hard, unrecoverable-by-normal-means failure (if the session round-tripped
|
|
through a different provider's ID format before failing) that requires
|
|
hand-editing a JSONL transcript to fix.
|
|
- This applies to *every* running `claude` process on the machine at switch
|
|
time, not just the one in the foreground shell — CLI, VS Code, desktop, any
|
|
of them.
|
|
|
|
## Requirements for the fix
|
|
|
|
1. **Detect running sessions before switching.** `claude-mode` should scan
|
|
for live `claude` processes (and ideally which project/cwd each belongs
|
|
to) before performing a switch.
|
|
2. **Warn or block, don't silently proceed.** At minimum, print a clear
|
|
warning naming the affected session(s)/PIDs/cwds and what will happen to
|
|
them (credential will be pulled out from under them on the next TTL
|
|
refresh). Consider requiring `--force` (or an explicit confirmation) to
|
|
proceed while sessions are running, and defaulting to "abort" otherwise.
|
|
3. **Prefer a safe path when sessions are detected:** e.g. offer to let the
|
|
user gracefully end/save those sessions first, or clearly instruct them
|
|
to restart affected sessions immediately after the switch completes.
|
|
4. **Document the real mechanism** (this file's "Root cause" section) in the
|
|
tool's own help/README, replacing any prior claim that a switch "does not
|
|
affect a running session" or that sessions "keep talking to the old
|
|
provider until restarted" — both are wrong; the credential moves under
|
|
them regardless.
|
|
5. **Make the unrecoverable failure mode recoverable.** Add a `claude-mode
|
|
repair` (or similar) command that:
|
|
- finds a given session's transcript,
|
|
- locates the last message with a valid `msg_...` id,
|
|
- backs up the original file,
|
|
- truncates to that point,
|
|
so this doesn't require manual `jq`/`head`/`grep` surgery next time.
|
|
This is the exact procedure used to fix session `1fffec9e` above.
|
|
|
|
## Repro steps (for verification once fixed)
|
|
|
|
1. Start a `claude` session in some project directory, mid-task.
|
|
2. In a separate shell, run `claude-mode <some-other-provider>` (something
|
|
with a different base URL/key format from the session's current mode).
|
|
3. Let the running session's next API-key TTL refresh happen (or just issue
|
|
another prompt in it) — observe the call fail.
|
|
4. Switch back: `claude-mode anthropic`.
|
|
5. Try `claude -r <that-session-id> -p "hi"` — currently fails with the
|
|
`previous_message_id` 400 error if the session ever got a non-`msg_`
|
|
completion in between. The fix should make step 2 impossible (or clearly
|
|
confirmed) rather than needing step 5's failure to be caught after the
|
|
fact.
|