* refactor(dream): replace two-phase Dream class with simple cron + process_direct
- Remove the heavyweight Dream class (AgentRunner-based two-phase system)
from nanobot/agent/memory.py
- Delete dream_phase1.md and dream_phase2.md templates
- New dream.md template serves as the consolidation prompt
- Cron callback uses agent.process_direct(prompt, session_key=\"dream\")
instead of agent.dream.run()
- Always performs git auto_commit after execution
- /dream command updated to use process_direct + git commit
- DreamConfig kept for backward compatibility; deprecated fields
(model_override, max_batch_size, max_iterations, annotate_line_ages)
are ignored but accepted in config
- interval_h remains configurable via agents.defaults.dream.interval_h
- Update tests and webui settings to match new architecture
* feat(loop): add ephemeral mode to process_direct, skip history writes for Dream
When ephemeral=True, _state_save skips enforce_file_cap (which calls
raw_archive -> append_history) and consolidator.maybe_consolidate_by_tokens.
This prevents Dream sessions from creating a positive feedback loop where
they process their own output. The session IS still saved to disk.
* fix(loop): skip extra hooks for ephemeral sessions (Dream)
* feat(dream): per-run timestamped sessions with rotation for WebUI
* test(config): restore DreamConfig schedule and alias tests
* fix(dream): include LLM response summary in git auto-commit message
The old two-phase Dream class included the Phase 1 analysis in the git
commit message body. The new single-phase version lost this. Restore it
by extracting resp.content from the process_direct return value and
appending it to the commit message in both the cron handler and the
/dream command.
* fix(test): accept ephemeral kwarg in test_openai_api fake_process
* refactor(dream): merge dream_session.py into MemoryStore
The standalone dream_session.py module only contained three small helpers
that all revolve around MemoryStore concerns (session keys, commit messages,
file pruning). Fold them into MemoryStore as @staticmethod to reduce
indirection and avoid a 35-line module with no independent reason to exist.
* fix(test): address code review — patch correct instance, use actual function
- Fix test_ephemeral_skips_raw_archive to patch loop.context.memory
instead of the fixture's separate MemoryStore instance
- Fix TestDreamCommitMessage to call MemoryStore.build_dream_commit_message
instead of reimplementing the logic inline
- Move Dream helpers in memory.py above the Consolidator section comment
to avoid misleading visual boundary
* fix(dream): gate cursor advancement and restrict tools
maintainer edit: Dream now processes backlog from the oldest unprocessed entries, only advances the cursor after a completed ephemeral run, and uses a restricted file-only tool registry for background consolidation.
* fix(dream): skip idle compact for dream sessions
Dream runs use internal dream:* sessions that are pruned by Dream retention. Exclude them from AutoCompact scheduling, archive execution, and summary injection so idle-session compaction cannot truncate Dream transcripts.
* fix(dream): keep batched history isolated
* feat(dream): tag archived memory for single-phase Dream
---------
Co-authored-by: Xubin Ren <52506698+Re-bin@users.noreply.github.com>
When background consolidation runs with a stale session reference (captured
before AutoCompact replaced the session via compact_idle_session), it could
operate on outdated data. Now, after acquiring the per-session lock, the
method refreshes its session reference from SessionManager.get_or_create().
If the session was replaced, it swaps in the fresh reference before doing
any consolidation work.
This prevents a race where AutoCompact truncates an idle session while a
background maybe_consolidate_by_tokens call is in flight with the old
session object.
Add Consolidator.compact_idle_session(session_key, max_suffix=8) that
performs hard-truncation of idle sessions under the per-session
consolidation lock. This is the single lock-protected path for AutoCompact
to use instead of modifying session state directly, fixing the race
condition between AutoCompact and Consolidator.
Behavior:
- Acquires per-session consolidation lock
- Invalidates cache and reloads fresh from disk
- Splits unconsolidated tail into archive prefix and retained suffix
- Archives prefix via LLM (with raw_archive fallback on failure)
- Persists _last_summary in session metadata on success
- Returns summary text, None on LLM failure, or '' if nothing to archive
Tests: 6 new tests covering prefix archival, empty session timestamp
refresh, (nothing) summary exclusion, LLM failure fallback,
last_consolidated offset, and lock acquisition verification.
#3412 stopped the headline raw_archive bloat but left four adjacent leaks
on the same pollution chain:
- archive() success path appended uncapped LLM summaries to history.jsonl,
so a misbehaving LLM could re-open the #3412 bug from the happy path.
- maybe_consolidate_by_tokens did not advance last_consolidated when
archive() fell back to raw_archive, causing duplicate [RAW] dumps of
the same chunk on every subsequent call.
- Dream's Phase 1/2 prompt injected MEMORY.md / SOUL.md / USER.md and
each history entry without caps, so any legacy oversized record (or an
unbounded user edit) would blow past the context window every dream.
- append_history itself had no default cap, leaving future new callers
one forgotten-cap-away from the same vector.
Changes:
- Cap LLM-produced summaries at 8K chars (_ARCHIVE_SUMMARY_MAX_CHARS)
before writing to history.jsonl.
- Advance session.last_consolidated after archive() regardless of whether
it summarized or raw-archived — both outcomes materialize the chunk;
still break the round loop on fallback so a degraded LLM isn't hammered.
- Truncate MEMORY.md / SOUL.md / USER.md and each history entry in Dream's
Phase 1 prompt preview (Phase 2 still reaches full files via read_file).
- Add _HISTORY_ENTRY_HARD_CAP (64K) as belt-and-suspenders default in
append_history with a once-per-store warning, so any new caller that
forgets its own tighter cap gets caught and observable.
Layer the caps by scope: raw_archive=16K, archive summary=8K,
append_history default=64K. Tight per-caller values cover expected
payloads; the wide default only catches regressions.
Tests: +9 regression tests covering each fix. Full suite: 2372 passed.
Made-with: Cursor
Cover two untested boundaries from #3412:
- _truncate_to_token_budget with positive budget exercises tiktoken
- _MAX_HISTORY_CHARS caps Recent History section in system prompt
Made-with: Cursor
Root cause: when consolidation LLM fails, raw_archive() dumped full message
content (~1MB) into history.jsonl with no size limit. Since build_system_prompt()
injects history.jsonl into every system prompt, all subsequent LLM calls exceeded
the 200K context window with error 1261.
Additionally, _cap_consolidation_boundary's 60-message cap caused consolidation
to get stuck on sessions with long tool chains (200+ iterations), triggering
the raw_archive fallback in the first place.
Three-layer fix:
- Remove _cap_consolidation_boundary: let pick_consolidation_boundary drive
chunk sizing based solely on token budget
- Truncate archive() input: use tiktoken to cap formatted text to the model's
input token budget before sending to consolidation LLM
- Truncate raw_archive() output: cap history.jsonl entries at 16K chars
When chat_with_retry returns an error response (finish_reason='error')
instead of raising an exception, archive() previously treated the error
message as a valid summary and wrote it to history.jsonl, while the
original session data was already cleared by /new — causing irreversible
data loss.
Fix: check finish_reason after the LLM call and raise RuntimeError on
error responses, which naturally falls through to the existing raw_archive
fallback. This preserves the original messages in history.jsonl instead
of losing them.
Fixes#3244
Make Consolidator.archive() return the summary string directly instead
of writing to history.jsonl then reading back via get_last_history_entry().
This eliminates a race condition where concurrent _archive calls for
different sessions could read each other's summaries from the shared
history file (cross-user context leak in multi-user deployments).
Also removes Consolidator.get_last_history_entry() — no longer needed.
Replace single-stage MemoryConsolidator with a two-stage architecture:
- Consolidator: lightweight token-budget triggered summarization,
appends to HISTORY.md with cursor-based tracking
- Dream: cron-scheduled two-phase processor that analyzes HISTORY.md
and updates SOUL.md, USER.md, MEMORY.md via AgentRunner with
edit_file tools for surgical, fault-tolerant updates
New files: MemoryStore (pure file I/O), Dream class, DreamConfig,
/dream and /dream-log commands. 89 tests covering all components.
