* refactor(dream): replace two-phase Dream class with simple cron + process_direct
- Remove the heavyweight Dream class (AgentRunner-based two-phase system)
from nanobot/agent/memory.py
- Delete dream_phase1.md and dream_phase2.md templates
- New dream.md template serves as the consolidation prompt
- Cron callback uses agent.process_direct(prompt, session_key=\"dream\")
instead of agent.dream.run()
- Always performs git auto_commit after execution
- /dream command updated to use process_direct + git commit
- DreamConfig kept for backward compatibility; deprecated fields
(model_override, max_batch_size, max_iterations, annotate_line_ages)
are ignored but accepted in config
- interval_h remains configurable via agents.defaults.dream.interval_h
- Update tests and webui settings to match new architecture
* feat(loop): add ephemeral mode to process_direct, skip history writes for Dream
When ephemeral=True, _state_save skips enforce_file_cap (which calls
raw_archive -> append_history) and consolidator.maybe_consolidate_by_tokens.
This prevents Dream sessions from creating a positive feedback loop where
they process their own output. The session IS still saved to disk.
* fix(loop): skip extra hooks for ephemeral sessions (Dream)
* feat(dream): per-run timestamped sessions with rotation for WebUI
* test(config): restore DreamConfig schedule and alias tests
* fix(dream): include LLM response summary in git auto-commit message
The old two-phase Dream class included the Phase 1 analysis in the git
commit message body. The new single-phase version lost this. Restore it
by extracting resp.content from the process_direct return value and
appending it to the commit message in both the cron handler and the
/dream command.
* fix(test): accept ephemeral kwarg in test_openai_api fake_process
* refactor(dream): merge dream_session.py into MemoryStore
The standalone dream_session.py module only contained three small helpers
that all revolve around MemoryStore concerns (session keys, commit messages,
file pruning). Fold them into MemoryStore as @staticmethod to reduce
indirection and avoid a 35-line module with no independent reason to exist.
* fix(test): address code review — patch correct instance, use actual function
- Fix test_ephemeral_skips_raw_archive to patch loop.context.memory
instead of the fixture's separate MemoryStore instance
- Fix TestDreamCommitMessage to call MemoryStore.build_dream_commit_message
instead of reimplementing the logic inline
- Move Dream helpers in memory.py above the Consolidator section comment
to avoid misleading visual boundary
* fix(dream): gate cursor advancement and restrict tools
maintainer edit: Dream now processes backlog from the oldest unprocessed entries, only advances the cursor after a completed ephemeral run, and uses a restricted file-only tool registry for background consolidation.
* fix(dream): skip idle compact for dream sessions
Dream runs use internal dream:* sessions that are pruned by Dream retention. Exclude them from AutoCompact scheduling, archive execution, and summary injection so idle-session compaction cannot truncate Dream transcripts.
* fix(dream): keep batched history isolated
* feat(dream): tag archived memory for single-phase Dream
---------
Co-authored-by: Xubin Ren <52506698+Re-bin@users.noreply.github.com>
#3412 stopped the headline raw_archive bloat but left four adjacent leaks
on the same pollution chain:
- archive() success path appended uncapped LLM summaries to history.jsonl,
so a misbehaving LLM could re-open the #3412 bug from the happy path.
- maybe_consolidate_by_tokens did not advance last_consolidated when
archive() fell back to raw_archive, causing duplicate [RAW] dumps of
the same chunk on every subsequent call.
- Dream's Phase 1/2 prompt injected MEMORY.md / SOUL.md / USER.md and
each history entry without caps, so any legacy oversized record (or an
unbounded user edit) would blow past the context window every dream.
- append_history itself had no default cap, leaving future new callers
one forgotten-cap-away from the same vector.
Changes:
- Cap LLM-produced summaries at 8K chars (_ARCHIVE_SUMMARY_MAX_CHARS)
before writing to history.jsonl.
- Advance session.last_consolidated after archive() regardless of whether
it summarized or raw-archived — both outcomes materialize the chunk;
still break the round loop on fallback so a degraded LLM isn't hammered.
- Truncate MEMORY.md / SOUL.md / USER.md and each history entry in Dream's
Phase 1 prompt preview (Phase 2 still reaches full files via read_file).
- Add _HISTORY_ENTRY_HARD_CAP (64K) as belt-and-suspenders default in
append_history with a once-per-store warning, so any new caller that
forgets its own tighter cap gets caught and observable.
Layer the caps by scope: raw_archive=16K, archive summary=8K,
append_history default=64K. Tight per-caller values cover expected
payloads; the wide default only catches regressions.
Tests: +9 regression tests covering each fix. Full suite: 2372 passed.
Made-with: Cursor
Follow-up to #3212, fully backward compatible:
- Extract the 14-day staleness threshold as `_STALE_THRESHOLD_DAYS` module
constant and pass it into the Phase 1 prompt template as
`{{ stale_threshold_days }}`. The number lived in three places before
(code threshold, prompt instruction, docstring); now there is one.
- Add `DreamConfig.annotate_line_ages` (default True = current behavior)
and propagate it through `Dream.__init__` and the gateway wiring in
cli/commands.py. Gives users a knob to disable the feature without a
code patch if an LLM reacts poorly to the `← Nd` suffix.
- Harden `_annotate_with_ages` against dirty working trees: when HEAD
blob line count disagrees with the working-tree content length, skip
annotation entirely instead of assigning ages to the wrong lines. The
previous `i >= len(ages)` guard only handled one direction of the
mismatch.
- Inline-comment the `max_iterations` 10→15 bump with a pointer to
exp002 so future blame has context.
- Add 4 regression tests: end-to-end `← 30d` reaches prompt, 14/15
threshold boundary, `annotate_line_ages=False` bypasses git entirely
(verified via `assert_not_called`), length-mismatch defense, and
template-var rendering.
Made-with: Cursor
Three improvements to Dream's memory consolidation:
1. Per-line git-blame age annotations: MEMORY.md lines get `← Nd` suffixes
(N>14) from dulwich annotate. SOUL.md/USER.md excluded as permanent.
LLM uses content judgment, not just age, to decide what to prune.
2. Dedup-aware Phase 1 prompt: reframed as dual-task (extract facts +
deduplicate existing files) with explicit redundancy patterns to scan for.
Validated through 20 experiments (exp-002 prompt + max_iter=15 was best,
averaging -1643 chars/5.4% compression per run).
3. Phase 1 analysis as commit body: dream git commits now include the full
Phase 1 analysis for transparency via /dream-log.
4. max_iterations raised from 10 to 15: 30% improvement over 10 with no
risk; 20 showed diminishing returns (exp-020: -701 vs exp-017: -1643).
Point Dream skill creation at a readable builtin skill-creator template, keep skill writes rooted at the workspace, and document the new skill discovery behavior in README.
Made-with: Cursor
Dream Phase 2 uses fail_on_tool_error=True, which terminates the entire
run on the first tool error (e.g. old_text not found in edit_file).
Normal agent runs default to False so the LLM can self-correct and retry.
Dream should behave the same way.
Replace single-stage MemoryConsolidator with a two-stage architecture:
- Consolidator: lightweight token-budget triggered summarization,
appends to HISTORY.md with cursor-based tracking
- Dream: cron-scheduled two-phase processor that analyzes HISTORY.md
and updates SOUL.md, USER.md, MEMORY.md via AgentRunner with
edit_file tools for surgical, fault-tolerant updates
New files: MemoryStore (pure file I/O), Dream class, DreamConfig,
/dream and /dream-log commands. 89 tests covering all components.
