mirror of
https://github.com/HKUDS/nanobot.git
synced 2026-08-04 16:38:49 +00:00
The recent-history section injected into the system prompt was capped by character count (_MAX_HISTORY_CHARS = 32_000). Characters are a poor proxy for tokens: ~32k chars of English is ~8k tokens, but the same char count of CJK text or code can be far more, so the cap could let the section blow well past its intended size on non English/code-heavy histories. Add a reusable truncate_text_to_tokens() helper (reusing the tiktoken cl100k_base encoder already used elsewhere, with a char-based fallback) and switch the digest cap to a token budget (_MAX_HISTORY_TOKENS = 8_000), matching the previous English-text size while holding regardless of content.