mirror of
https://github.com/HKUDS/nanobot.git
synced 2026-08-04 16:38:49 +00:00
89d8c055a8
Symptom ------- LLM requests intermittently fail with: 'utf-8' codec can't encode characters in position N-N+1: surrogates not allowed when messages contain emoji-heavy content (e.g. HTML with mixed emoji + JSON round-trips). This blocks the affected session until the session file is quarantined. Root cause ---------- Surrogate sanitization was only applied at the CLI entry point (nanobot/cli/commands.py: _sanitize_surrogates). Requests entering the LLM provider layer through other channels (Feishu, cron, webui, tool results, memory injection) had no defensive cleaning, so any message that happened to carry unpaired UTF-16 surrogates (from an upstream JSON round-trip with ensure_ascii=True on ill-formed input, memory rehydration, or third-party content) would blow up at json.dumps -> HTTP encode time inside the provider client. Fix --- 1. Extract sanitize_surrogates() and sanitize_surrogates_deep() into nanobot/utils/helpers.py as the single source of truth. Both use utf-16-le round-tripping with errors='surrogatepass' / 'replace', so paired surrogates reconstruct back into their real code point and lone surrogates collapse to U+FFFD. 2. Make nanobot/cli/commands.py:_sanitize_surrogates a thin wrapper that re-exports the shared helper (backward compatible). 3. Add defense-in-depth at the LLM provider boundary in nanobot/providers/base.py:_sanitize_empty_content by running sanitize_surrogates_deep over each message and its content blocks right before requests are serialized to JSON. Non-goals --------- - truncate_text() is intentionally left untouched. Python str slicing cannot split a single code point into surrogate halves, so it is not the source of lone surrogates. - session/manager storage layer is untouched. Archived sessions reproduced the failure only through the request path, not through storage. Verification ------------ - New regression suite tests/providers/test_sanitize_surrogates.py covers: paired surrogate reconstruction, lone surrogate replacement, identity return on clean input (zero allocation), deep recursion on dict/list/tuple, provider _sanitize_empty_content integration, and full utf-8 encodability of the sanitized request body. - 14/14 new tests pass; full existing test module also green. - Replayed 58 archived real session messages plus adversarial lone-surrogate injection through the provider path with no encode errors after the fix. Impact ------ - No behaviour change for clean inputs (sanitize_surrogates_deep is an identity return when no surrogate is present). - Fails-safe: unpaired surrogates degrade to U+FFFD instead of aborting the entire request.