mirror of
https://github.com/HKUDS/nanobot.git
synced 2026-05-19 16:12:30 +00:00
feat: add image generation tool and WebUI mode
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Xubin Ren
Xubin Ren
parent
3a2f47d720
commit
e936ed48bd
@ -14,6 +14,7 @@ Start here for setup, everyday usage, and deployment.
|
||||
| Chat apps | [`chat-apps.md`](./chat-apps.md) | Connect nanobot to Telegram, Discord, WeChat, and more |
|
||||
| Agent social network | [`agent-social-network.md`](./agent-social-network.md) | Join external agent communities from nanobot |
|
||||
| Configuration | [`configuration.md`](./configuration.md) | Providers, tools, channels, MCP, and runtime settings |
|
||||
| Image generation | [`image-generation.md`](./image-generation.md) | Configure image providers, WebUI image mode, and generated artifacts |
|
||||
| Multiple instances | [`multiple-instances.md`](./multiple-instances.md) | Run isolated bots with separate configs and workspaces |
|
||||
| CLI reference | [`cli-reference.md`](./cli-reference.md) | Core CLI commands and common entrypoints |
|
||||
| In-chat commands | [`chat-commands.md`](./chat-commands.md) | Slash commands and periodic task behavior |
|
||||
|
||||
@ -915,6 +915,12 @@ If you want to always use the local conversion, you can force it using:
|
||||
|--------|------|---------|-------------|
|
||||
| `useJinaReader` | boolean | `true` | If true, Jina Reader will be preferred over the local conversion |
|
||||
|
||||
## Image Generation
|
||||
|
||||
Image generation is configured under `tools.imageGeneration` and uses provider credentials from `providers.openrouter` or `providers.aihubmix`.
|
||||
|
||||
See [Image Generation](./image-generation.md) for WebUI usage, provider examples, artifact storage, and troubleshooting.
|
||||
|
||||
## MCP (Model Context Protocol)
|
||||
|
||||
> [!TIP]
|
||||
|
||||
200
docs/image-generation.md
Normal file
200
docs/image-generation.md
Normal file
@ -0,0 +1,200 @@
|
||||
# Image Generation
|
||||
|
||||
nanobot can generate and edit images through the `generate_image` tool. In the WebUI, users can enable **Image Generation** from the composer, choose an aspect ratio, and keep iterating on generated images inside the same chat.
|
||||
|
||||
The feature is disabled by default. Enable it in `~/.nanobot/config.json`, configure a supported image provider, then restart the gateway.
|
||||
|
||||
## Quick Setup
|
||||
|
||||
OpenRouter example:
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"openrouter": {
|
||||
"apiKey": "${OPENROUTER_API_KEY}"
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "openrouter",
|
||||
"model": "openai/gpt-5.4-image-2",
|
||||
"defaultAspectRatio": "1:1",
|
||||
"defaultImageSize": "1K"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
AIHubMix example:
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"aihubmix": {
|
||||
"apiKey": "${AIHUBMIX_API_KEY}"
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "aihubmix",
|
||||
"model": "gpt-image-2-free",
|
||||
"defaultAspectRatio": "1:1",
|
||||
"defaultImageSize": "1K"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> Prefer environment variables for API keys. nanobot resolves `${VAR_NAME}` values from the environment at startup.
|
||||
|
||||
## WebUI Usage
|
||||
|
||||
In the WebUI composer:
|
||||
|
||||
1. Click **Image Generation**.
|
||||
2. Choose an aspect ratio: `Auto`, `1:1`, `3:4`, `9:16`, `4:3`, or `16:9`.
|
||||
3. Describe the image or the edit you want.
|
||||
4. Attach reference images when editing an existing image.
|
||||
|
||||
Generated images are rendered as assistant media in the chat. Follow-up prompts such as "make it warmer", "change the background", or "try a 16:9 version" can reuse the most recent generated artifact.
|
||||
|
||||
The WebUI hides provider storage details from the user. The agent sees the saved artifact path internally and can pass it back to `generate_image` as `reference_images` for iterative edits.
|
||||
|
||||
## Configuration Reference
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `tools.imageGeneration.enabled` | boolean | `false` | Register the `generate_image` tool |
|
||||
| `tools.imageGeneration.provider` | string | `"openrouter"` | Image provider name. Currently `openrouter` and `aihubmix` are supported |
|
||||
| `tools.imageGeneration.model` | string | `"openai/gpt-5.4-image-2"` | Provider model name |
|
||||
| `tools.imageGeneration.defaultAspectRatio` | string | `"1:1"` | Default ratio when the prompt/tool call does not specify one |
|
||||
| `tools.imageGeneration.defaultImageSize` | string | `"1K"` | Default size hint, for example `1K`, `2K`, `4K`, or `1024x1024` |
|
||||
| `tools.imageGeneration.maxImagesPerTurn` | number | `4` | Maximum `count` accepted by one tool call. Valid range: `1` to `8` |
|
||||
| `tools.imageGeneration.saveDir` | string | `"generated"` | Relative directory under nanobot's media directory for generated artifacts |
|
||||
|
||||
Provider settings reuse normal provider config fields:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `providers.<name>.apiKey` | Provider API key. Prefer `${ENV_VAR}` |
|
||||
| `providers.<name>.apiBase` | Optional custom base URL |
|
||||
| `providers.<name>.extraHeaders` | Headers merged into provider requests |
|
||||
| `providers.<name>.extraBody` | Extra JSON fields merged into provider request bodies |
|
||||
|
||||
Both camelCase and snake_case config keys are accepted, but docs use camelCase to match `config.json`.
|
||||
|
||||
## Provider Notes
|
||||
|
||||
### OpenRouter
|
||||
|
||||
OpenRouter uses a chat-completions style image response. Configure:
|
||||
|
||||
```json
|
||||
{
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "openrouter",
|
||||
"model": "openai/gpt-5.4-image-2"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use a model that supports image generation and image editing if you want reference-image edits.
|
||||
|
||||
### AIHubMix
|
||||
|
||||
AIHubMix `gpt-image-2-free` is supported through AIHubMix's unified predictions API. Internally nanobot calls:
|
||||
|
||||
```text
|
||||
/v1/models/openai/gpt-image-2-free/predictions
|
||||
```
|
||||
|
||||
Configure:
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"aihubmix": {
|
||||
"apiKey": "${AIHUBMIX_API_KEY}",
|
||||
"extraBody": {
|
||||
"quality": "low"
|
||||
}
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "aihubmix",
|
||||
"model": "gpt-image-2-free"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`quality: low` is optional. It can make free image models faster and less likely to time out, but it is not required for correctness.
|
||||
|
||||
## Artifacts
|
||||
|
||||
Generated images are stored under the active nanobot instance's media directory:
|
||||
|
||||
```text
|
||||
~/.nanobot/media/generated/YYYY-MM-DD/img_<id>.<ext>
|
||||
~/.nanobot/media/generated/YYYY-MM-DD/img_<id>.json
|
||||
```
|
||||
|
||||
For non-default config locations, the media directory is relative to the active config file's directory.
|
||||
|
||||
The JSON sidecar stores:
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `id` | Short generated image id, such as `img_ab12cd34ef56` |
|
||||
| `path` | Local image path used internally for follow-up edits |
|
||||
| `mime` | Detected image MIME type |
|
||||
| `prompt` | Prompt used for the generation |
|
||||
| `model` | Provider model |
|
||||
| `provider` | Provider name |
|
||||
| `source_images` | Reference image paths used for edits |
|
||||
| `created_at` | Creation timestamp |
|
||||
|
||||
Do not paste base64 image payloads into chat. The agent should keep local artifact paths internal unless the user explicitly asks for debugging details.
|
||||
|
||||
## Prompting
|
||||
|
||||
Good image prompts include:
|
||||
|
||||
- Subject and scene.
|
||||
- Composition, camera, or layout.
|
||||
- Style, mood, lighting, and color palette.
|
||||
- Exact text that must appear in the image, quoted.
|
||||
- Constraints such as "keep the same character" or "preserve the logo".
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
A minimal app icon for nanobot: friendly robot head, rounded square, soft blue and white palette, clean vector style, no text
|
||||
```
|
||||
|
||||
For edits, describe what should change and what must stay fixed:
|
||||
|
||||
```text
|
||||
Use the reference image. Keep the same robot and composition, change the palette to warm orange, and add a subtle sunrise background.
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Check |
|
||||
|---------|-------|
|
||||
| `generate_image` is not available | Set `tools.imageGeneration.enabled` to `true` and restart the gateway |
|
||||
| Missing API key error | Configure `providers.<provider>.apiKey`; if using `${VAR_NAME}`, confirm the environment variable is visible to the gateway process |
|
||||
| `unsupported image generation provider` | Use `openrouter` or `aihubmix` |
|
||||
| AIHubMix says `Incorrect model ID` | Use `model: "gpt-image-2-free"`; nanobot expands it to the required `openai/gpt-image-2-free` model path internally |
|
||||
| Generation times out | Try a smaller/default image size, set AIHubMix `extraBody.quality` to `"low"`, or retry later |
|
||||
| Reference image rejected | Reference image paths must be inside the workspace or nanobot media directory and must be valid image files |
|
||||
|
||||
@ -30,6 +30,7 @@ from nanobot.agent.tools.ask import (
|
||||
from nanobot.agent.tools.cron import CronTool
|
||||
from nanobot.agent.tools.file_state import FileStateStore, bind_file_states, reset_file_states
|
||||
from nanobot.agent.tools.filesystem import EditFileTool, ListDirTool, ReadFileTool, WriteFileTool
|
||||
from nanobot.agent.tools.image_generation import ImageGenerationTool
|
||||
from nanobot.agent.tools.message import MessageTool
|
||||
from nanobot.agent.tools.notebook import NotebookEditTool
|
||||
from nanobot.agent.tools.registry import ToolRegistry
|
||||
@ -45,9 +46,11 @@ from nanobot.config.schema import AgentDefaults
|
||||
from nanobot.providers.base import LLMProvider
|
||||
from nanobot.providers.factory import ProviderSnapshot
|
||||
from nanobot.session.manager import Session, SessionManager
|
||||
from nanobot.utils.artifacts import generated_image_paths_from_messages
|
||||
from nanobot.utils.document import extract_documents
|
||||
from nanobot.utils.helpers import image_placeholder_text
|
||||
from nanobot.utils.helpers import truncate_text as truncate_text_fn
|
||||
from nanobot.utils.image_generation_intent import image_generation_prompt
|
||||
from nanobot.utils.progress_events import (
|
||||
build_tool_event_finish_payloads,
|
||||
build_tool_event_start_payload,
|
||||
@ -58,7 +61,13 @@ from nanobot.utils.runtime import EMPTY_FINAL_RESPONSE_MESSAGE
|
||||
from nanobot.utils.webui_titles import mark_webui_session, maybe_generate_webui_title_after_turn
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from nanobot.config.schema import ChannelsConfig, ExecToolConfig, ToolsConfig, WebToolsConfig
|
||||
from nanobot.config.schema import (
|
||||
ChannelsConfig,
|
||||
ExecToolConfig,
|
||||
ProviderConfig,
|
||||
ToolsConfig,
|
||||
WebToolsConfig,
|
||||
)
|
||||
from nanobot.cron.service import CronService
|
||||
|
||||
|
||||
@ -215,6 +224,8 @@ class AgentLoop:
|
||||
unified_session: bool = False,
|
||||
disabled_skills: list[str] | None = None,
|
||||
tools_config: ToolsConfig | None = None,
|
||||
image_generation_provider_config: ProviderConfig | None = None,
|
||||
image_generation_provider_configs: dict[str, ProviderConfig] | None = None,
|
||||
provider_snapshot_loader: Callable[[], ProviderSnapshot] | None = None,
|
||||
provider_signature: tuple[object, ...] | None = None,
|
||||
):
|
||||
@ -250,6 +261,13 @@ class AgentLoop:
|
||||
)
|
||||
self.web_config = web_config or WebToolsConfig()
|
||||
self.exec_config = exec_config or ExecToolConfig()
|
||||
self.tools_config = _tc
|
||||
self._image_generation_provider_configs = dict(image_generation_provider_configs or {})
|
||||
if (
|
||||
image_generation_provider_config is not None
|
||||
and "openrouter" not in self._image_generation_provider_configs
|
||||
):
|
||||
self._image_generation_provider_configs["openrouter"] = image_generation_provider_config
|
||||
self.cron_service = cron_service
|
||||
self.restrict_to_workspace = restrict_to_workspace
|
||||
self._start_time = time.time()
|
||||
@ -404,6 +422,14 @@ class AgentLoop:
|
||||
user_agent=self.web_config.user_agent,
|
||||
)
|
||||
)
|
||||
if self.tools_config.image_generation.enabled:
|
||||
self.tools.register(
|
||||
ImageGenerationTool(
|
||||
workspace=self.workspace,
|
||||
config=self.tools_config.image_generation,
|
||||
provider_configs=self._image_generation_provider_configs,
|
||||
)
|
||||
)
|
||||
self.tools.register(MessageTool(send_callback=self.bus.publish_outbound, workspace=self.workspace))
|
||||
self.tools.register(SpawnTool(manager=self.subagents))
|
||||
if self.cron_service:
|
||||
@ -1063,12 +1089,12 @@ class AgentLoop:
|
||||
self.context.build_system_prompt(channel=msg.channel),
|
||||
history,
|
||||
pending_ask_id,
|
||||
msg.content,
|
||||
image_generation_prompt(msg.content, msg.metadata),
|
||||
)
|
||||
else:
|
||||
initial_messages = self.context.build_messages(
|
||||
history=history,
|
||||
current_message=msg.content,
|
||||
current_message=image_generation_prompt(msg.content, msg.metadata),
|
||||
session_summary=pending,
|
||||
media=msg.media if msg.media else None,
|
||||
channel=msg.channel,
|
||||
@ -1143,6 +1169,11 @@ class AgentLoop:
|
||||
|
||||
# Skip the already-persisted user message when saving the turn
|
||||
save_skip = 1 + len(history) + (1 if user_persisted_early else 0)
|
||||
generated_media = generated_image_paths_from_messages(all_msgs[save_skip:])
|
||||
if generated_media and all_msgs and all_msgs[-1].get("role") == "assistant":
|
||||
existing_media = all_msgs[-1].get("media")
|
||||
media = existing_media if isinstance(existing_media, list) else []
|
||||
all_msgs[-1]["media"] = list(dict.fromkeys([*media, *generated_media]))
|
||||
self._save_turn(session, all_msgs, save_skip)
|
||||
session.enforce_file_cap(on_archive=self.context.memory.raw_archive)
|
||||
self._clear_pending_user_turn(session)
|
||||
@ -1175,6 +1206,7 @@ class AgentLoop:
|
||||
channel=msg.channel,
|
||||
chat_id=msg.chat_id,
|
||||
content=final_content,
|
||||
media=generated_media,
|
||||
metadata=meta,
|
||||
buttons=buttons,
|
||||
)
|
||||
|
||||
192
nanobot/agent/tools/image_generation.py
Normal file
192
nanobot/agent/tools/image_generation.py
Normal file
@ -0,0 +1,192 @@
|
||||
"""Image generation tool."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from nanobot.agent.tools.base import Tool, tool_parameters
|
||||
from nanobot.agent.tools.schema import (
|
||||
ArraySchema,
|
||||
IntegerSchema,
|
||||
StringSchema,
|
||||
tool_parameters_schema,
|
||||
)
|
||||
from nanobot.config.paths import get_media_dir
|
||||
from nanobot.config.schema import ImageGenerationToolConfig
|
||||
from nanobot.providers.image_generation import (
|
||||
AIHubMixImageGenerationClient,
|
||||
ImageGenerationError,
|
||||
OpenRouterImageGenerationClient,
|
||||
)
|
||||
from nanobot.utils.artifacts import (
|
||||
ArtifactError,
|
||||
generated_image_tool_result,
|
||||
store_generated_image_artifact,
|
||||
)
|
||||
from nanobot.utils.helpers import detect_image_mime
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from nanobot.config.schema import ProviderConfig
|
||||
|
||||
|
||||
@tool_parameters(
|
||||
tool_parameters_schema(
|
||||
prompt=StringSchema(
|
||||
"Detailed image generation or edit prompt. Include style, subject, composition, colors, and constraints.",
|
||||
min_length=1,
|
||||
),
|
||||
reference_images=ArraySchema(
|
||||
StringSchema("Local path of an existing image artifact or user-provided image to use as an edit reference."),
|
||||
description="Optional local image paths. Use generated artifact paths for iterative edits.",
|
||||
),
|
||||
aspect_ratio=StringSchema(
|
||||
"Optional output aspect ratio, e.g. 1:1, 16:9, 9:16, 4:3.",
|
||||
),
|
||||
image_size=StringSchema(
|
||||
"Optional output size hint supported by the configured provider, e.g. 1K, 2K, 4K, or 1024x1024.",
|
||||
),
|
||||
count=IntegerSchema(
|
||||
description="Number of images to generate in this turn.",
|
||||
minimum=1,
|
||||
maximum=8,
|
||||
),
|
||||
required=["prompt"],
|
||||
)
|
||||
)
|
||||
class ImageGenerationTool(Tool):
|
||||
"""Generate persistent image artifacts through the configured image provider."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
workspace: str | Path,
|
||||
config: ImageGenerationToolConfig,
|
||||
provider_config: ProviderConfig | None = None,
|
||||
provider_configs: dict[str, ProviderConfig] | None = None,
|
||||
) -> None:
|
||||
self.workspace = Path(workspace).expanduser()
|
||||
self.config = config
|
||||
self.provider_configs = dict(provider_configs or {})
|
||||
if provider_config is not None and "openrouter" not in self.provider_configs:
|
||||
self.provider_configs["openrouter"] = provider_config
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "generate_image"
|
||||
|
||||
@property
|
||||
def description(self) -> str:
|
||||
return (
|
||||
"Generate or edit images and store them as persistent artifacts. "
|
||||
"Returns artifact ids and local paths. For edits, pass prior generated image paths "
|
||||
"or user image paths as reference_images."
|
||||
)
|
||||
|
||||
def _provider_config(self) -> ProviderConfig | None:
|
||||
return self.provider_configs.get(self.config.provider)
|
||||
|
||||
def _provider_client(self) -> OpenRouterImageGenerationClient | AIHubMixImageGenerationClient | None:
|
||||
provider = self._provider_config()
|
||||
kwargs = {
|
||||
"api_key": provider.api_key if provider else None,
|
||||
"api_base": provider.api_base if provider else None,
|
||||
"extra_headers": provider.extra_headers if provider else None,
|
||||
"extra_body": provider.extra_body if provider else None,
|
||||
}
|
||||
if self.config.provider == "openrouter":
|
||||
return OpenRouterImageGenerationClient(**kwargs)
|
||||
if self.config.provider == "aihubmix":
|
||||
return AIHubMixImageGenerationClient(**kwargs)
|
||||
return None
|
||||
|
||||
def _missing_api_key_error(self) -> str:
|
||||
provider = self.config.provider
|
||||
if provider == "openrouter":
|
||||
return "Error: OpenRouter API key is not configured. Set providers.openrouter.apiKey."
|
||||
if provider == "aihubmix":
|
||||
return "Error: AIHubMix API key is not configured. Set providers.aihubmix.apiKey."
|
||||
return f"Error: {provider} API key is not configured."
|
||||
|
||||
def _resolve_reference_image(self, value: str) -> str:
|
||||
raw_path = Path(value).expanduser()
|
||||
path = raw_path if raw_path.is_absolute() else self.workspace / raw_path
|
||||
try:
|
||||
resolved = path.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise ImageGenerationError(f"reference image not found: {value}") from exc
|
||||
|
||||
allowed_roots = [self.workspace.resolve(), get_media_dir().resolve()]
|
||||
if not any(_is_relative_to(resolved, root) for root in allowed_roots):
|
||||
raise ImageGenerationError(
|
||||
"reference_images must be inside the workspace or nanobot media directory"
|
||||
)
|
||||
if not resolved.is_file():
|
||||
raise ImageGenerationError(f"reference image is not a file: {value}")
|
||||
raw = resolved.read_bytes()
|
||||
if detect_image_mime(raw) is None:
|
||||
raise ImageGenerationError(f"unsupported reference image: {value}")
|
||||
return str(resolved)
|
||||
|
||||
def _resolve_reference_images(self, values: list[str] | None) -> list[str]:
|
||||
if not values:
|
||||
return []
|
||||
return [self._resolve_reference_image(value) for value in values if value]
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
prompt: str,
|
||||
reference_images: list[str] | None = None,
|
||||
aspect_ratio: str | None = None,
|
||||
image_size: str | None = None,
|
||||
count: int | None = None,
|
||||
**kwargs: Any,
|
||||
) -> str:
|
||||
client = self._provider_client()
|
||||
if client is None:
|
||||
return f"Error: unsupported image generation provider '{self.config.provider}'"
|
||||
provider = self._provider_config()
|
||||
if not provider or not provider.api_key:
|
||||
return self._missing_api_key_error()
|
||||
|
||||
requested = count or 1
|
||||
if requested > self.config.max_images_per_turn:
|
||||
return (
|
||||
"Error: count exceeds tools.imageGeneration.maxImagesPerTurn "
|
||||
f"({self.config.max_images_per_turn})"
|
||||
)
|
||||
|
||||
try:
|
||||
refs = self._resolve_reference_images(reference_images)
|
||||
artifacts: list[dict[str, Any]] = []
|
||||
while len(artifacts) < requested:
|
||||
response = await client.generate(
|
||||
prompt=prompt,
|
||||
model=self.config.model,
|
||||
reference_images=refs,
|
||||
aspect_ratio=aspect_ratio or self.config.default_aspect_ratio,
|
||||
image_size=image_size or self.config.default_image_size,
|
||||
)
|
||||
for image_data_url in response.images:
|
||||
artifact = store_generated_image_artifact(
|
||||
image_data_url,
|
||||
prompt=prompt,
|
||||
model=self.config.model,
|
||||
source_images=refs,
|
||||
save_dir=self.config.save_dir,
|
||||
provider=self.config.provider,
|
||||
)
|
||||
artifacts.append(artifact)
|
||||
if len(artifacts) >= requested:
|
||||
break
|
||||
return generated_image_tool_result(artifacts)
|
||||
except (ArtifactError, ImageGenerationError, OSError) as exc:
|
||||
return f"Error: {exc}"
|
||||
|
||||
|
||||
def _is_relative_to(path: Path, root: Path) -> bool:
|
||||
try:
|
||||
path.relative_to(root)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
@ -158,7 +158,7 @@ class MessageTool(Tool):
|
||||
metadata = dict(self._default_metadata.get()) if same_target else {}
|
||||
if message_id:
|
||||
metadata["message_id"] = message_id
|
||||
if self._record_channel_delivery_var.get():
|
||||
if self._record_channel_delivery_var.get() or media:
|
||||
metadata["_record_channel_delivery"] = True
|
||||
|
||||
msg = OutboundMessage(
|
||||
|
||||
@ -1215,6 +1215,13 @@ class WebSocketChannel(BaseChannel):
|
||||
metadata: dict[str, Any] = {"remote": getattr(connection, "remote_address", None)}
|
||||
if envelope.get("webui") is True:
|
||||
metadata["webui"] = True
|
||||
image_generation = envelope.get("image_generation")
|
||||
if isinstance(image_generation, dict) and image_generation.get("enabled") is True:
|
||||
aspect_ratio = image_generation.get("aspect_ratio")
|
||||
metadata["image_generation"] = {
|
||||
"enabled": True,
|
||||
"aspect_ratio": aspect_ratio if isinstance(aspect_ratio, str) else None,
|
||||
}
|
||||
await self._handle_message(
|
||||
sender_id=client_id,
|
||||
chat_id=cid,
|
||||
@ -1258,7 +1265,14 @@ class WebSocketChannel(BaseChannel):
|
||||
# Snapshot the subscriber set so ConnectionClosed cleanups mid-iteration are safe.
|
||||
conns = list(self._subs.get(msg.chat_id, ()))
|
||||
if not conns:
|
||||
self.logger.warning("no active subscribers for chat_id={}", msg.chat_id)
|
||||
if (
|
||||
msg.metadata.get("_progress")
|
||||
or msg.metadata.get("_turn_end")
|
||||
or msg.metadata.get("_session_updated")
|
||||
):
|
||||
self.logger.debug("no active subscribers for chat_id={}", msg.chat_id)
|
||||
else:
|
||||
self.logger.warning("no active subscribers for chat_id={}", msg.chat_id)
|
||||
return
|
||||
# Signal that the agent has fully finished processing the current turn.
|
||||
if msg.metadata.get("_turn_end"):
|
||||
|
||||
@ -528,6 +528,7 @@ def serve(
|
||||
raise typer.Exit(1)
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from nanobot.agent.loop import AgentLoop
|
||||
from nanobot.api.server import create_app
|
||||
from nanobot.bus.queue import MessageBus
|
||||
@ -571,6 +572,10 @@ def serve(
|
||||
consolidation_ratio=runtime_config.agents.defaults.consolidation_ratio,
|
||||
max_messages=runtime_config.agents.defaults.max_messages,
|
||||
tools_config=runtime_config.tools,
|
||||
image_generation_provider_configs={
|
||||
"openrouter": runtime_config.providers.openrouter,
|
||||
"aihubmix": runtime_config.providers.aihubmix,
|
||||
},
|
||||
)
|
||||
|
||||
model_name = runtime_config.agents.defaults.model
|
||||
@ -696,6 +701,10 @@ def _run_gateway(
|
||||
consolidation_ratio=config.agents.defaults.consolidation_ratio,
|
||||
max_messages=config.agents.defaults.max_messages,
|
||||
tools_config=config.tools,
|
||||
image_generation_provider_configs={
|
||||
"openrouter": config.providers.openrouter,
|
||||
"aihubmix": config.providers.aihubmix,
|
||||
},
|
||||
provider_snapshot_loader=load_provider_snapshot,
|
||||
provider_signature=provider_snapshot.signature,
|
||||
)
|
||||
@ -735,7 +744,10 @@ def _run_gateway(
|
||||
):
|
||||
key = session_key or _channel_session_key(msg.channel, msg.chat_id)
|
||||
session = session_manager.get_or_create(key)
|
||||
session.add_message("assistant", msg.content, _channel_delivery=True)
|
||||
extra: dict[str, Any] = {"_channel_delivery": True}
|
||||
if msg.media:
|
||||
extra["media"] = list(msg.media)
|
||||
session.add_message("assistant", msg.content, **extra)
|
||||
session_manager.save(session)
|
||||
await bus.publish_outbound(msg)
|
||||
|
||||
|
||||
@ -253,12 +253,25 @@ class MyToolConfig(Base):
|
||||
allow_set: bool = False # let `my` modify loop state (read-only if False)
|
||||
|
||||
|
||||
class ImageGenerationToolConfig(Base):
|
||||
"""Image generation tool configuration."""
|
||||
|
||||
enabled: bool = False
|
||||
provider: str = "openrouter"
|
||||
model: str = "openai/gpt-5.4-image-2"
|
||||
default_aspect_ratio: str = "1:1"
|
||||
default_image_size: str = "1K"
|
||||
max_images_per_turn: int = Field(default=4, ge=1, le=8)
|
||||
save_dir: str = "generated"
|
||||
|
||||
|
||||
class ToolsConfig(Base):
|
||||
"""Tools configuration."""
|
||||
|
||||
web: WebToolsConfig = Field(default_factory=WebToolsConfig)
|
||||
exec: ExecToolConfig = Field(default_factory=ExecToolConfig)
|
||||
my: MyToolConfig = Field(default_factory=MyToolConfig)
|
||||
image_generation: ImageGenerationToolConfig = Field(default_factory=ImageGenerationToolConfig)
|
||||
restrict_to_workspace: bool = False # restrict all tool access to workspace directory
|
||||
mcp_servers: dict[str, MCPServerConfig] = Field(default_factory=dict)
|
||||
ssrf_whitelist: list[str] = Field(default_factory=list) # CIDR ranges to exempt from SSRF blocking (e.g. ["100.64.0.0/10"] for Tailscale)
|
||||
|
||||
@ -87,6 +87,10 @@ class Nanobot:
|
||||
session_ttl_minutes=defaults.session_ttl_minutes,
|
||||
consolidation_ratio=defaults.consolidation_ratio,
|
||||
tools_config=config.tools,
|
||||
image_generation_provider_configs={
|
||||
"openrouter": config.providers.openrouter,
|
||||
"aihubmix": config.providers.aihubmix,
|
||||
},
|
||||
)
|
||||
return cls(loop)
|
||||
|
||||
|
||||
395
nanobot/providers/image_generation.py
Normal file
395
nanobot/providers/image_generation.py
Normal file
@ -0,0 +1,395 @@
|
||||
"""Image generation provider helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from nanobot.providers.registry import find_by_name
|
||||
from nanobot.utils.helpers import detect_image_mime
|
||||
|
||||
_OPENROUTER_ATTRIBUTION_HEADERS = {
|
||||
"HTTP-Referer": "https://github.com/HKUDS/nanobot",
|
||||
"X-OpenRouter-Title": "nanobot",
|
||||
"X-OpenRouter-Categories": "cli-agent,personal-agent",
|
||||
}
|
||||
_DEFAULT_TIMEOUT_S = 120.0
|
||||
_AIHUBMIX_TIMEOUT_S = 300.0
|
||||
_AIHUBMIX_ASPECT_RATIO_SIZES = {
|
||||
"1:1": "1024x1024",
|
||||
"3:4": "1024x1536",
|
||||
"9:16": "1024x1536",
|
||||
"4:3": "1536x1024",
|
||||
"16:9": "1536x1024",
|
||||
}
|
||||
|
||||
|
||||
class ImageGenerationError(RuntimeError):
|
||||
"""Raised when the image generation provider cannot return images."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GeneratedImageResponse:
|
||||
"""Images and optional text returned by the provider."""
|
||||
|
||||
images: list[str]
|
||||
content: str
|
||||
raw: dict[str, Any]
|
||||
|
||||
|
||||
def _provider_base_url(provider: str, api_base: str | None, fallback: str) -> str:
|
||||
if api_base:
|
||||
return api_base.rstrip("/")
|
||||
spec = find_by_name(provider)
|
||||
if spec and spec.default_api_base:
|
||||
return spec.default_api_base.rstrip("/")
|
||||
return fallback
|
||||
|
||||
|
||||
def image_path_to_data_url(path: str | Path) -> str:
|
||||
"""Convert a local image path to an image data URL."""
|
||||
p = Path(path).expanduser()
|
||||
raw = p.read_bytes()
|
||||
mime = detect_image_mime(raw)
|
||||
if mime is None:
|
||||
raise ImageGenerationError(f"unsupported reference image: {p}")
|
||||
encoded = base64.b64encode(raw).decode("ascii")
|
||||
return f"data:{mime};base64,{encoded}"
|
||||
|
||||
|
||||
def _b64_png_data_url(value: str) -> str:
|
||||
return f"data:image/png;base64,{value}"
|
||||
|
||||
|
||||
def _aihubmix_size(aspect_ratio: str | None, image_size: str | None) -> str:
|
||||
"""Return an OpenAI Images API size string for AIHubMix.
|
||||
|
||||
The WebUI emits compact size hints like ``1K`` for OpenRouter. AIHubMix's
|
||||
Images API expects OpenAI-style dimensions or ``auto``, so only pass
|
||||
through explicit dimension strings and otherwise derive the closest
|
||||
supported orientation from aspect ratio.
|
||||
"""
|
||||
if image_size and "x" in image_size.lower():
|
||||
return image_size
|
||||
if aspect_ratio in _AIHUBMIX_ASPECT_RATIO_SIZES:
|
||||
return _AIHUBMIX_ASPECT_RATIO_SIZES[aspect_ratio]
|
||||
return "auto"
|
||||
|
||||
|
||||
def _aihubmix_model_path(model: str) -> str:
|
||||
if "/" in model:
|
||||
return model
|
||||
if model.startswith(("gpt-image-", "dall-e-")):
|
||||
return f"openai/{model}"
|
||||
return model
|
||||
|
||||
|
||||
async def _download_image_data_url(
|
||||
client: httpx.AsyncClient,
|
||||
url: str,
|
||||
) -> str:
|
||||
response = await client.get(url)
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = response.text[:500]
|
||||
raise ImageGenerationError(f"failed to download generated image: {detail}") from exc
|
||||
raw = response.content
|
||||
mime = detect_image_mime(raw)
|
||||
if mime is None:
|
||||
raise ImageGenerationError("generated image URL did not return a supported image")
|
||||
encoded = base64.b64encode(raw).decode("ascii")
|
||||
return f"data:{mime};base64,{encoded}"
|
||||
|
||||
|
||||
class OpenRouterImageGenerationClient:
|
||||
"""Small async client for OpenRouter Chat Completions image generation."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
api_key: str | None,
|
||||
api_base: str | None = None,
|
||||
extra_headers: dict[str, str] | None = None,
|
||||
extra_body: dict[str, Any] | None = None,
|
||||
timeout: float = _DEFAULT_TIMEOUT_S,
|
||||
client: httpx.AsyncClient | None = None,
|
||||
) -> None:
|
||||
self.api_key = api_key
|
||||
self.api_base = _provider_base_url(
|
||||
"openrouter",
|
||||
api_base,
|
||||
"https://openrouter.ai/api/v1",
|
||||
)
|
||||
self.extra_headers = extra_headers or {}
|
||||
self.extra_body = extra_body or {}
|
||||
self.timeout = timeout
|
||||
self._client = client
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
*,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_images: list[str] | None = None,
|
||||
aspect_ratio: str | None = None,
|
||||
image_size: str | None = None,
|
||||
) -> GeneratedImageResponse:
|
||||
if not self.api_key:
|
||||
raise ImageGenerationError(
|
||||
"OpenRouter API key is not configured. Set providers.openrouter.apiKey."
|
||||
)
|
||||
|
||||
content: str | list[dict[str, Any]]
|
||||
references = list(reference_images or [])
|
||||
if references:
|
||||
blocks: list[dict[str, Any]] = [{"type": "text", "text": prompt}]
|
||||
blocks.extend(
|
||||
{"type": "image_url", "image_url": {"url": image_path_to_data_url(path)}}
|
||||
for path in references
|
||||
)
|
||||
content = blocks
|
||||
else:
|
||||
content = prompt
|
||||
|
||||
body: dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": [{"role": "user", "content": content}],
|
||||
"modalities": ["image", "text"],
|
||||
"stream": False,
|
||||
}
|
||||
image_config: dict[str, str] = {}
|
||||
if aspect_ratio:
|
||||
image_config["aspect_ratio"] = aspect_ratio
|
||||
if image_size:
|
||||
image_config["image_size"] = image_size
|
||||
if image_config:
|
||||
body["image_config"] = image_config
|
||||
body.update(self.extra_body)
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
**_OPENROUTER_ATTRIBUTION_HEADERS,
|
||||
**self.extra_headers,
|
||||
}
|
||||
url = f"{self.api_base}/chat/completions"
|
||||
|
||||
if self._client is not None:
|
||||
response = await self._client.post(url, headers=headers, json=body)
|
||||
else:
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.post(url, headers=headers, json=body)
|
||||
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = response.text[:500]
|
||||
raise ImageGenerationError(f"OpenRouter image generation failed: {detail}") from exc
|
||||
|
||||
data = response.json()
|
||||
images: list[str] = []
|
||||
text_parts: list[str] = []
|
||||
for choice in data.get("choices") or []:
|
||||
if not isinstance(choice, dict):
|
||||
continue
|
||||
message = choice.get("message") or {}
|
||||
if isinstance(message.get("content"), str):
|
||||
text_parts.append(message["content"])
|
||||
for image in message.get("images") or []:
|
||||
if not isinstance(image, dict):
|
||||
continue
|
||||
image_url = image.get("image_url") or image.get("imageUrl") or {}
|
||||
url_value = image_url.get("url") if isinstance(image_url, dict) else None
|
||||
if isinstance(url_value, str) and url_value.startswith("data:image/"):
|
||||
images.append(url_value)
|
||||
|
||||
if not images:
|
||||
provider_error = data.get("error") if isinstance(data, dict) else None
|
||||
if provider_error:
|
||||
raise ImageGenerationError(f"OpenRouter returned no images: {provider_error}")
|
||||
raise ImageGenerationError("OpenRouter returned no images for this request")
|
||||
|
||||
return GeneratedImageResponse(
|
||||
images=images,
|
||||
content="\n".join(part for part in text_parts if part).strip(),
|
||||
raw=data,
|
||||
)
|
||||
|
||||
|
||||
class AIHubMixImageGenerationClient:
|
||||
"""Small async client for AIHubMix unified image generation."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
api_key: str | None,
|
||||
api_base: str | None = None,
|
||||
extra_headers: dict[str, str] | None = None,
|
||||
extra_body: dict[str, Any] | None = None,
|
||||
timeout: float = _AIHUBMIX_TIMEOUT_S,
|
||||
client: httpx.AsyncClient | None = None,
|
||||
) -> None:
|
||||
self.api_key = api_key
|
||||
self.api_base = _provider_base_url(
|
||||
"aihubmix",
|
||||
api_base,
|
||||
"https://aihubmix.com/v1",
|
||||
)
|
||||
self.extra_headers = extra_headers or {}
|
||||
self.extra_body = extra_body or {}
|
||||
self.timeout = timeout
|
||||
self._client = client
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
*,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_images: list[str] | None = None,
|
||||
aspect_ratio: str | None = None,
|
||||
image_size: str | None = None,
|
||||
) -> GeneratedImageResponse:
|
||||
if not self.api_key:
|
||||
raise ImageGenerationError(
|
||||
"AIHubMix API key is not configured. Set providers.aihubmix.apiKey."
|
||||
)
|
||||
|
||||
refs = list(reference_images or [])
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
**self.extra_headers,
|
||||
}
|
||||
size = _aihubmix_size(aspect_ratio, image_size)
|
||||
|
||||
if self._client is not None:
|
||||
return await self._generate_with_client(
|
||||
self._client,
|
||||
prompt=prompt,
|
||||
model=model,
|
||||
reference_images=refs,
|
||||
size=size,
|
||||
headers=headers,
|
||||
)
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
return await self._generate_with_client(
|
||||
client,
|
||||
prompt=prompt,
|
||||
model=model,
|
||||
reference_images=refs,
|
||||
size=size,
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
async def _generate_with_client(
|
||||
self,
|
||||
client: httpx.AsyncClient,
|
||||
*,
|
||||
prompt: str,
|
||||
model: str,
|
||||
reference_images: list[str],
|
||||
size: str,
|
||||
headers: dict[str, str],
|
||||
) -> GeneratedImageResponse:
|
||||
image_input: str | list[str] | None = None
|
||||
if reference_images:
|
||||
image_refs = [image_path_to_data_url(path) for path in reference_images]
|
||||
image_input = image_refs[0] if len(image_refs) == 1 else image_refs
|
||||
|
||||
input_body: dict[str, Any] = {
|
||||
"prompt": prompt,
|
||||
"n": 1,
|
||||
"size": size,
|
||||
}
|
||||
if image_input is not None:
|
||||
input_body["image"] = image_input
|
||||
input_body.update(self.extra_body)
|
||||
|
||||
body = {"input": input_body}
|
||||
model_path = _aihubmix_model_path(model)
|
||||
url = f"{self.api_base}/models/{model_path}/predictions"
|
||||
try:
|
||||
response = await client.post(
|
||||
url,
|
||||
headers={**headers, "Content-Type": "application/json"},
|
||||
json=body,
|
||||
)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise ImageGenerationError("AIHubMix image generation timed out") from exc
|
||||
except httpx.RequestError as exc:
|
||||
raise ImageGenerationError(f"AIHubMix image generation request failed: {exc}") from exc
|
||||
|
||||
try:
|
||||
response.raise_for_status()
|
||||
except httpx.HTTPStatusError as exc:
|
||||
detail = response.text[:500]
|
||||
raise ImageGenerationError(f"AIHubMix image generation failed: {detail}") from exc
|
||||
|
||||
payload = response.json()
|
||||
images = await _aihubmix_images_from_payload(client, payload)
|
||||
|
||||
if not images:
|
||||
provider_error = payload.get("error") if isinstance(payload, dict) else None
|
||||
if provider_error:
|
||||
raise ImageGenerationError(f"AIHubMix returned no images: {provider_error}")
|
||||
raise ImageGenerationError("AIHubMix returned no images for this request")
|
||||
|
||||
return GeneratedImageResponse(images=images, content="", raw=payload)
|
||||
|
||||
|
||||
async def _aihubmix_images_from_payload(
|
||||
client: httpx.AsyncClient,
|
||||
payload: dict[str, Any],
|
||||
) -> list[str]:
|
||||
images: list[str] = []
|
||||
candidates: list[Any] = []
|
||||
if "data" in payload:
|
||||
candidates.append(payload["data"])
|
||||
if "output" in payload:
|
||||
candidates.append(payload["output"])
|
||||
|
||||
async def collect(value: Any) -> None:
|
||||
if isinstance(value, list):
|
||||
for item in value:
|
||||
await collect(item)
|
||||
return
|
||||
if isinstance(value, str):
|
||||
if value.startswith("data:image/"):
|
||||
images.append(value)
|
||||
elif value.startswith(("http://", "https://")):
|
||||
images.append(await _download_image_data_url(client, value))
|
||||
return
|
||||
if not isinstance(value, dict):
|
||||
return
|
||||
|
||||
b64_json = value.get("b64_json")
|
||||
if isinstance(b64_json, str) and b64_json:
|
||||
images.append(_b64_png_data_url(b64_json))
|
||||
elif b64_json is not None:
|
||||
await collect(b64_json)
|
||||
|
||||
bytes_base64 = value.get("bytesBase64") or value.get("bytes_base64") or value.get("base64")
|
||||
if isinstance(bytes_base64, str) and bytes_base64:
|
||||
images.append(_b64_png_data_url(bytes_base64))
|
||||
|
||||
image_url = value.get("image_url") or value.get("imageUrl")
|
||||
if isinstance(image_url, dict):
|
||||
await collect(image_url.get("url"))
|
||||
elif image_url is not None:
|
||||
await collect(image_url)
|
||||
|
||||
url_value = value.get("url")
|
||||
if url_value is not None:
|
||||
await collect(url_value)
|
||||
|
||||
for key in ("images", "image", "output"):
|
||||
if key in value:
|
||||
await collect(value[key])
|
||||
|
||||
for candidate in candidates:
|
||||
await collect(candidate)
|
||||
return images
|
||||
109
nanobot/skills/image-generation/SKILL.md
Normal file
109
nanobot/skills/image-generation/SKILL.md
Normal file
@ -0,0 +1,109 @@
|
||||
---
|
||||
name: image-generation
|
||||
description: Generate images and iteratively edit saved image artifacts.
|
||||
---
|
||||
|
||||
# Image Generation
|
||||
|
||||
Use the `generate_image` tool when the user asks you to create, render, draw, design, generate, or edit an image.
|
||||
|
||||
If the `generate_image` tool is not available in the current tool list, tell the user that image generation is not enabled for this nanobot instance.
|
||||
|
||||
## When To Use
|
||||
|
||||
- Text-to-image: call `generate_image` with a concrete `prompt`.
|
||||
- Image editing: pass the saved artifact path or user image path in `reference_images`.
|
||||
- Iterative edits in the same conversation: prefer the most recent generated image artifact if the user says things like "make it brighter", "change the background", or "try another version".
|
||||
- Ambiguous edits: ask a short clarifying question if multiple recent images could be the target.
|
||||
|
||||
## Prompt Rules
|
||||
|
||||
Write prompts with enough detail for image models:
|
||||
|
||||
- Subject and scene.
|
||||
- Composition and camera or layout.
|
||||
- Style, mood, lighting, and color palette.
|
||||
- Text that must appear in the image, quoted exactly.
|
||||
- Constraints such as "keep the same character", "preserve the logo", or "do not change the background".
|
||||
|
||||
## Artifact Rules
|
||||
|
||||
The tool stores generated images as persistent artifacts under nanobot's media directory and returns structured metadata:
|
||||
|
||||
- `id`: generated image id, such as `img_ab12cd34ef56`.
|
||||
- `path`: local file path for internal follow-up edits.
|
||||
- `mime`: image MIME type.
|
||||
- `prompt`, `model`, and `source_images`: provenance for follow-up edits.
|
||||
|
||||
In normal user-facing replies, do not expose local filesystem paths. Keep the reply natural, for example "Done, I generated it." You may include the short image `id` when it helps the user refer to a specific image, but keep raw `path` internal unless the user explicitly asks for debug details or a local artifact reference. Never paste base64.
|
||||
|
||||
For follow-up edits, pass the prior artifact `path` to `reference_images`. If the user provides a new uploaded image, use that path as the reference instead.
|
||||
|
||||
## Provider Notes
|
||||
|
||||
Do not ask users to paste API keys into chat. If configuration is needed, describe the fields and remind them to restart the gateway after changing config.
|
||||
|
||||
For OpenRouter, the image tool expects:
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"openrouter": {
|
||||
"apiKey": "sk-or-..."
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "openrouter",
|
||||
"model": "openai/gpt-5.4-image-2"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
For AIHubMix, the image tool expects:
|
||||
|
||||
```json
|
||||
{
|
||||
"providers": {
|
||||
"aihubmix": {
|
||||
"apiKey": "sk-..."
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"imageGeneration": {
|
||||
"enabled": true,
|
||||
"provider": "aihubmix",
|
||||
"model": "gpt-image-2-free"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
AIHubMix `gpt-image-2-free` uses AIHubMix's unified predictions endpoint internally (`/v1/models/openai/gpt-image-2-free/predictions`), not the OpenAI Images `/v1/images/generations` endpoint. If it fails with "Incorrect model ID", do not assume the key lacks permission until the provider config, model name, and gateway restart have been checked.
|
||||
|
||||
`providers.aihubmix.extraBody` can be used for provider-specific options. For example, `"extraBody": {"quality": "low"}` is optional but can make `gpt-image-2-free` faster and less likely to time out.
|
||||
|
||||
## Examples
|
||||
|
||||
Generate a new image:
|
||||
|
||||
```text
|
||||
generate_image(
|
||||
prompt="A minimal app icon for nanobot: friendly robot head, rounded square, soft blue and white palette, clean vector style, no text",
|
||||
aspect_ratio="1:1",
|
||||
image_size="1K"
|
||||
)
|
||||
```
|
||||
|
||||
Edit the latest generated artifact:
|
||||
|
||||
```text
|
||||
generate_image(
|
||||
prompt="Use the reference image. Keep the same robot and composition, but change the palette to warm orange and add a subtle sunrise background.",
|
||||
reference_images=["/home/user/.nanobot/media/generated/2026-05-08/img_ab12cd34ef56.png"],
|
||||
aspect_ratio="1:1",
|
||||
image_size="1K"
|
||||
)
|
||||
```
|
||||
161
nanobot/utils/artifacts.py
Normal file
161
nanobot/utils/artifacts.py
Normal file
@ -0,0 +1,161 @@
|
||||
"""Artifact persistence helpers for generated media."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import re
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from nanobot.config.paths import get_media_dir
|
||||
from nanobot.utils.helpers import detect_image_mime, ensure_dir
|
||||
|
||||
_DATA_IMAGE_RE = re.compile(r"^data:(image/[A-Za-z0-9.+-]+);base64,(.*)$", re.DOTALL)
|
||||
_MIME_EXTENSIONS = {
|
||||
"image/png": ".png",
|
||||
"image/jpeg": ".jpg",
|
||||
"image/webp": ".webp",
|
||||
"image/gif": ".gif",
|
||||
}
|
||||
_GENERATE_IMAGE_TOOL_NAME = "generate_image"
|
||||
|
||||
|
||||
class ArtifactError(ValueError):
|
||||
"""Raised when an artifact cannot be safely decoded or stored."""
|
||||
|
||||
|
||||
def decode_image_data_url(data_url: str) -> tuple[bytes, str]:
|
||||
"""Decode a base64 image data URL and return ``(bytes, mime)``."""
|
||||
match = _DATA_IMAGE_RE.match(data_url.strip())
|
||||
if match is None:
|
||||
raise ArtifactError("expected a base64 image data URL")
|
||||
|
||||
declared_mime, encoded = match.groups()
|
||||
try:
|
||||
raw = base64.b64decode(encoded, validate=True)
|
||||
except binascii.Error as exc:
|
||||
raise ArtifactError("invalid base64 image payload") from exc
|
||||
|
||||
detected_mime = detect_image_mime(raw)
|
||||
if detected_mime is None:
|
||||
raise ArtifactError("unsupported or unrecognized image data")
|
||||
if declared_mime != detected_mime:
|
||||
declared_mime = detected_mime
|
||||
return raw, declared_mime
|
||||
|
||||
|
||||
def _safe_relative_dir(save_dir: str) -> Path:
|
||||
normalized = save_dir.replace("\\", "/").strip("/")
|
||||
if not normalized:
|
||||
raise ArtifactError("save_dir must not be empty")
|
||||
rel = PurePosixPath(normalized)
|
||||
if rel.is_absolute() or any(part in {"", ".", ".."} for part in rel.parts):
|
||||
raise ArtifactError("save_dir must be a safe relative path")
|
||||
return Path(*rel.parts)
|
||||
|
||||
|
||||
def _artifact_root(save_dir: str) -> Path:
|
||||
media_root = get_media_dir().resolve()
|
||||
root = (media_root / _safe_relative_dir(save_dir)).resolve()
|
||||
try:
|
||||
root.relative_to(media_root)
|
||||
except ValueError as exc:
|
||||
raise ArtifactError("artifact directory escapes media root") from exc
|
||||
return root
|
||||
|
||||
|
||||
def store_generated_image_artifact(
|
||||
data_url: str,
|
||||
*,
|
||||
prompt: str,
|
||||
model: str,
|
||||
source_images: list[str] | None = None,
|
||||
save_dir: str = "generated",
|
||||
provider: str = "openrouter",
|
||||
created_at: datetime | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Persist a generated image and sidecar metadata under the media root."""
|
||||
raw, mime = decode_image_data_url(data_url)
|
||||
ext = _MIME_EXTENSIONS.get(mime)
|
||||
if ext is None:
|
||||
raise ArtifactError(f"unsupported image MIME type: {mime}")
|
||||
|
||||
now = created_at or datetime.now().astimezone()
|
||||
day_dir = ensure_dir(_artifact_root(save_dir) / now.strftime("%Y-%m-%d"))
|
||||
artifact_id = f"img_{uuid.uuid4().hex[:12]}"
|
||||
image_path = day_dir / f"{artifact_id}{ext}"
|
||||
metadata_path = day_dir / f"{artifact_id}.json"
|
||||
|
||||
image_path.write_bytes(raw)
|
||||
metadata: dict[str, Any] = {
|
||||
"id": artifact_id,
|
||||
"path": str(image_path),
|
||||
"mime": mime,
|
||||
"prompt": prompt,
|
||||
"model": model,
|
||||
"provider": provider,
|
||||
"source_images": list(source_images or []),
|
||||
"created_at": now.isoformat(),
|
||||
}
|
||||
metadata_path.write_text(
|
||||
json.dumps(metadata, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
return metadata
|
||||
|
||||
|
||||
def generated_image_tool_result(artifacts: list[dict[str, Any]]) -> str:
|
||||
"""Return the compact structured result exposed to the LLM."""
|
||||
return json.dumps(
|
||||
{
|
||||
"artifacts": artifacts,
|
||||
"next_step": (
|
||||
"Use these artifact paths as reference_images for follow-up edits. "
|
||||
"Mention the image id/path to the user; do not paste base64."
|
||||
),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
||||
|
||||
def _extract_text_payload(content: Any) -> str | None:
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
parts: list[str] = []
|
||||
for block in content:
|
||||
if isinstance(block, dict) and isinstance(block.get("text"), str):
|
||||
parts.append(block["text"])
|
||||
return "\n".join(parts) if parts else None
|
||||
return None
|
||||
|
||||
|
||||
def generated_image_paths_from_messages(messages: list[dict[str, Any]]) -> list[str]:
|
||||
"""Collect generated image artifact paths from generate_image tool results."""
|
||||
paths: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for message in messages:
|
||||
if message.get("role") != "tool" or message.get("name") != _GENERATE_IMAGE_TOOL_NAME:
|
||||
continue
|
||||
payload = _extract_text_payload(message.get("content"))
|
||||
if not payload:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(payload)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
artifacts = data.get("artifacts") if isinstance(data, dict) else None
|
||||
if not isinstance(artifacts, list):
|
||||
continue
|
||||
for artifact in artifacts:
|
||||
if not isinstance(artifact, dict):
|
||||
continue
|
||||
path = artifact.get("path")
|
||||
if isinstance(path, str) and path and path not in seen:
|
||||
paths.append(path)
|
||||
seen.add(path)
|
||||
return paths
|
||||
27
nanobot/utils/image_generation_intent.py
Normal file
27
nanobot/utils/image_generation_intent.py
Normal file
@ -0,0 +1,27 @@
|
||||
"""Helpers for WebUI image-generation intent metadata."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
IMAGE_GENERATION_METADATA_KEY = "image_generation"
|
||||
|
||||
|
||||
def image_generation_prompt(content: str, metadata: dict[str, Any] | None) -> str:
|
||||
"""Decorate a user prompt when WebUI image mode is enabled."""
|
||||
raw = (metadata or {}).get(IMAGE_GENERATION_METADATA_KEY)
|
||||
if not isinstance(raw, dict) or raw.get("enabled") is not True:
|
||||
return content
|
||||
|
||||
aspect_ratio = raw.get("aspect_ratio")
|
||||
if isinstance(aspect_ratio, str) and aspect_ratio.strip():
|
||||
instruction = (
|
||||
"The user selected WebUI image generation mode. Use the generate_image tool. "
|
||||
f"When calling generate_image, pass aspect_ratio={aspect_ratio!r}."
|
||||
)
|
||||
else:
|
||||
instruction = (
|
||||
"The user selected WebUI image generation mode. Use the generate_image tool. "
|
||||
"Choose the most suitable aspect_ratio yourself from the prompt and intended use."
|
||||
)
|
||||
return f"{content}\n\n[WebUI image generation instruction: {instruction}]"
|
||||
89
tests/agent/test_loop_image_generation_media.py
Normal file
89
tests/agent/test_loop_image_generation_media.py
Normal file
@ -0,0 +1,89 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from nanobot.agent.loop import AgentLoop
|
||||
from nanobot.bus.events import InboundMessage
|
||||
from nanobot.bus.queue import MessageBus
|
||||
from nanobot.config.loader import set_config_path
|
||||
from nanobot.config.schema import ImageGenerationToolConfig, ProviderConfig, ToolsConfig
|
||||
from nanobot.providers.base import LLMResponse, ToolCallRequest
|
||||
from nanobot.providers.image_generation import GeneratedImageResponse
|
||||
|
||||
PNG_DATA_URL = (
|
||||
"data:image/png;base64,"
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
class FakeImageClient:
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
pass
|
||||
|
||||
async def generate(self, **kwargs: Any) -> GeneratedImageResponse:
|
||||
return GeneratedImageResponse(images=[PNG_DATA_URL], content="", raw={})
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generated_image_media_is_attached_to_final_assistant_message(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
monkeypatch.setattr(
|
||||
"nanobot.agent.tools.image_generation.OpenRouterImageGenerationClient",
|
||||
FakeImageClient,
|
||||
)
|
||||
provider = MagicMock()
|
||||
provider.get_default_model.return_value = "test-model"
|
||||
provider.generation.max_tokens = 4096
|
||||
provider.chat_with_retry = AsyncMock(
|
||||
side_effect=[
|
||||
LLMResponse(
|
||||
content="",
|
||||
finish_reason="tool_calls",
|
||||
tool_calls=[
|
||||
ToolCallRequest(
|
||||
id="call_img",
|
||||
name="generate_image",
|
||||
arguments={"prompt": "draw a tiny icon"},
|
||||
)
|
||||
],
|
||||
),
|
||||
LLMResponse(content="Done", finish_reason="stop"),
|
||||
]
|
||||
)
|
||||
provider.chat_stream_with_retry = AsyncMock()
|
||||
loop = AgentLoop(
|
||||
bus=MessageBus(),
|
||||
provider=provider,
|
||||
workspace=tmp_path,
|
||||
model="test-model",
|
||||
tools_config=ToolsConfig(
|
||||
image_generation=ImageGenerationToolConfig(enabled=True),
|
||||
),
|
||||
image_generation_provider_config=ProviderConfig(api_key="sk-or-test"),
|
||||
)
|
||||
loop.consolidator.maybe_consolidate_by_tokens = AsyncMock(return_value=False) # type: ignore[method-assign]
|
||||
|
||||
result = await loop._process_message(
|
||||
InboundMessage(
|
||||
channel="websocket",
|
||||
sender_id="user",
|
||||
chat_id="chat-image",
|
||||
content="draw an icon",
|
||||
)
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert result.content == "Done"
|
||||
assert len(result.media) == 1
|
||||
assert Path(result.media[0]).is_file()
|
||||
|
||||
session = loop.sessions.get_or_create("websocket:chat-image")
|
||||
assert session.messages[-1]["role"] == "assistant"
|
||||
assert session.messages[-1]["media"] == result.media
|
||||
204
tests/providers/test_image_generation.py
Normal file
204
tests/providers/test_image_generation.py
Normal file
@ -0,0 +1,204 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from nanobot.providers.image_generation import (
|
||||
AIHubMixImageGenerationClient,
|
||||
GeneratedImageResponse,
|
||||
ImageGenerationError,
|
||||
OpenRouterImageGenerationClient,
|
||||
)
|
||||
|
||||
PNG_BYTES = (
|
||||
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"
|
||||
b"\x00\x00\x00\x01\x08\x04\x00\x00\x00\xb5\x1c\x0c\x02"
|
||||
b"\x00\x00\x00\x0bIDATx\xdacd\xfc\xff\x1f\x00\x03\x03"
|
||||
b"\x02\x00\xef\xbf\xa7\xdb\x00\x00\x00\x00IEND\xaeB`\x82"
|
||||
)
|
||||
PNG_DATA_URL = (
|
||||
"data:image/png;base64,"
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
class FakeResponse:
|
||||
def __init__(
|
||||
self,
|
||||
payload: dict[str, Any],
|
||||
status_code: int = 200,
|
||||
content: bytes = b"",
|
||||
) -> None:
|
||||
self._payload = payload
|
||||
self.status_code = status_code
|
||||
self.text = str(payload)
|
||||
self.content = content
|
||||
self.request = httpx.Request("POST", "https://openrouter.ai/api/v1/chat/completions")
|
||||
|
||||
def json(self) -> dict[str, Any]:
|
||||
return self._payload
|
||||
|
||||
def raise_for_status(self) -> None:
|
||||
if self.status_code >= 400:
|
||||
response = httpx.Response(self.status_code, request=self.request, text=self.text)
|
||||
raise httpx.HTTPStatusError("failed", request=self.request, response=response)
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, response: FakeResponse) -> None:
|
||||
self.response = response
|
||||
self.get_response = response
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
self.get_calls: list[dict[str, Any]] = []
|
||||
|
||||
async def post(self, url: str, **kwargs: Any) -> FakeResponse:
|
||||
self.calls.append({"url": url, **kwargs})
|
||||
return self.response
|
||||
|
||||
async def get(self, url: str, **kwargs: Any) -> FakeResponse:
|
||||
self.get_calls.append({"url": url, **kwargs})
|
||||
return self.get_response
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openrouter_image_generation_payload_and_response(tmp_path: Path) -> None:
|
||||
ref = tmp_path / "ref.png"
|
||||
ref.write_bytes(PNG_BYTES)
|
||||
fake = FakeClient(
|
||||
FakeResponse(
|
||||
{
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": "done",
|
||||
"images": [{"image_url": {"url": PNG_DATA_URL}}],
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
)
|
||||
client = OpenRouterImageGenerationClient(
|
||||
api_key="sk-or-test",
|
||||
api_base="https://openrouter.ai/api/v1/",
|
||||
extra_headers={"X-Test": "1"},
|
||||
client=fake, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
response = await client.generate(
|
||||
prompt="make this blue",
|
||||
model="openai/gpt-5.4-image-2",
|
||||
reference_images=[str(ref)],
|
||||
aspect_ratio="16:9",
|
||||
image_size="2K",
|
||||
)
|
||||
|
||||
assert isinstance(response, GeneratedImageResponse)
|
||||
assert response.images == [PNG_DATA_URL]
|
||||
assert response.content == "done"
|
||||
|
||||
call = fake.calls[0]
|
||||
assert call["url"] == "https://openrouter.ai/api/v1/chat/completions"
|
||||
assert call["headers"]["Authorization"] == "Bearer sk-or-test"
|
||||
assert call["headers"]["X-Test"] == "1"
|
||||
body = call["json"]
|
||||
assert body["modalities"] == ["image", "text"]
|
||||
assert body["image_config"] == {"aspect_ratio": "16:9", "image_size": "2K"}
|
||||
assert body["messages"][0]["content"][0] == {"type": "text", "text": "make this blue"}
|
||||
assert body["messages"][0]["content"][1]["image_url"]["url"].startswith("data:image/png;base64,")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openrouter_image_generation_requires_images() -> None:
|
||||
fake = FakeClient(FakeResponse({"choices": [{"message": {"content": "text only"}}]}))
|
||||
client = OpenRouterImageGenerationClient(api_key="sk-or-test", client=fake) # type: ignore[arg-type]
|
||||
|
||||
with pytest.raises(ImageGenerationError, match="returned no images"):
|
||||
await client.generate(prompt="draw", model="model")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_openrouter_image_generation_requires_api_key() -> None:
|
||||
client = OpenRouterImageGenerationClient(api_key=None)
|
||||
|
||||
with pytest.raises(ImageGenerationError, match="API key"):
|
||||
await client.generate(prompt="draw", model="model")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aihubmix_image_generation_payload_and_response() -> None:
|
||||
raw_b64 = PNG_DATA_URL.removeprefix("data:image/png;base64,")
|
||||
fake = FakeClient(FakeResponse({"output": {"b64_json": [{"bytesBase64": raw_b64}]}}))
|
||||
client = AIHubMixImageGenerationClient(
|
||||
api_key="sk-ahm-test",
|
||||
api_base="https://aihubmix.com/v1/",
|
||||
extra_headers={"APP-Code": "nanobot"},
|
||||
extra_body={"quality": "low"},
|
||||
client=fake, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
response = await client.generate(
|
||||
prompt="draw a logo",
|
||||
model="gpt-image-2-free",
|
||||
aspect_ratio="16:9",
|
||||
image_size="1K",
|
||||
)
|
||||
|
||||
assert response.images == [PNG_DATA_URL]
|
||||
call = fake.calls[0]
|
||||
assert call["url"] == "https://aihubmix.com/v1/models/openai/gpt-image-2-free/predictions"
|
||||
assert call["headers"]["Authorization"] == "Bearer sk-ahm-test"
|
||||
assert call["headers"]["APP-Code"] == "nanobot"
|
||||
assert call["json"] == {
|
||||
"input": {
|
||||
"prompt": "draw a logo",
|
||||
"n": 1,
|
||||
"size": "1536x1024",
|
||||
"quality": "low",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aihubmix_image_edit_payload_uses_reference_images(tmp_path: Path) -> None:
|
||||
raw_b64 = PNG_DATA_URL.removeprefix("data:image/png;base64,")
|
||||
fake = FakeClient(FakeResponse({"output": [{"b64_json": raw_b64}]}))
|
||||
ref = tmp_path / "ref.png"
|
||||
ref.write_bytes(PNG_BYTES)
|
||||
client = AIHubMixImageGenerationClient(
|
||||
api_key="sk-ahm-test",
|
||||
client=fake, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
response = await client.generate(
|
||||
prompt="edit this",
|
||||
model="gpt-image-2-free",
|
||||
reference_images=[str(ref)],
|
||||
aspect_ratio="1:1",
|
||||
)
|
||||
|
||||
assert response.images == [PNG_DATA_URL]
|
||||
call = fake.calls[0]
|
||||
assert call["url"] == "https://aihubmix.com/v1/models/openai/gpt-image-2-free/predictions"
|
||||
assert call["json"]["input"]["prompt"] == "edit this"
|
||||
assert call["json"]["input"]["n"] == 1
|
||||
assert call["json"]["input"]["size"] == "1024x1024"
|
||||
assert call["json"]["input"]["image"].startswith("data:image/png;base64,")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_aihubmix_image_generation_downloads_url_response() -> None:
|
||||
fake = FakeClient(FakeResponse({"data": [{"url": "https://cdn.example/image.png"}]}))
|
||||
fake.get_response = FakeResponse({}, content=PNG_BYTES)
|
||||
client = AIHubMixImageGenerationClient(
|
||||
api_key="sk-ahm-test",
|
||||
client=fake, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
response = await client.generate(prompt="draw", model="gpt-image-2-free")
|
||||
|
||||
assert response.images[0].startswith("data:image/png;base64,")
|
||||
assert fake.get_calls[0]["url"] == "https://cdn.example/image.png"
|
||||
154
tests/tools/test_image_generation_tool.py
Normal file
154
tests/tools/test_image_generation_tool.py
Normal file
@ -0,0 +1,154 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from nanobot.agent.tools.image_generation import ImageGenerationTool
|
||||
from nanobot.config.loader import set_config_path
|
||||
from nanobot.config.schema import ImageGenerationToolConfig, ProviderConfig
|
||||
from nanobot.providers.image_generation import GeneratedImageResponse
|
||||
|
||||
PNG_BYTES = (
|
||||
b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01"
|
||||
b"\x00\x00\x00\x01\x08\x04\x00\x00\x00\xb5\x1c\x0c\x02"
|
||||
b"\x00\x00\x00\x0bIDATx\xdacd\xfc\xff\x1f\x00\x03\x03"
|
||||
b"\x02\x00\xef\xbf\xa7\xdb\x00\x00\x00\x00IEND\xaeB`\x82"
|
||||
)
|
||||
PNG_DATA_URL = (
|
||||
"data:image/png;base64,"
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
class FakeImageClient:
|
||||
instances: list["FakeImageClient"] = []
|
||||
|
||||
def __init__(self, **kwargs: Any) -> None:
|
||||
self.kwargs = kwargs
|
||||
self.calls: list[dict[str, Any]] = []
|
||||
self.instances.append(self)
|
||||
|
||||
async def generate(self, **kwargs: Any) -> GeneratedImageResponse:
|
||||
self.calls.append(kwargs)
|
||||
return GeneratedImageResponse(images=[PNG_DATA_URL], content="", raw={})
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generate_image_tool_stores_artifact_and_source_images(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
FakeImageClient.instances = []
|
||||
monkeypatch.setattr(
|
||||
"nanobot.agent.tools.image_generation.OpenRouterImageGenerationClient",
|
||||
FakeImageClient,
|
||||
)
|
||||
ref = tmp_path / "ref.png"
|
||||
ref.write_bytes(PNG_BYTES)
|
||||
tool = ImageGenerationTool(
|
||||
workspace=tmp_path,
|
||||
config=ImageGenerationToolConfig(enabled=True, max_images_per_turn=2),
|
||||
provider_config=ProviderConfig(api_key="sk-or-test"),
|
||||
)
|
||||
|
||||
result = await tool.execute(
|
||||
prompt="make this blue",
|
||||
reference_images=["ref.png"],
|
||||
aspect_ratio="16:9",
|
||||
image_size="2K",
|
||||
count=2,
|
||||
)
|
||||
|
||||
payload = json.loads(result)
|
||||
artifacts = payload["artifacts"]
|
||||
assert len(artifacts) == 2
|
||||
assert Path(artifacts[0]["path"]).is_file()
|
||||
assert artifacts[0]["source_images"] == [str(ref.resolve())]
|
||||
assert artifacts[0]["model"] == "openai/gpt-5.4-image-2"
|
||||
|
||||
fake = FakeImageClient.instances[0]
|
||||
assert fake.kwargs["api_key"] == "sk-or-test"
|
||||
assert len(fake.calls) == 2
|
||||
assert fake.calls[0]["aspect_ratio"] == "16:9"
|
||||
assert fake.calls[0]["image_size"] == "2K"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generate_image_tool_reports_missing_key(tmp_path: Path) -> None:
|
||||
tool = ImageGenerationTool(
|
||||
workspace=tmp_path,
|
||||
config=ImageGenerationToolConfig(enabled=True),
|
||||
provider_config=ProviderConfig(),
|
||||
)
|
||||
|
||||
result = await tool.execute(prompt="draw")
|
||||
|
||||
assert result.startswith("Error: OpenRouter API key is not configured")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generate_image_tool_selects_aihubmix_provider(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
FakeImageClient.instances = []
|
||||
monkeypatch.setattr(
|
||||
"nanobot.agent.tools.image_generation.AIHubMixImageGenerationClient",
|
||||
FakeImageClient,
|
||||
)
|
||||
tool = ImageGenerationTool(
|
||||
workspace=tmp_path,
|
||||
config=ImageGenerationToolConfig(
|
||||
enabled=True,
|
||||
provider="aihubmix",
|
||||
model="gpt-image-2-free",
|
||||
),
|
||||
provider_configs={
|
||||
"openrouter": ProviderConfig(api_key="sk-or-test"),
|
||||
"aihubmix": ProviderConfig(api_key="sk-ahm-test", extra_body={"quality": "low"}),
|
||||
},
|
||||
)
|
||||
|
||||
result = await tool.execute(prompt="draw a poster", aspect_ratio="3:4")
|
||||
|
||||
payload = json.loads(result)
|
||||
assert len(payload["artifacts"]) == 1
|
||||
fake = FakeImageClient.instances[0]
|
||||
assert fake.kwargs["api_key"] == "sk-ahm-test"
|
||||
assert fake.kwargs["extra_body"] == {"quality": "low"}
|
||||
assert fake.calls[0]["model"] == "gpt-image-2-free"
|
||||
assert fake.calls[0]["aspect_ratio"] == "3:4"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generate_image_tool_reports_missing_aihubmix_key(tmp_path: Path) -> None:
|
||||
tool = ImageGenerationTool(
|
||||
workspace=tmp_path,
|
||||
config=ImageGenerationToolConfig(enabled=True, provider="aihubmix"),
|
||||
provider_configs={"aihubmix": ProviderConfig()},
|
||||
)
|
||||
|
||||
result = await tool.execute(prompt="draw")
|
||||
|
||||
assert result.startswith("Error: AIHubMix API key is not configured")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_generate_image_tool_rejects_reference_outside_workspace(tmp_path: Path) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
outside = tmp_path.parent / "outside.png"
|
||||
outside.write_bytes(PNG_BYTES)
|
||||
tool = ImageGenerationTool(
|
||||
workspace=tmp_path,
|
||||
config=ImageGenerationToolConfig(enabled=True),
|
||||
provider_config=ProviderConfig(api_key="sk-or-test"),
|
||||
)
|
||||
|
||||
result = await tool.execute(prompt="edit", reference_images=[str(outside)])
|
||||
|
||||
assert "reference_images must be inside the workspace" in result
|
||||
@ -55,6 +55,25 @@ async def test_message_tool_marks_channel_delivery_only_when_enabled() -> None:
|
||||
assert sent[1].metadata == {"_record_channel_delivery": True}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_message_tool_records_media_deliveries() -> None:
|
||||
sent: list[OutboundMessage] = []
|
||||
|
||||
async def _send(msg: OutboundMessage) -> None:
|
||||
sent.append(msg)
|
||||
|
||||
tool = MessageTool(send_callback=_send)
|
||||
|
||||
await tool.execute(
|
||||
content="image",
|
||||
channel="websocket",
|
||||
chat_id="chat-1",
|
||||
media=["/tmp/generated.png"],
|
||||
)
|
||||
|
||||
assert sent[0].metadata == {"_record_channel_delivery": True}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_message_tool_inherits_metadata_for_same_target() -> None:
|
||||
sent: list[OutboundMessage] = []
|
||||
|
||||
84
tests/utils/test_artifacts.py
Normal file
84
tests/utils/test_artifacts.py
Normal file
@ -0,0 +1,84 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from nanobot.config.loader import set_config_path
|
||||
from nanobot.utils.artifacts import (
|
||||
ArtifactError,
|
||||
decode_image_data_url,
|
||||
generated_image_paths_from_messages,
|
||||
generated_image_tool_result,
|
||||
store_generated_image_artifact,
|
||||
)
|
||||
|
||||
PNG_DATA_URL = (
|
||||
"data:image/png;base64,"
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+/p9sAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
|
||||
def test_decode_image_data_url_validates_image_payload() -> None:
|
||||
raw, mime = decode_image_data_url(PNG_DATA_URL)
|
||||
|
||||
assert raw.startswith(b"\x89PNG")
|
||||
assert mime == "image/png"
|
||||
|
||||
with pytest.raises(ArtifactError):
|
||||
decode_image_data_url("data:image/png;base64,not-base64")
|
||||
|
||||
|
||||
def test_store_generated_image_artifact_writes_image_and_sidecar(tmp_path: Path) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
created_at = datetime(2026, 5, 8, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
artifact = store_generated_image_artifact(
|
||||
PNG_DATA_URL,
|
||||
prompt="draw a tiny pixel",
|
||||
model="openai/gpt-5.4-image-2",
|
||||
source_images=["/tmp/ref.png"],
|
||||
save_dir="generated",
|
||||
created_at=created_at,
|
||||
)
|
||||
|
||||
image_path = Path(artifact["path"])
|
||||
assert image_path.is_file()
|
||||
assert image_path.parent == tmp_path / "media" / "generated" / "2026-05-08"
|
||||
assert artifact["id"].startswith("img_")
|
||||
assert artifact["mime"] == "image/png"
|
||||
|
||||
sidecar = image_path.with_suffix(".json")
|
||||
metadata = json.loads(sidecar.read_text(encoding="utf-8"))
|
||||
assert metadata["path"] == str(image_path)
|
||||
assert metadata["source_images"] == ["/tmp/ref.png"]
|
||||
|
||||
|
||||
def test_store_generated_image_artifact_rejects_unsafe_save_dir(tmp_path: Path) -> None:
|
||||
set_config_path(tmp_path / "config.json")
|
||||
|
||||
with pytest.raises(ArtifactError):
|
||||
store_generated_image_artifact(
|
||||
PNG_DATA_URL,
|
||||
prompt="x",
|
||||
model="m",
|
||||
save_dir="../outside",
|
||||
)
|
||||
|
||||
|
||||
def test_generated_image_paths_from_tool_results() -> None:
|
||||
result = generated_image_tool_result(
|
||||
[
|
||||
{"id": "img_1", "path": "/tmp/one.png"},
|
||||
{"id": "img_2", "path": "/tmp/two.png"},
|
||||
]
|
||||
)
|
||||
|
||||
assert generated_image_paths_from_messages(
|
||||
[
|
||||
{"role": "tool", "name": "generate_image", "content": result},
|
||||
{"role": "tool", "name": "other", "content": result},
|
||||
]
|
||||
) == ["/tmp/one.png", "/tmp/two.png"]
|
||||
25
tests/utils/test_image_generation_intent.py
Normal file
25
tests/utils/test_image_generation_intent.py
Normal file
@ -0,0 +1,25 @@
|
||||
from nanobot.utils.image_generation_intent import image_generation_prompt
|
||||
|
||||
|
||||
def test_image_generation_prompt_ignores_plain_messages() -> None:
|
||||
assert image_generation_prompt("hello", {}) == "hello"
|
||||
|
||||
|
||||
def test_image_generation_prompt_uses_auto_aspect_instruction() -> None:
|
||||
prompt = image_generation_prompt(
|
||||
"Draw a poster",
|
||||
{"image_generation": {"enabled": True, "aspect_ratio": None}},
|
||||
)
|
||||
|
||||
assert "Draw a poster" in prompt
|
||||
assert "Use the generate_image tool" in prompt
|
||||
assert "Choose the most suitable aspect_ratio yourself" in prompt
|
||||
|
||||
|
||||
def test_image_generation_prompt_uses_selected_aspect_ratio() -> None:
|
||||
prompt = image_generation_prompt(
|
||||
"Draw a banner",
|
||||
{"image_generation": {"enabled": True, "aspect_ratio": "16:9"}},
|
||||
)
|
||||
|
||||
assert "aspect_ratio='16:9'" in prompt
|
||||
@ -17,7 +17,7 @@ import {
|
||||
saveSecret,
|
||||
} from "@/lib/bootstrap";
|
||||
import { NanobotClient } from "@/lib/nanobot-client";
|
||||
import { ClientProvider } from "@/providers/ClientProvider";
|
||||
import { ClientProvider, useClient } from "@/providers/ClientProvider";
|
||||
import type { ChatSummary } from "@/lib/types";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
@ -34,6 +34,7 @@ type BootState =
|
||||
};
|
||||
|
||||
const SIDEBAR_STORAGE_KEY = "nanobot-webui.sidebar";
|
||||
const RESTART_STARTED_KEY = "nanobot-webui.restartStartedAt";
|
||||
const SIDEBAR_WIDTH = 272;
|
||||
type ShellView = "chat" | "settings";
|
||||
|
||||
@ -237,6 +238,7 @@ export default function App() {
|
||||
|
||||
function Shell({ onModelNameChange, onLogout }: { onModelNameChange: (modelName: string | null) => void; onLogout: () => void }) {
|
||||
const { t, i18n } = useTranslation();
|
||||
const { client } = useClient();
|
||||
const { theme, toggle } = useTheme();
|
||||
const { sessions, loading, refresh, createChat, deleteChat } = useSessions();
|
||||
const [activeKey, setActiveKey] = useState<string | null>(null);
|
||||
@ -249,6 +251,8 @@ function Shell({ onModelNameChange, onLogout }: { onModelNameChange: (modelName:
|
||||
label: string;
|
||||
} | null>(null);
|
||||
const lastSessionsLen = useRef(0);
|
||||
const restartSawDisconnectRef = useRef(false);
|
||||
const [restartToast, setRestartToast] = useState<string | null>(null);
|
||||
|
||||
useEffect(() => {
|
||||
try {
|
||||
@ -326,6 +330,43 @@ function Shell({ onModelNameChange, onLogout }: { onModelNameChange: (modelName:
|
||||
setMobileSidebarOpen(false);
|
||||
}, []);
|
||||
|
||||
const onRestart = useCallback(() => {
|
||||
const chatId = activeSession?.chatId ?? client.defaultChatId;
|
||||
if (!chatId) return;
|
||||
restartSawDisconnectRef.current = false;
|
||||
try {
|
||||
window.localStorage.setItem(RESTART_STARTED_KEY, String(Date.now()));
|
||||
} catch {
|
||||
// ignore storage errors
|
||||
}
|
||||
client.sendMessage(chatId, "/restart");
|
||||
}, [activeSession?.chatId, client]);
|
||||
|
||||
useEffect(() => {
|
||||
return client.onStatus((status) => {
|
||||
let startedAt = 0;
|
||||
try {
|
||||
startedAt = Number(window.localStorage.getItem(RESTART_STARTED_KEY) ?? "0");
|
||||
} catch {
|
||||
startedAt = 0;
|
||||
}
|
||||
if (!startedAt) return;
|
||||
if (status !== "open") {
|
||||
restartSawDisconnectRef.current = true;
|
||||
return;
|
||||
}
|
||||
const elapsedMs = Date.now() - startedAt;
|
||||
if (!restartSawDisconnectRef.current && elapsedMs < 1500) return;
|
||||
try {
|
||||
window.localStorage.removeItem(RESTART_STARTED_KEY);
|
||||
} catch {
|
||||
// ignore storage errors
|
||||
}
|
||||
setRestartToast(t("app.restart.completed", { seconds: (elapsedMs / 1000).toFixed(1) }));
|
||||
window.setTimeout(() => setRestartToast(null), 3_500);
|
||||
});
|
||||
}, [client, t]);
|
||||
|
||||
const onTurnEnd = useCallback(() => {
|
||||
void refresh();
|
||||
}, [refresh]);
|
||||
@ -414,6 +455,7 @@ function Shell({ onModelNameChange, onLogout }: { onModelNameChange: (modelName:
|
||||
onBackToChat={() => setView("chat")}
|
||||
onModelNameChange={onModelNameChange}
|
||||
onLogout={onLogout}
|
||||
onRestart={onRestart}
|
||||
/>
|
||||
) : (
|
||||
<ThreadShell
|
||||
@ -437,6 +479,14 @@ function Shell({ onModelNameChange, onLogout }: { onModelNameChange: (modelName:
|
||||
onCancel={() => setPendingDelete(null)}
|
||||
onConfirm={onConfirmDelete}
|
||||
/>
|
||||
{restartToast ? (
|
||||
<div
|
||||
role="status"
|
||||
className="fixed left-1/2 top-4 z-50 -translate-x-1/2 rounded-full border border-border/70 bg-popover px-4 py-2 text-sm font-medium text-popover-foreground shadow-lg"
|
||||
>
|
||||
{restartToast}
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@ -142,7 +142,9 @@ function MessageMedia({
|
||||
align === "right" ? "justify-end" : "justify-start",
|
||||
)}
|
||||
>
|
||||
{images.length > 0 ? <UserImages images={images} align={align} /> : null}
|
||||
{images.length > 0 ? (
|
||||
<UserImages images={images} align={align} size={align === "left" ? "large" : "compact"} />
|
||||
) : null}
|
||||
{nonImages.map((item, i) => (
|
||||
<MediaCell key={`${item.url ?? item.name ?? item.kind}-${i}`} media={item} />
|
||||
))}
|
||||
@ -208,9 +210,11 @@ function MediaCell({ media }: { media: UIMediaAttachment }) {
|
||||
function UserImages({
|
||||
images,
|
||||
align = "right",
|
||||
size = "compact",
|
||||
}: {
|
||||
images: UIImage[];
|
||||
align?: "left" | "right";
|
||||
size?: "compact" | "large";
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
// Only real-URL images can open in the lightbox; historical-replay
|
||||
@ -230,6 +234,7 @@ function UserImages({
|
||||
<div
|
||||
className={cn(
|
||||
"flex flex-wrap items-end gap-2",
|
||||
size === "large" && "gap-3",
|
||||
align === "right" ? "ml-auto justify-end" : "mr-auto justify-start",
|
||||
)}
|
||||
>
|
||||
@ -237,6 +242,7 @@ function UserImages({
|
||||
<UserImageCell
|
||||
key={`${img.url ?? "placeholder"}-${i}`}
|
||||
image={img}
|
||||
size={size}
|
||||
placeholderLabel={t("message.imageAttachment")}
|
||||
openLabel={t("lightbox.open")}
|
||||
onOpen={
|
||||
@ -261,18 +267,23 @@ function UserImages({
|
||||
|
||||
function UserImageCell({
|
||||
image,
|
||||
size,
|
||||
placeholderLabel,
|
||||
openLabel,
|
||||
onOpen,
|
||||
}: {
|
||||
image: UIImage;
|
||||
size: "compact" | "large";
|
||||
placeholderLabel: string;
|
||||
openLabel: string;
|
||||
onOpen?: () => void;
|
||||
}) {
|
||||
const hasUrl = typeof image.url === "string" && image.url.length > 0;
|
||||
const tileClasses = cn(
|
||||
"relative h-24 w-24 overflow-hidden rounded-[14px] border border-border/60 bg-muted/40",
|
||||
"relative overflow-hidden border border-border/60 bg-muted/40",
|
||||
size === "large"
|
||||
? "h-56 w-[min(100%,22rem)] rounded-[18px] sm:h-72 sm:w-[26rem]"
|
||||
: "h-24 w-24 rounded-[14px]",
|
||||
"shadow-[0_6px_18px_-14px_rgba(0,0,0,0.45)]",
|
||||
);
|
||||
|
||||
@ -296,7 +307,7 @@ function UserImageCell({
|
||||
loading="lazy"
|
||||
decoding="async"
|
||||
draggable={false}
|
||||
className="h-full w-full object-cover"
|
||||
className={cn("h-full w-full", size === "large" ? "object-contain" : "object-cover")}
|
||||
/>
|
||||
</button>
|
||||
);
|
||||
|
||||
@ -16,12 +16,14 @@ interface SettingsViewProps {
|
||||
onBackToChat: () => void;
|
||||
onModelNameChange: (modelName: string | null) => void;
|
||||
onLogout?: () => void;
|
||||
onRestart?: () => void;
|
||||
}
|
||||
|
||||
export function SettingsView({
|
||||
onBackToChat,
|
||||
onModelNameChange,
|
||||
onLogout,
|
||||
onRestart,
|
||||
}: SettingsViewProps) {
|
||||
const { token } = useClient();
|
||||
const [settings, setSettings] = useState<SettingsPayload | null>(null);
|
||||
@ -119,6 +121,7 @@ export function SettingsView({
|
||||
saving={saving}
|
||||
onSave={save}
|
||||
onLogout={onLogout}
|
||||
onRestart={onRestart}
|
||||
/>
|
||||
) : null}
|
||||
</main>
|
||||
@ -134,6 +137,7 @@ function SettingsSection({
|
||||
saving,
|
||||
onSave,
|
||||
onLogout,
|
||||
onRestart,
|
||||
}: {
|
||||
form: {
|
||||
model: string;
|
||||
@ -148,6 +152,7 @@ function SettingsSection({
|
||||
saving: boolean;
|
||||
onSave: () => void;
|
||||
onLogout?: () => void;
|
||||
onRestart?: () => void;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
return (
|
||||
@ -200,6 +205,19 @@ function SettingsSection({
|
||||
</SettingsGroup>
|
||||
</section>
|
||||
|
||||
{onRestart && (
|
||||
<section>
|
||||
<h2 className="mb-2 px-2 text-xs font-medium text-muted-foreground">{t("app.system.section")}</h2>
|
||||
<SettingsGroup>
|
||||
<SettingsRow title={t("app.system.restartHint")}>
|
||||
<Button size="sm" variant="outline" onClick={onRestart}>
|
||||
{t("app.system.restart")}
|
||||
</Button>
|
||||
</SettingsRow>
|
||||
</SettingsGroup>
|
||||
</section>
|
||||
)}
|
||||
|
||||
{onLogout && (
|
||||
<section>
|
||||
<h2 className="mb-2 px-2 text-xs font-medium text-muted-foreground">{t("app.account.section")}</h2>
|
||||
|
||||
@ -10,6 +10,8 @@ import {
|
||||
Activity,
|
||||
ArrowUp,
|
||||
BookOpen,
|
||||
Check,
|
||||
ChevronDown,
|
||||
CircleHelp,
|
||||
History,
|
||||
ImageIcon,
|
||||
@ -33,7 +35,7 @@ import {
|
||||
MAX_IMAGES_PER_MESSAGE,
|
||||
} from "@/hooks/useAttachedImages";
|
||||
import { useClipboardAndDrop } from "@/hooks/useClipboardAndDrop";
|
||||
import type { SendImage } from "@/hooks/useNanobotStream";
|
||||
import type { SendImage, SendOptions } from "@/hooks/useNanobotStream";
|
||||
import type { SlashCommand } from "@/lib/types";
|
||||
import { cn } from "@/lib/utils";
|
||||
|
||||
@ -48,13 +50,16 @@ function formatBytes(n: number): string {
|
||||
}
|
||||
|
||||
interface ThreadComposerProps {
|
||||
onSend: (content: string, images?: SendImage[]) => void;
|
||||
onSend: (content: string, images?: SendImage[], options?: SendOptions) => void;
|
||||
disabled?: boolean;
|
||||
placeholder?: string;
|
||||
isStreaming?: boolean;
|
||||
modelLabel?: string | null;
|
||||
variant?: "thread" | "hero";
|
||||
slashCommands?: SlashCommand[];
|
||||
imageMode?: boolean;
|
||||
onImageModeChange?: (enabled: boolean) => void;
|
||||
onStop?: () => void;
|
||||
}
|
||||
|
||||
const COMMAND_ICONS: Record<string, LucideIcon> = {
|
||||
@ -69,10 +74,28 @@ const COMMAND_ICONS: Record<string, LucideIcon> = {
|
||||
"undo-2": Undo2,
|
||||
};
|
||||
|
||||
type ImageAspectRatio = "auto" | "1:1" | "3:4" | "9:16" | "4:3" | "16:9";
|
||||
|
||||
const IMAGE_ASPECT_RATIOS: ImageAspectRatio[] = ["auto", "1:1", "3:4", "9:16", "4:3", "16:9"];
|
||||
|
||||
function slashCommandI18nKey(command: string): string {
|
||||
return command.replace(/^\//, "").replace(/-/g, "_");
|
||||
}
|
||||
|
||||
function scrollNearestOverflowParent(target: EventTarget | null, deltaY: number) {
|
||||
if (!(target instanceof Element) || deltaY === 0) return;
|
||||
let el: HTMLElement | null = target.parentElement;
|
||||
while (el) {
|
||||
const style = window.getComputedStyle(el);
|
||||
const canScroll = /(auto|scroll)/.test(style.overflowY) && el.scrollHeight > el.clientHeight;
|
||||
if (canScroll) {
|
||||
el.scrollTop += deltaY;
|
||||
return;
|
||||
}
|
||||
el = el.parentElement;
|
||||
}
|
||||
}
|
||||
|
||||
export function ThreadComposer({
|
||||
onSend,
|
||||
disabled,
|
||||
@ -81,19 +104,38 @@ export function ThreadComposer({
|
||||
modelLabel = null,
|
||||
variant = "thread",
|
||||
slashCommands = [],
|
||||
imageMode: controlledImageMode,
|
||||
onImageModeChange,
|
||||
onStop,
|
||||
}: ThreadComposerProps) {
|
||||
const { t } = useTranslation();
|
||||
const [value, setValue] = useState("");
|
||||
const [inlineError, setInlineError] = useState<string | null>(null);
|
||||
const [slashMenuDismissed, setSlashMenuDismissed] = useState(false);
|
||||
const [selectedCommandIndex, setSelectedCommandIndex] = useState(0);
|
||||
const [uncontrolledImageMode, setUncontrolledImageMode] = useState(false);
|
||||
const [imageAspectRatio, setImageAspectRatio] = useState<ImageAspectRatio>("auto");
|
||||
const [aspectMenuOpen, setAspectMenuOpen] = useState(false);
|
||||
const textareaRef = useRef<HTMLTextAreaElement>(null);
|
||||
const fileInputRef = useRef<HTMLInputElement>(null);
|
||||
const aspectControlRef = useRef<HTMLDivElement>(null);
|
||||
const chipRefs = useRef(new Map<string, HTMLButtonElement>());
|
||||
const isHero = variant === "hero";
|
||||
const imageMode = controlledImageMode ?? uncontrolledImageMode;
|
||||
const setImageMode = useCallback(
|
||||
(enabled: boolean) => {
|
||||
if (controlledImageMode === undefined) {
|
||||
setUncontrolledImageMode(enabled);
|
||||
}
|
||||
onImageModeChange?.(enabled);
|
||||
},
|
||||
[controlledImageMode, onImageModeChange],
|
||||
);
|
||||
const resolvedPlaceholder = isStreaming
|
||||
? t("thread.composer.placeholderStreaming")
|
||||
: placeholder ?? t("thread.composer.placeholderThread");
|
||||
: imageMode
|
||||
? t("thread.composer.imageMode.placeholder")
|
||||
: placeholder ?? t("thread.composer.placeholderThread");
|
||||
|
||||
const { images, enqueue, remove, clear, encoding, full } =
|
||||
useAttachedImages();
|
||||
@ -190,6 +232,38 @@ export function ThreadComposer({
|
||||
}
|
||||
}, [filteredSlashCommands.length, selectedCommandIndex]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!aspectMenuOpen) return;
|
||||
|
||||
const closeOnPointerDown = (event: PointerEvent) => {
|
||||
const target = event.target;
|
||||
if (target instanceof Node && aspectControlRef.current?.contains(target)) return;
|
||||
setAspectMenuOpen(false);
|
||||
};
|
||||
const closeOnKeyDown = (event: KeyboardEvent) => {
|
||||
if (event.key === "Escape") {
|
||||
setAspectMenuOpen(false);
|
||||
textareaRef.current?.focus();
|
||||
}
|
||||
};
|
||||
const closeOnScroll = () => setAspectMenuOpen(false);
|
||||
const closeOnWheel = (event: WheelEvent) => {
|
||||
setAspectMenuOpen(false);
|
||||
scrollNearestOverflowParent(event.target, event.deltaY);
|
||||
};
|
||||
|
||||
document.addEventListener("pointerdown", closeOnPointerDown, true);
|
||||
document.addEventListener("keydown", closeOnKeyDown);
|
||||
document.addEventListener("scroll", closeOnScroll, true);
|
||||
document.addEventListener("wheel", closeOnWheel, { capture: true, passive: true });
|
||||
return () => {
|
||||
document.removeEventListener("pointerdown", closeOnPointerDown, true);
|
||||
document.removeEventListener("keydown", closeOnKeyDown);
|
||||
document.removeEventListener("scroll", closeOnScroll, true);
|
||||
document.removeEventListener("wheel", closeOnWheel, true);
|
||||
};
|
||||
}, [aspectMenuOpen]);
|
||||
|
||||
const resizeTextarea = useCallback(() => {
|
||||
requestAnimationFrame(() => {
|
||||
const el = textareaRef.current;
|
||||
@ -227,7 +301,15 @@ export function ThreadComposer({
|
||||
preview: { url: img.dataUrl, name: img.file.name },
|
||||
}))
|
||||
: undefined;
|
||||
onSend(trimmed, payload);
|
||||
const options: SendOptions | undefined = imageMode
|
||||
? {
|
||||
imageGeneration: {
|
||||
enabled: true,
|
||||
aspect_ratio: imageAspectRatio === "auto" ? null : imageAspectRatio,
|
||||
},
|
||||
}
|
||||
: undefined;
|
||||
onSend(trimmed, payload, options);
|
||||
setValue("");
|
||||
setInlineError(null);
|
||||
// Bubble owns the data URL copy; safe to revoke every staged blob
|
||||
@ -235,7 +317,7 @@ export function ThreadComposer({
|
||||
clear();
|
||||
setSlashMenuDismissed(false);
|
||||
resizeTextarea();
|
||||
}, [canSend, clear, onSend, readyImages, resizeTextarea, value]);
|
||||
}, [canSend, clear, imageAspectRatio, imageMode, onSend, readyImages, resizeTextarea, value]);
|
||||
|
||||
const onKeyDown = (e: ReactKeyboardEvent<HTMLTextAreaElement>) => {
|
||||
if (showSlashMenu) {
|
||||
@ -312,6 +394,7 @@ export function ThreadComposer({
|
||||
);
|
||||
|
||||
const attachButtonDisabled = disabled || full;
|
||||
const showStopButton = isStreaming && !!onStop;
|
||||
|
||||
return (
|
||||
<form
|
||||
@ -336,7 +419,7 @@ export function ThreadComposer({
|
||||
) : null}
|
||||
<div
|
||||
className={cn(
|
||||
"relative mx-auto flex w-full flex-col overflow-hidden transition-all duration-200",
|
||||
"relative mx-auto flex w-full flex-col overflow-visible transition-all duration-200",
|
||||
isHero
|
||||
? "max-w-[58rem] rounded-[28px] border border-black/[0.035] bg-card shadow-[0_20px_55px_rgba(15,23,42,0.08)] dark:border-white/[0.06] dark:shadow-[0_24px_55px_rgba(0,0,0,0.34)]"
|
||||
: "max-w-[49.5rem] rounded-[22px] border border-black/[0.035] bg-card shadow-[0_12px_30px_rgba(15,23,42,0.07)] dark:border-white/[0.06] dark:shadow-[0_16px_34px_rgba(0,0,0,0.28)]",
|
||||
@ -439,6 +522,59 @@ export function ThreadComposer({
|
||||
>
|
||||
<Plus className={cn(isHero ? "h-5 w-5" : "h-4 w-4")} />
|
||||
</Button>
|
||||
<div ref={aspectControlRef} className="relative flex items-center gap-1">
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
disabled={disabled}
|
||||
aria-pressed={imageMode}
|
||||
aria-label={t("thread.composer.imageMode.toggle")}
|
||||
onClick={() => {
|
||||
setImageMode(!imageMode);
|
||||
setAspectMenuOpen(false);
|
||||
textareaRef.current?.focus();
|
||||
}}
|
||||
className={cn(
|
||||
"rounded-full border border-border/55 px-2.5 font-medium shadow-[0_2px_8px_rgba(15,23,42,0.04)]",
|
||||
isHero ? "h-9 text-[12px]" : "h-7.5 text-[10.5px]",
|
||||
imageMode
|
||||
? "border-primary/30 bg-primary/10 text-primary hover:bg-primary/12"
|
||||
: "bg-card text-muted-foreground hover:bg-card hover:text-foreground",
|
||||
)}
|
||||
>
|
||||
<ImageIcon className={cn("mr-1.5", isHero ? "h-4 w-4" : "h-3.5 w-3.5")} />
|
||||
{t("thread.composer.imageMode.label")}
|
||||
</Button>
|
||||
{imageMode ? (
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
disabled={disabled}
|
||||
aria-haspopup="listbox"
|
||||
aria-expanded={aspectMenuOpen}
|
||||
aria-label={t("thread.composer.imageMode.aspectAria")}
|
||||
onClick={() => setAspectMenuOpen((open) => !open)}
|
||||
className={cn(
|
||||
"rounded-full border border-border/55 bg-card px-2.5 font-medium text-foreground/80 shadow-[0_2px_8px_rgba(15,23,42,0.04)] hover:bg-card",
|
||||
isHero ? "h-9 text-[12px]" : "h-7.5 text-[10.5px]",
|
||||
)}
|
||||
>
|
||||
<span>{t(`thread.composer.imageMode.aspect.${imageAspectRatio.replace(":", "_")}`)}</span>
|
||||
<ChevronDown className={cn("ml-1.5", isHero ? "h-3.5 w-3.5" : "h-3 w-3")} />
|
||||
</Button>
|
||||
) : null}
|
||||
{imageMode && aspectMenuOpen ? (
|
||||
<ImageAspectMenu
|
||||
selected={imageAspectRatio}
|
||||
isHero={isHero}
|
||||
onSelect={(ratio) => {
|
||||
setImageAspectRatio(ratio);
|
||||
setAspectMenuOpen(false);
|
||||
textareaRef.current?.focus();
|
||||
}}
|
||||
/>
|
||||
) : null}
|
||||
</div>
|
||||
{modelLabel ? (
|
||||
<span
|
||||
title={modelLabel}
|
||||
@ -465,19 +601,25 @@ export function ThreadComposer({
|
||||
</div>
|
||||
<span className={cn(isHero ? "hidden" : "sm:hidden")} aria-hidden />
|
||||
<Button
|
||||
type="submit"
|
||||
type={showStopButton ? "button" : "submit"}
|
||||
size="icon"
|
||||
disabled={!canSend}
|
||||
aria-label={t("thread.composer.send")}
|
||||
disabled={showStopButton ? disabled : !canSend}
|
||||
aria-label={showStopButton ? t("thread.composer.stop") : t("thread.composer.send")}
|
||||
onClick={showStopButton ? onStop : undefined}
|
||||
className={cn(
|
||||
isHero
|
||||
? "h-9 w-9 rounded-full border border-foreground bg-foreground text-background shadow-[0_4px_12px_rgba(15,23,42,0.20)] hover:bg-foreground/90 disabled:border-foreground/35 disabled:bg-foreground/35 disabled:text-background/80"
|
||||
: "rounded-full border border-foreground bg-foreground text-background shadow-[0_3px_10px_rgba(15,23,42,0.18)] transition-transform hover:bg-foreground/90 disabled:border-foreground/35 disabled:bg-foreground/35 disabled:text-background/80",
|
||||
"rounded-full transition-transform",
|
||||
showStopButton
|
||||
? "border border-border/70 bg-card text-foreground/85 shadow-[0_3px_10px_rgba(15,23,42,0.08)] hover:bg-muted/65 hover:text-foreground disabled:text-muted-foreground/50"
|
||||
: isHero
|
||||
? "border border-foreground bg-foreground text-background shadow-[0_4px_12px_rgba(15,23,42,0.20)] hover:bg-foreground/90 disabled:border-foreground/35 disabled:bg-foreground/35 disabled:text-background/80"
|
||||
: "border border-foreground bg-foreground text-background shadow-[0_3px_10px_rgba(15,23,42,0.18)] hover:bg-foreground/90 disabled:border-foreground/35 disabled:bg-foreground/35 disabled:text-background/80",
|
||||
isHero ? "" : "h-7.5 w-7.5",
|
||||
canSend && "hover:scale-[1.03] active:scale-95",
|
||||
(canSend || showStopButton) && "hover:scale-[1.03] active:scale-95",
|
||||
)}
|
||||
>
|
||||
{isStreaming ? (
|
||||
{showStopButton ? (
|
||||
<Square className={cn("fill-current stroke-current", isHero ? "h-3 w-3" : "h-2.5 w-2.5")} />
|
||||
) : isStreaming ? (
|
||||
<Loader2 className={cn(isHero ? "h-4.5 w-4.5" : "h-4 w-4", "animate-spin")} />
|
||||
) : (
|
||||
<ArrowUp className={cn(isHero ? "h-4.5 w-4.5" : "h-4 w-4")} />
|
||||
@ -497,6 +639,59 @@ interface SlashCommandPaletteProps {
|
||||
onChoose: (command: SlashCommand) => void;
|
||||
}
|
||||
|
||||
function ImageAspectMenu({
|
||||
selected,
|
||||
isHero,
|
||||
onSelect,
|
||||
}: {
|
||||
selected: ImageAspectRatio;
|
||||
isHero: boolean;
|
||||
onSelect: (ratio: ImageAspectRatio) => void;
|
||||
}) {
|
||||
const { t } = useTranslation();
|
||||
return (
|
||||
<div
|
||||
role="listbox"
|
||||
aria-label={t("thread.composer.imageMode.aspectAria")}
|
||||
className={cn(
|
||||
"absolute left-0 z-30 w-44 overflow-hidden rounded-[16px] border",
|
||||
isHero ? "top-full mt-2" : "bottom-full mb-2",
|
||||
"border-border/65 bg-popover p-1.5 text-popover-foreground shadow-[0_16px_45px_rgba(15,23,42,0.16)]",
|
||||
"dark:border-white/10 dark:shadow-[0_18px_45px_rgba(0,0,0,0.42)]",
|
||||
isHero ? "text-[12px]" : "text-[11.5px]",
|
||||
)}
|
||||
>
|
||||
<div className="px-2 pb-1 pt-1 font-medium text-muted-foreground/70">
|
||||
{t("thread.composer.imageMode.aspectLabel")}
|
||||
</div>
|
||||
{IMAGE_ASPECT_RATIOS.map((ratio) => {
|
||||
const label = t(`thread.composer.imageMode.aspect.${ratio.replace(":", "_")}`);
|
||||
return (
|
||||
<button
|
||||
key={ratio}
|
||||
type="button"
|
||||
role="option"
|
||||
aria-selected={selected === ratio}
|
||||
onMouseDown={(e) => {
|
||||
e.preventDefault();
|
||||
onSelect(ratio);
|
||||
}}
|
||||
className={cn(
|
||||
"flex w-full items-center justify-between rounded-[11px] px-2.5 py-2 text-left transition-colors",
|
||||
selected === ratio
|
||||
? "bg-primary/10 text-foreground"
|
||||
: "text-foreground/86 hover:bg-accent/55",
|
||||
)}
|
||||
>
|
||||
<span>{label}</span>
|
||||
{selected === ratio ? <Check className="h-3.5 w-3.5 text-primary" /> : null}
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function SlashCommandPalette({
|
||||
commands,
|
||||
selectedIndex,
|
||||
@ -511,7 +706,7 @@ function SlashCommandPalette({
|
||||
aria-label={t("thread.composer.slash.ariaLabel")}
|
||||
className={cn(
|
||||
"absolute bottom-full left-1/2 z-30 mb-2 max-h-[22rem] w-[calc(100%-0.5rem)] -translate-x-1/2 overflow-hidden rounded-[18px] border",
|
||||
"border-border/65 bg-popover/98 p-1.5 text-popover-foreground shadow-[0_18px_55px_rgba(15,23,42,0.18)] backdrop-blur",
|
||||
"border-border/65 bg-popover p-1.5 text-popover-foreground shadow-[0_18px_55px_rgba(15,23,42,0.18)]",
|
||||
"dark:border-white/10 dark:shadow-[0_22px_55px_rgba(0,0,0,0.45)]",
|
||||
isHero ? "max-w-[58rem]" : "max-w-[49.5rem]",
|
||||
)}
|
||||
|
||||
@ -4,9 +4,12 @@ import {
|
||||
BookOpen,
|
||||
ChevronRight,
|
||||
Code2,
|
||||
ImageIcon,
|
||||
LayoutGrid,
|
||||
Lightbulb,
|
||||
MoreHorizontal,
|
||||
Palette,
|
||||
Sparkles,
|
||||
} from "lucide-react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
|
||||
@ -15,7 +18,7 @@ import { ThreadComposer } from "@/components/thread/ThreadComposer";
|
||||
import { ThreadHeader } from "@/components/thread/ThreadHeader";
|
||||
import { StreamErrorNotice } from "@/components/thread/StreamErrorNotice";
|
||||
import { ThreadViewport } from "@/components/thread/ThreadViewport";
|
||||
import { useNanobotStream } from "@/hooks/useNanobotStream";
|
||||
import { useNanobotStream, type SendImage, type SendOptions } from "@/hooks/useNanobotStream";
|
||||
import { useSessionHistory } from "@/hooks/useSessions";
|
||||
import { listSlashCommands } from "@/lib/api";
|
||||
import type { ChatSummary, SlashCommand, UIMessage } from "@/lib/types";
|
||||
@ -52,6 +55,21 @@ const QUICK_ACTION_KEYS = [
|
||||
{ key: "more", icon: MoreHorizontal, tone: "text-muted-foreground/65" },
|
||||
] as const;
|
||||
|
||||
const IMAGE_QUICK_ACTION_KEYS = [
|
||||
{ key: "icon", icon: ImageIcon, tone: "text-[#4f9de8]" },
|
||||
{ key: "sticker", icon: Sparkles, tone: "text-[#f25b8f]" },
|
||||
{ key: "poster", icon: Palette, tone: "text-[#eba45d]" },
|
||||
{ key: "product", icon: LayoutGrid, tone: "text-[#53c59d]" },
|
||||
{ key: "portrait", icon: ImageIcon, tone: "text-[#a877e7]" },
|
||||
{ key: "edit", icon: MoreHorizontal, tone: "text-muted-foreground/65" },
|
||||
] as const;
|
||||
|
||||
interface PendingFirstMessage {
|
||||
content: string;
|
||||
images?: SendImage[];
|
||||
options?: SendOptions;
|
||||
}
|
||||
|
||||
export function ThreadShell({
|
||||
session,
|
||||
title,
|
||||
@ -67,10 +85,11 @@ export function ThreadShell({
|
||||
const chatId = session?.chatId ?? null;
|
||||
const historyKey = session?.key ?? null;
|
||||
const { messages: historical, loading, hasPendingToolCalls } = useSessionHistory(historyKey);
|
||||
const { client, modelName, token } = useClient();
|
||||
const { modelName, token } = useClient();
|
||||
const [booting, setBooting] = useState(false);
|
||||
const [slashCommands, setSlashCommands] = useState<SlashCommand[]>([]);
|
||||
const pendingFirstRef = useRef<string | null>(null);
|
||||
const [heroImageMode, setHeroImageMode] = useState(false);
|
||||
const pendingFirstRef = useRef<PendingFirstMessage | null>(null);
|
||||
const messageCacheRef = useRef<Map<string, UIMessage[]>>(new Map());
|
||||
const lastCachedChatIdRef = useRef<string | null>(null);
|
||||
|
||||
@ -82,6 +101,7 @@ export function ThreadShell({
|
||||
messages,
|
||||
isStreaming,
|
||||
send,
|
||||
stop,
|
||||
setMessages,
|
||||
streamError,
|
||||
dismissStreamError,
|
||||
@ -109,7 +129,11 @@ export function ThreadShell({
|
||||
// When the user switches away and back, keep the local in-memory thread
|
||||
// state (including not-yet-persisted messages) instead of replacing it with
|
||||
// whatever the history endpoint currently knows about.
|
||||
setMessages(cached && cached.length > 0 ? cached : historical);
|
||||
setMessages((prev) => {
|
||||
if (cached && cached.length > 0) return cached;
|
||||
if (historical.length === 0 && prev.length > 0) return prev;
|
||||
return historical;
|
||||
});
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [loading, chatId, historical]);
|
||||
|
||||
@ -142,18 +166,9 @@ export function ThreadShell({
|
||||
const pending = pendingFirstRef.current;
|
||||
if (!pending) return;
|
||||
pendingFirstRef.current = null;
|
||||
client.sendMessage(chatId, pending);
|
||||
setMessages((prev) => [
|
||||
...prev,
|
||||
{
|
||||
id: crypto.randomUUID(),
|
||||
role: "user",
|
||||
content: pending,
|
||||
createdAt: Date.now(),
|
||||
},
|
||||
]);
|
||||
send(pending.content, pending.images, pending.options);
|
||||
setBooting(false);
|
||||
}, [chatId, client, setMessages]);
|
||||
}, [chatId, send]);
|
||||
|
||||
useEffect(() => {
|
||||
let cancelled = false;
|
||||
@ -171,10 +186,10 @@ export function ThreadShell({
|
||||
}, [token]);
|
||||
|
||||
const handleWelcomeSend = useCallback(
|
||||
async (content: string) => {
|
||||
async (content: string, images?: SendImage[], options?: SendOptions) => {
|
||||
if (booting) return;
|
||||
setBooting(true);
|
||||
pendingFirstRef.current = content;
|
||||
pendingFirstRef.current = { content, images, options };
|
||||
const newId = await onCreateChat?.();
|
||||
if (!newId) {
|
||||
pendingFirstRef.current = null;
|
||||
@ -186,20 +201,27 @@ export function ThreadShell({
|
||||
|
||||
const handleQuickAction = useCallback(
|
||||
(prompt: string) => {
|
||||
const options: SendOptions | undefined = heroImageMode
|
||||
? { imageGeneration: { enabled: true, aspect_ratio: null } }
|
||||
: undefined;
|
||||
if (session) {
|
||||
send(prompt);
|
||||
send(prompt, undefined, options);
|
||||
return;
|
||||
}
|
||||
void handleWelcomeSend(prompt);
|
||||
void handleWelcomeSend(prompt, undefined, options);
|
||||
},
|
||||
[handleWelcomeSend, send, session],
|
||||
[handleWelcomeSend, heroImageMode, send, session],
|
||||
);
|
||||
|
||||
const quickActionItems = heroImageMode ? IMAGE_QUICK_ACTION_KEYS : QUICK_ACTION_KEYS;
|
||||
const quickActionPrefix = heroImageMode
|
||||
? "thread.empty.imageQuickActions"
|
||||
: "thread.empty.quickActions";
|
||||
const quickActions = (
|
||||
<div className="mx-auto grid w-full max-w-[58rem] grid-cols-2 gap-3 pt-4 sm:grid-cols-3 lg:grid-cols-6 lg:gap-4">
|
||||
{QUICK_ACTION_KEYS.map(({ key, icon: Icon, tone }) => {
|
||||
const title = t(`thread.empty.quickActions.${key}.title`);
|
||||
const prompt = t(`thread.empty.quickActions.${key}.prompt`);
|
||||
{quickActionItems.map(({ key, icon: Icon, tone }) => {
|
||||
const title = t(`${quickActionPrefix}.${key}.title`);
|
||||
const prompt = t(`${quickActionPrefix}.${key}.prompt`);
|
||||
return (
|
||||
<button
|
||||
key={key}
|
||||
@ -247,6 +269,9 @@ export function ThreadShell({
|
||||
modelLabel={toModelBadgeLabel(modelName)}
|
||||
variant={showHeroComposer ? "hero" : "thread"}
|
||||
slashCommands={slashCommands}
|
||||
imageMode={showHeroComposer ? heroImageMode : undefined}
|
||||
onImageModeChange={showHeroComposer ? setHeroImageMode : undefined}
|
||||
onStop={stop}
|
||||
/>
|
||||
) : (
|
||||
<ThreadComposer
|
||||
@ -260,6 +285,8 @@ export function ThreadShell({
|
||||
}
|
||||
modelLabel={toModelBadgeLabel(modelName)}
|
||||
variant="hero"
|
||||
imageMode={heroImageMode}
|
||||
onImageModeChange={setHeroImageMode}
|
||||
/>
|
||||
)}
|
||||
{showHeroComposer ? quickActions : null}
|
||||
|
||||
@ -5,6 +5,7 @@ import { toMediaAttachment } from "@/lib/media";
|
||||
import type { StreamError } from "@/lib/nanobot-client";
|
||||
import type {
|
||||
InboundEvent,
|
||||
OutboundImageGeneration,
|
||||
OutboundMedia,
|
||||
UIImage,
|
||||
UIMessage,
|
||||
@ -34,6 +35,10 @@ export interface SendImage {
|
||||
preview: UIImage;
|
||||
}
|
||||
|
||||
export interface SendOptions {
|
||||
imageGeneration?: OutboundImageGeneration;
|
||||
}
|
||||
|
||||
export function useNanobotStream(
|
||||
chatId: string | null,
|
||||
initialMessages: UIMessage[] = [],
|
||||
@ -42,7 +47,8 @@ export function useNanobotStream(
|
||||
): {
|
||||
messages: UIMessage[];
|
||||
isStreaming: boolean;
|
||||
send: (content: string, images?: SendImage[]) => void;
|
||||
send: (content: string, images?: SendImage[], options?: SendOptions) => void;
|
||||
stop: () => void;
|
||||
setMessages: React.Dispatch<React.SetStateAction<UIMessage[]>>;
|
||||
/** Latest transport-level fault raised since the last ``dismissStreamError``.
|
||||
* ``null`` when there is nothing to show. */
|
||||
@ -62,6 +68,7 @@ export function useNanobotStream(
|
||||
const [isStreaming, setIsStreaming] = useState(initialStreaming || hasPendingToolCalls);
|
||||
const [streamError, setStreamError] = useState<StreamError | null>(null);
|
||||
const buffer = useRef<StreamBuffer | null>(null);
|
||||
const suppressStreamUntilTurnEndRef = useRef(false);
|
||||
/** Timer that defers ``isStreaming = false`` after ``stream_end``.
|
||||
*
|
||||
* When the model finishes a text segment and calls a tool, the server
|
||||
@ -77,31 +84,29 @@ export function useNanobotStream(
|
||||
|
||||
const dismissStreamError = useCallback(() => setStreamError(null), []);
|
||||
|
||||
// Reset local state when switching chats. ``streamError`` is scoped to the
|
||||
// send that triggered it, so a chat swap should wipe it out: a stale
|
||||
// "Message too large" banner on a freshly-opened chat-B would confuse the
|
||||
// user about which send actually failed (and in which chat).
|
||||
useEffect(() => {
|
||||
setMessages(initialMessages);
|
||||
// Check if the new chat's last message is a trace row — if so, the
|
||||
// model may still be processing.
|
||||
setIsStreaming(
|
||||
initialMessages.length > 0
|
||||
? initialMessages[initialMessages.length - 1].kind === "trace"
|
||||
: false,
|
||||
);
|
||||
// Also consider hasPendingToolCalls from session history.
|
||||
if (hasPendingToolCalls) {
|
||||
setIsStreaming(true);
|
||||
}
|
||||
setStreamError(null);
|
||||
buffer.current = null;
|
||||
if (streamEndTimerRef.current !== null) {
|
||||
clearTimeout(streamEndTimerRef.current);
|
||||
streamEndTimerRef.current = null;
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [chatId, initialMessages, hasPendingToolCalls]);
|
||||
// Reset local state when switching chats. Do not reset on every
|
||||
// ``initialMessages`` update: a brand-new chat can receive an empty/404
|
||||
// history response after the optimistic first message has already rendered.
|
||||
useEffect(() => {
|
||||
setMessages(initialMessages);
|
||||
setIsStreaming(
|
||||
(initialMessages.length > 0
|
||||
? initialMessages[initialMessages.length - 1].kind === "trace"
|
||||
: false) || hasPendingToolCalls,
|
||||
);
|
||||
setStreamError(null);
|
||||
buffer.current = null;
|
||||
suppressStreamUntilTurnEndRef.current = false;
|
||||
if (streamEndTimerRef.current !== null) {
|
||||
clearTimeout(streamEndTimerRef.current);
|
||||
streamEndTimerRef.current = null;
|
||||
}
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [chatId]);
|
||||
|
||||
useEffect(() => {
|
||||
if (hasPendingToolCalls) setIsStreaming(true);
|
||||
}, [hasPendingToolCalls]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!chatId) return;
|
||||
@ -116,6 +121,7 @@ export function useNanobotStream(
|
||||
}
|
||||
|
||||
if (ev.event === "delta") {
|
||||
if (suppressStreamUntilTurnEndRef.current) return;
|
||||
const id = buffer.current?.messageId ?? crypto.randomUUID();
|
||||
if (!buffer.current) {
|
||||
buffer.current = { messageId: id, parts: [] };
|
||||
@ -141,6 +147,10 @@ export function useNanobotStream(
|
||||
}
|
||||
|
||||
if (ev.event === "stream_end") {
|
||||
if (suppressStreamUntilTurnEndRef.current) {
|
||||
buffer.current = null;
|
||||
return;
|
||||
}
|
||||
// stream_end only means the text segment finished — the model may
|
||||
// still be executing tools. Do NOT reset isStreaming here; the
|
||||
// definitive "turn is complete" signal is ``turn_end``.
|
||||
@ -160,6 +170,7 @@ export function useNanobotStream(
|
||||
setMessages((prev) =>
|
||||
prev.map((m) => (m.isStreaming ? { ...m, isStreaming: false } : m)),
|
||||
);
|
||||
suppressStreamUntilTurnEndRef.current = false;
|
||||
onTurnEnd?.();
|
||||
return;
|
||||
}
|
||||
@ -170,6 +181,12 @@ export function useNanobotStream(
|
||||
}
|
||||
|
||||
if (ev.event === "message") {
|
||||
if (
|
||||
suppressStreamUntilTurnEndRef.current &&
|
||||
(ev.kind === "tool_hint" || ev.kind === "progress")
|
||||
) {
|
||||
return;
|
||||
}
|
||||
// Intermediate agent breadcrumbs (tool-call hints, raw progress).
|
||||
// Attach them to the last trace row if it was the last emitted item
|
||||
// so a sequence of calls collapses into one compact trace group.
|
||||
@ -203,6 +220,7 @@ export function useNanobotStream(
|
||||
const media = ev.media_urls?.length
|
||||
? ev.media_urls.map((m) => toMediaAttachment(m))
|
||||
: ev.media?.map((url) => toMediaAttachment({ url }));
|
||||
const hasMedia = !!media && media.length > 0;
|
||||
|
||||
// A complete (non-streamed) assistant message. If a stream was in
|
||||
// flight, drop the placeholder so we don't render the text twice.
|
||||
@ -221,10 +239,13 @@ export function useNanobotStream(
|
||||
content,
|
||||
createdAt: Date.now(),
|
||||
...(ev.buttons && ev.buttons.length > 0 ? { buttons: ev.buttons } : {}),
|
||||
...(media && media.length > 0 ? { media } : {}),
|
||||
...(hasMedia ? { media } : {}),
|
||||
},
|
||||
];
|
||||
});
|
||||
if (hasMedia) {
|
||||
suppressStreamUntilTurnEndRef.current = true;
|
||||
}
|
||||
return;
|
||||
}
|
||||
// ``attached`` / ``error`` frames aren't actionable here; the client
|
||||
@ -243,7 +264,7 @@ export function useNanobotStream(
|
||||
}, [chatId, client, onTurnEnd]);
|
||||
|
||||
const send = useCallback(
|
||||
(content: string, images?: SendImage[]) => {
|
||||
(content: string, images?: SendImage[], options?: SendOptions) => {
|
||||
if (!chatId) return;
|
||||
const hasImages = !!images && images.length > 0;
|
||||
// Text is optional when images are attached — the agent will still see
|
||||
@ -265,15 +286,30 @@ export function useNanobotStream(
|
||||
// right away, before the first delta arrives from the server.
|
||||
setIsStreaming(true);
|
||||
const wireMedia = hasImages ? images!.map((i) => i.media) : undefined;
|
||||
client.sendMessage(chatId, content, wireMedia);
|
||||
if (options) {
|
||||
client.sendMessage(chatId, content, wireMedia, options);
|
||||
} else {
|
||||
client.sendMessage(chatId, content, wireMedia);
|
||||
}
|
||||
},
|
||||
[chatId, client],
|
||||
);
|
||||
|
||||
const stop = useCallback(() => {
|
||||
if (!chatId) return;
|
||||
setIsStreaming(false);
|
||||
setMessages((prev) =>
|
||||
prev.map((m) => (m.isStreaming ? { ...m, isStreaming: false } : m)),
|
||||
);
|
||||
suppressStreamUntilTurnEndRef.current = false;
|
||||
client.sendMessage(chatId, "/stop");
|
||||
}, [chatId, client]);
|
||||
|
||||
return {
|
||||
messages,
|
||||
isStreaming,
|
||||
send,
|
||||
stop,
|
||||
setMessages,
|
||||
streamError,
|
||||
dismissStreamError,
|
||||
|
||||
@ -21,6 +21,14 @@
|
||||
"logoutHint": "Disconnect this browser from the gateway.",
|
||||
"logout": "Sign out"
|
||||
},
|
||||
"system": {
|
||||
"section": "System",
|
||||
"restartHint": "Restart nanobot to apply runtime changes.",
|
||||
"restart": "Restart nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "Restart completed in {{seconds}}s."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -104,6 +112,32 @@
|
||||
"title": "More",
|
||||
"prompt": "Show me a few useful ways you can help in this workspace."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "Design an app icon",
|
||||
"prompt": "Generate a clean 1:1 app icon for nanobot: friendly robot, simple vector style, soft blue and white palette, no text."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "Make a sticker",
|
||||
"prompt": "Generate a cute sticker-style image of a tiny robot assistant, transparent-looking background, expressive and playful."
|
||||
},
|
||||
"poster": {
|
||||
"title": "Create a poster",
|
||||
"prompt": "Generate a polished poster concept for a personal AI assistant, modern composition, strong visual hierarchy, suitable for a landing page."
|
||||
},
|
||||
"product": {
|
||||
"title": "Product mockup",
|
||||
"prompt": "Generate a clean product mockup image for a conversational AI web app, minimal interface, premium lighting, realistic device frame."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "Stylized portrait",
|
||||
"prompt": "Generate a stylized portrait of a friendly AI companion, soft lighting, detailed but approachable, modern illustration style."
|
||||
},
|
||||
"edit": {
|
||||
"title": "Edit an image",
|
||||
"prompt": "Help me edit an image. Ask me to upload or reference the image first, then generate the edited result."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -120,7 +154,23 @@
|
||||
"inputAria": "Message input",
|
||||
"sendHint": "Enter to send · Shift+Enter for newline",
|
||||
"send": "Send message",
|
||||
"stop": "Stop response",
|
||||
"attachImage": "Attach image",
|
||||
"imageMode": {
|
||||
"label": "Image Generation",
|
||||
"toggle": "Toggle image generation mode",
|
||||
"placeholder": "Describe or edit an image…",
|
||||
"aspectAria": "Image aspect ratio",
|
||||
"aspectLabel": "Image aspect",
|
||||
"aspect": {
|
||||
"auto": "Auto",
|
||||
"1_1": "Square 1:1",
|
||||
"3_4": "Portrait 3:4",
|
||||
"9_16": "Story 9:16",
|
||||
"4_3": "Landscape 4:3",
|
||||
"16_9": "Wide 16:9"
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"search": "Search",
|
||||
"reason": "Reason",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "No se pudo conectar con nanobot",
|
||||
"gatewayHint": "Asegúrate de que la gateway esté en ejecución (`nanobot gateway`) y de que esta página esté abierta en la misma máquina."
|
||||
},
|
||||
"system": {
|
||||
"section": "Sistema",
|
||||
"restartHint": "Reinicia nanobot para aplicar los cambios de ejecución.",
|
||||
"restart": "Reiniciar nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "Reinicio completado en {{seconds}} s."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "Más",
|
||||
"prompt": "Muéstrame algunas formas útiles en las que puedes ayudar en este workspace."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "Diseñar un icono de app",
|
||||
"prompt": "Genera un icono de app 1:1 limpio para nanobot: robot amigable, estilo vectorial simple, paleta suave azul y blanca, sin texto."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "Crear un sticker",
|
||||
"prompt": "Genera una imagen estilo sticker de un pequeño asistente robot, con fondo de apariencia transparente, expresivo y divertido."
|
||||
},
|
||||
"poster": {
|
||||
"title": "Crear un póster",
|
||||
"prompt": "Genera un concepto de póster pulido para un asistente personal de IA, composición moderna, jerarquía visual fuerte, apto para una landing page."
|
||||
},
|
||||
"product": {
|
||||
"title": "Mockup de producto",
|
||||
"prompt": "Genera una imagen limpia de mockup de producto para una app web de IA conversacional, interfaz mínima, iluminación premium, marco de dispositivo realista."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "Retrato estilizado",
|
||||
"prompt": "Genera un retrato estilizado de un compañero de IA amigable, luz suave, detallado pero cercano, estilo de ilustración moderna."
|
||||
},
|
||||
"edit": {
|
||||
"title": "Editar una imagen",
|
||||
"prompt": "Ayúdame a editar una imagen. Primero pídeme que suba o indique la imagen, y luego genera el resultado editado."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "Entrada de mensaje",
|
||||
"sendHint": "Enter para enviar · Shift+Enter para nueva línea",
|
||||
"send": "Enviar mensaje",
|
||||
"stop": "Detener respuesta",
|
||||
"attachImage": "Adjuntar imagen",
|
||||
"imageMode": {
|
||||
"label": "Generar imagen",
|
||||
"toggle": "Activar o desactivar modo de generación de imágenes",
|
||||
"placeholder": "Describe o edita una imagen…",
|
||||
"aspectAria": "Relación de aspecto de imagen",
|
||||
"aspectLabel": "Formato de imagen",
|
||||
"aspect": {
|
||||
"auto": "Auto",
|
||||
"1_1": "Cuadrado 1:1",
|
||||
"3_4": "Vertical 3:4",
|
||||
"9_16": "Historia 9:16",
|
||||
"4_3": "Horizontal 4:3",
|
||||
"16_9": "Panorámico 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "Procesando…",
|
||||
"remove": "Quitar adjunto",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}} (auto)",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "Impossible de joindre nanobot",
|
||||
"gatewayHint": "Assurez-vous que la gateway est en cours d’exécution (`nanobot gateway`) et que cette page est ouverte sur la même machine."
|
||||
},
|
||||
"system": {
|
||||
"section": "Système",
|
||||
"restartHint": "Redémarrez nanobot pour appliquer les changements d’exécution.",
|
||||
"restart": "Redémarrer nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "Redémarrage terminé en {{seconds}} s."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "Plus",
|
||||
"prompt": "Montrez-moi quelques façons utiles dont vous pouvez m’aider dans cet espace de travail."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "Créer une icône d’app",
|
||||
"prompt": "Générez une icône d’application 1:1 propre pour nanobot : robot sympathique, style vectoriel simple, palette douce bleu et blanc, sans texte."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "Créer un sticker",
|
||||
"prompt": "Générez une image façon sticker d’un petit assistant robot, avec un fond d’apparence transparente, expressive et ludique."
|
||||
},
|
||||
"poster": {
|
||||
"title": "Créer une affiche",
|
||||
"prompt": "Générez un concept d’affiche soigné pour un assistant IA personnel, composition moderne, hiérarchie visuelle forte, adapté à une landing page."
|
||||
},
|
||||
"product": {
|
||||
"title": "Maquette produit",
|
||||
"prompt": "Générez une maquette produit propre pour une application web d’IA conversationnelle, interface minimale, éclairage premium, cadre d’appareil réaliste."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "Portrait stylisé",
|
||||
"prompt": "Générez un portrait stylisé d’un compagnon IA sympathique, lumière douce, détaillé mais accessible, style illustration moderne."
|
||||
},
|
||||
"edit": {
|
||||
"title": "Modifier une image",
|
||||
"prompt": "Aidez-moi à modifier une image. Demandez-moi d’abord de téléverser ou d’indiquer l’image, puis générez le résultat modifié."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "Champ de message",
|
||||
"sendHint": "Entrée pour envoyer · Maj+Entrée pour un retour à la ligne",
|
||||
"send": "Envoyer le message",
|
||||
"stop": "Arrêter la réponse",
|
||||
"attachImage": "Joindre une image",
|
||||
"imageMode": {
|
||||
"label": "Génération d’image",
|
||||
"toggle": "Activer ou désactiver le mode génération d’image",
|
||||
"placeholder": "Décrire ou modifier une image…",
|
||||
"aspectAria": "Format de l’image",
|
||||
"aspectLabel": "Format de l’image",
|
||||
"aspect": {
|
||||
"auto": "Auto",
|
||||
"1_1": "Carré 1:1",
|
||||
"3_4": "Portrait 3:4",
|
||||
"9_16": "Story 9:16",
|
||||
"4_3": "Paysage 4:3",
|
||||
"16_9": "Large 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "Traitement…",
|
||||
"remove": "Retirer la pièce jointe",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}} (auto)",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "Tidak dapat menjangkau nanobot",
|
||||
"gatewayHint": "Pastikan gateway sedang berjalan (`nanobot gateway`) dan halaman ini dibuka pada mesin yang sama."
|
||||
},
|
||||
"system": {
|
||||
"section": "Sistem",
|
||||
"restartHint": "Mulai ulang nanobot untuk menerapkan perubahan runtime.",
|
||||
"restart": "Mulai ulang nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "Mulai ulang selesai dalam {{seconds}} dtk."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "Lainnya",
|
||||
"prompt": "Tunjukkan beberapa cara berguna Anda dapat membantu di workspace ini."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "Desain ikon aplikasi",
|
||||
"prompt": "Buat ikon aplikasi 1:1 yang bersih untuk nanobot: robot ramah, gaya vektor sederhana, palet biru dan putih lembut, tanpa teks."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "Buat stiker",
|
||||
"prompt": "Buat gambar gaya stiker yang lucu dari asisten robot kecil, latar terlihat transparan, ekspresif dan menyenangkan."
|
||||
},
|
||||
"poster": {
|
||||
"title": "Buat poster",
|
||||
"prompt": "Buat konsep poster yang rapi untuk asisten AI pribadi, komposisi modern, hierarki visual kuat, cocok untuk landing page."
|
||||
},
|
||||
"product": {
|
||||
"title": "Mockup produk",
|
||||
"prompt": "Buat gambar mockup produk yang bersih untuk aplikasi web AI percakapan, antarmuka minimal, pencahayaan premium, bingkai perangkat realistis."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "Potret bergaya",
|
||||
"prompt": "Buat potret bergaya dari pendamping AI yang ramah, pencahayaan lembut, detail tetapi tetap mudah didekati, gaya ilustrasi modern."
|
||||
},
|
||||
"edit": {
|
||||
"title": "Edit gambar",
|
||||
"prompt": "Bantu saya mengedit gambar. Minta saya mengunggah atau menyebutkan gambar terlebih dahulu, lalu buat hasil editnya."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "Input pesan",
|
||||
"sendHint": "Enter untuk kirim · Shift+Enter untuk baris baru",
|
||||
"send": "Kirim pesan",
|
||||
"stop": "Hentikan respons",
|
||||
"attachImage": "Lampirkan gambar",
|
||||
"imageMode": {
|
||||
"label": "Buat gambar",
|
||||
"toggle": "Alihkan mode pembuatan gambar",
|
||||
"placeholder": "Deskripsikan atau edit gambar…",
|
||||
"aspectAria": "Rasio aspek gambar",
|
||||
"aspectLabel": "Rasio gambar",
|
||||
"aspect": {
|
||||
"auto": "Otomatis",
|
||||
"1_1": "Persegi 1:1",
|
||||
"3_4": "Potret 3:4",
|
||||
"9_16": "Story 9:16",
|
||||
"4_3": "Lanskap 4:3",
|
||||
"16_9": "Lebar 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "Memproses…",
|
||||
"remove": "Hapus lampiran",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}} (auto)",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "nanobot に接続できませんでした",
|
||||
"gatewayHint": "gateway（`nanobot gateway`）が起動しており、このページが同じマシン上で開かれていることを確認してください。"
|
||||
},
|
||||
"system": {
|
||||
"section": "システム",
|
||||
"restartHint": "実行時の変更を適用するには nanobot を再起動します。",
|
||||
"restart": "nanobot を再起動"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "{{seconds}} 秒で再起動が完了しました。"
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "その他",
|
||||
"prompt": "このワークスペースであなたが手伝える便利な方法をいくつか見せてください。"
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "アプリアイコンを作る",
|
||||
"prompt": "nanobot のクリーンな 1:1 アプリアイコンを生成してください。親しみやすいロボット、シンプルなベクター風、柔らかい青と白の配色、文字なし。"
|
||||
},
|
||||
"sticker": {
|
||||
"title": "ステッカーを作る",
|
||||
"prompt": "小さなロボットアシスタントのかわいいステッカー風画像を生成してください。透明風の背景で、表情豊かで遊び心のある雰囲気。"
|
||||
},
|
||||
"poster": {
|
||||
"title": "ポスターを作る",
|
||||
"prompt": "個人向け AI アシスタントの洗練されたポスター案を生成してください。モダンな構図、強い視覚階層、ランディングページ向け。"
|
||||
},
|
||||
"product": {
|
||||
"title": "製品モックアップ",
|
||||
"prompt": "会話型 AI Web アプリのクリーンな製品モックアップ画像を生成してください。ミニマルな UI、上質なライティング、リアルなデバイスフレーム。"
|
||||
},
|
||||
"portrait": {
|
||||
"title": "スタイル付きポートレート",
|
||||
"prompt": "親しみやすい AI コンパニオンのスタイル付きポートレートを生成してください。柔らかい光、細部は豊かで近づきやすい、モダンなイラスト風。"
|
||||
},
|
||||
"edit": {
|
||||
"title": "画像を編集",
|
||||
"prompt": "画像編集を手伝ってください。まず編集する画像のアップロードまたは指定を求め、その後に編集後の結果を生成してください。"
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "メッセージ入力欄",
|
||||
"sendHint": "Enter で送信 · Shift+Enter で改行",
|
||||
"send": "メッセージを送信",
|
||||
"stop": "応答を停止",
|
||||
"attachImage": "画像を添付",
|
||||
"imageMode": {
|
||||
"label": "画像生成",
|
||||
"toggle": "画像生成モードを切り替え",
|
||||
"placeholder": "画像を説明または編集…",
|
||||
"aspectAria": "画像のアスペクト比",
|
||||
"aspectLabel": "画像の比率",
|
||||
"aspect": {
|
||||
"auto": "自動",
|
||||
"1_1": "正方形 1:1",
|
||||
"3_4": "縦長 3:4",
|
||||
"9_16": "ストーリー 9:16",
|
||||
"4_3": "横長 4:3",
|
||||
"16_9": "ワイド 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "処理中…",
|
||||
"remove": "添付を削除",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}}（自動圧縮）",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "nanobot에 연결할 수 없습니다",
|
||||
"gatewayHint": "gateway(`nanobot gateway`)가 실행 중인지, 그리고 이 페이지가 같은 머신에서 열려 있는지 확인하세요."
|
||||
},
|
||||
"system": {
|
||||
"section": "시스템",
|
||||
"restartHint": "런타임 변경 사항을 적용하려면 nanobot을 다시 시작하세요.",
|
||||
"restart": "nanobot 다시 시작"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "{{seconds}}초 만에 다시 시작되었습니다."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "더 보기",
|
||||
"prompt": "이 워크스페이스에서 도와줄 수 있는 유용한 방법을 몇 가지 보여 주세요."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "앱 아이콘 디자인",
|
||||
"prompt": "nanobot을 위한 깔끔한 1:1 앱 아이콘을 생성해 주세요. 친근한 로봇, 단순한 벡터 스타일, 부드러운 파란색과 흰색 팔레트, 텍스트 없음."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "스티커 만들기",
|
||||
"prompt": "작은 로봇 도우미의 귀여운 스티커 스타일 이미지를 생성해 주세요. 투명해 보이는 배경, 표정이 풍부하고 장난스러운 느낌."
|
||||
},
|
||||
"poster": {
|
||||
"title": "포스터 만들기",
|
||||
"prompt": "개인 AI 도우미를 위한 세련된 포스터 콘셉트를 생성해 주세요. 현대적인 구성, 강한 시각적 계층, 랜딩 페이지에 어울리는 스타일."
|
||||
},
|
||||
"product": {
|
||||
"title": "제품 목업",
|
||||
"prompt": "대화형 AI 웹 앱을 위한 깔끔한 제품 목업 이미지를 생성해 주세요. 미니멀한 인터페이스, 고급스러운 조명, 현실적인 기기 프레임."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "스타일화된 초상화",
|
||||
"prompt": "친근한 AI 동반자의 스타일화된 초상화를 생성해 주세요. 부드러운 조명, 세밀하지만 다가가기 쉬운 분위기, 현대적인 일러스트 스타일."
|
||||
},
|
||||
"edit": {
|
||||
"title": "이미지 편집",
|
||||
"prompt": "이미지 편집을 도와주세요. 먼저 편집할 이미지를 업로드하거나 지정하게 한 뒤, 편집된 결과를 생성해 주세요."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "메시지 입력",
|
||||
"sendHint": "Enter로 전송 · Shift+Enter로 줄바꿈",
|
||||
"send": "메시지 보내기",
|
||||
"stop": "응답 중지",
|
||||
"attachImage": "이미지 첨부",
|
||||
"imageMode": {
|
||||
"label": "이미지 생성",
|
||||
"toggle": "이미지 생성 모드 전환",
|
||||
"placeholder": "이미지를 설명하거나 편집하세요…",
|
||||
"aspectAria": "이미지 화면 비율",
|
||||
"aspectLabel": "이미지 비율",
|
||||
"aspect": {
|
||||
"auto": "자동",
|
||||
"1_1": "정사각형 1:1",
|
||||
"3_4": "세로 3:4",
|
||||
"9_16": "스토리 9:16",
|
||||
"4_3": "가로 4:3",
|
||||
"16_9": "와이드 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "처리 중…",
|
||||
"remove": "첨부 제거",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}} (자동 압축)",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "Không thể kết nối tới nanobot",
|
||||
"gatewayHint": "Hãy chắc chắn gateway đang chạy (`nanobot gateway`) và trang này được mở trên cùng máy."
|
||||
},
|
||||
"system": {
|
||||
"section": "Hệ thống",
|
||||
"restartHint": "Khởi động lại nanobot để áp dụng thay đổi runtime.",
|
||||
"restart": "Khởi động lại nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "Khởi động lại hoàn tất sau {{seconds}} giây."
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "Thêm",
|
||||
"prompt": "Cho tôi xem vài cách hữu ích mà bạn có thể giúp trong workspace này."
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "Thiết kế biểu tượng app",
|
||||
"prompt": "Tạo một biểu tượng ứng dụng 1:1 gọn gàng cho nanobot: robot thân thiện, phong cách vector đơn giản, bảng màu xanh trắng dịu, không có chữ."
|
||||
},
|
||||
"sticker": {
|
||||
"title": "Tạo sticker",
|
||||
"prompt": "Tạo một hình kiểu sticker dễ thương của trợ lý robot nhỏ, nền trông như trong suốt, biểu cảm và vui nhộn."
|
||||
},
|
||||
"poster": {
|
||||
"title": "Tạo poster",
|
||||
"prompt": "Tạo một ý tưởng poster chỉn chu cho trợ lý AI cá nhân, bố cục hiện đại, phân cấp thị giác rõ, phù hợp cho landing page."
|
||||
},
|
||||
"product": {
|
||||
"title": "Mockup sản phẩm",
|
||||
"prompt": "Tạo một hình mockup sản phẩm gọn gàng cho ứng dụng web AI hội thoại, giao diện tối giản, ánh sáng cao cấp, khung thiết bị chân thực."
|
||||
},
|
||||
"portrait": {
|
||||
"title": "Chân dung cách điệu",
|
||||
"prompt": "Tạo chân dung cách điệu của một người bạn đồng hành AI thân thiện, ánh sáng mềm, chi tiết nhưng dễ gần, phong cách minh họa hiện đại."
|
||||
},
|
||||
"edit": {
|
||||
"title": "Chỉnh sửa ảnh",
|
||||
"prompt": "Giúp tôi chỉnh sửa một ảnh. Trước tiên hãy yêu cầu tôi tải lên hoặc chỉ định ảnh, rồi tạo kết quả đã chỉnh sửa."
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "Ô nhập tin nhắn",
|
||||
"sendHint": "Enter để gửi · Shift+Enter để xuống dòng",
|
||||
"send": "Gửi tin nhắn",
|
||||
"stop": "Dừng phản hồi",
|
||||
"attachImage": "Đính kèm ảnh",
|
||||
"imageMode": {
|
||||
"label": "Tạo ảnh",
|
||||
"toggle": "Bật/tắt chế độ tạo ảnh",
|
||||
"placeholder": "Mô tả hoặc chỉnh sửa ảnh…",
|
||||
"aspectAria": "Tỷ lệ khung hình ảnh",
|
||||
"aspectLabel": "Tỷ lệ ảnh",
|
||||
"aspect": {
|
||||
"auto": "Tự động",
|
||||
"1_1": "Vuông 1:1",
|
||||
"3_4": "Dọc 3:4",
|
||||
"9_16": "Story 9:16",
|
||||
"4_3": "Ngang 4:3",
|
||||
"16_9": "Rộng 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "Đang xử lý…",
|
||||
"remove": "Xóa tệp đính kèm",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}} (tự động)",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "无法连接到 nanobot",
|
||||
"gatewayHint": "请确认 gateway 已启动（`nanobot gateway`），并且当前页面与 gateway 运行在同一台机器上。"
|
||||
},
|
||||
"system": {
|
||||
"section": "系统",
|
||||
"restartHint": "重启 nanobot 以应用运行时更改。",
|
||||
"restart": "重启 nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "重启已完成，用时 {{seconds}} 秒。"
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -92,6 +100,32 @@
|
||||
"title": "更多",
|
||||
"prompt": "展示几个你在这个工作区里可以帮我的实用方式。"
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "设计应用图标",
|
||||
"prompt": "生成一个干净的 1:1 nanobot 应用图标：友好的机器人，简洁矢量风格，蓝白柔和配色，不要文字。"
|
||||
},
|
||||
"sticker": {
|
||||
"title": "制作贴纸",
|
||||
"prompt": "生成一张可爱的贴纸风小机器人助手图片，背景像透明贴纸，表情活泼有趣。"
|
||||
},
|
||||
"poster": {
|
||||
"title": "创建海报",
|
||||
"prompt": "生成一张个人 AI 助手的精致海报概念图，现代构图，视觉层级清晰，适合落地页展示。"
|
||||
},
|
||||
"product": {
|
||||
"title": "产品样机",
|
||||
"prompt": "生成一张对话式 AI Web 应用的干净产品样机图，极简界面，高级光影，真实设备边框。"
|
||||
},
|
||||
"portrait": {
|
||||
"title": "风格化头像",
|
||||
"prompt": "生成一个友好的 AI 伙伴风格化头像，柔和光线，细节丰富但亲切，现代插画风格。"
|
||||
},
|
||||
"edit": {
|
||||
"title": "编辑图片",
|
||||
"prompt": "帮我编辑一张图片。先让我上传或指定要编辑的图片，然后生成编辑后的结果。"
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -108,7 +142,23 @@
|
||||
"inputAria": "消息输入框",
|
||||
"sendHint": "Enter 发送 · Shift+Enter 换行",
|
||||
"send": "发送消息",
|
||||
"stop": "停止响应",
|
||||
"attachImage": "添加图片",
|
||||
"imageMode": {
|
||||
"label": "图片生成",
|
||||
"toggle": "切换图片生成模式",
|
||||
"placeholder": "描述或编辑图片…",
|
||||
"aspectAria": "图片画幅",
|
||||
"aspectLabel": "图片画幅",
|
||||
"aspect": {
|
||||
"auto": "自动",
|
||||
"1_1": "方形 1:1",
|
||||
"3_4": "竖版 3:4",
|
||||
"9_16": "故事版 9:16",
|
||||
"4_3": "横版 4:3",
|
||||
"16_9": "宽屏 16:9"
|
||||
}
|
||||
},
|
||||
"tools": {
|
||||
"search": "搜索",
|
||||
"reason": "推理",
|
||||
|
||||
@ -9,6 +9,14 @@
|
||||
"title": "無法連線到 nanobot",
|
||||
"gatewayHint": "請確認 gateway 已啟動（`nanobot gateway`），並且目前頁面與 gateway 在同一台機器上開啟。"
|
||||
},
|
||||
"system": {
|
||||
"section": "系統",
|
||||
"restartHint": "重新啟動 nanobot 以套用執行階段變更。",
|
||||
"restart": "重新啟動 nanobot"
|
||||
},
|
||||
"restart": {
|
||||
"completed": "重新啟動已完成，耗時 {{seconds}} 秒。"
|
||||
},
|
||||
"documentTitle": {
|
||||
"base": "nanobot",
|
||||
"chat": "{{title}} · nanobot"
|
||||
@ -80,6 +88,32 @@
|
||||
"title": "更多",
|
||||
"prompt": "展示幾個你在這個工作區裡可以幫我的實用方式。"
|
||||
}
|
||||
},
|
||||
"imageQuickActions": {
|
||||
"icon": {
|
||||
"title": "設計應用程式圖示",
|
||||
"prompt": "生成一個乾淨的 1:1 nanobot 應用程式圖示：友善的機器人、簡潔向量風格、柔和藍白配色，不要文字。"
|
||||
},
|
||||
"sticker": {
|
||||
"title": "製作貼圖",
|
||||
"prompt": "生成一張可愛貼圖風格的小型機器人助理圖片，背景像透明貼紙，表情活潑有趣。"
|
||||
},
|
||||
"poster": {
|
||||
"title": "建立海報",
|
||||
"prompt": "生成一張個人 AI 助理的精緻海報概念圖，現代構圖、清楚的視覺層級，適合登陸頁展示。"
|
||||
},
|
||||
"product": {
|
||||
"title": "產品樣機",
|
||||
"prompt": "生成一張對話式 AI Web 應用的乾淨產品樣機圖，極簡介面、高級光影、真實裝置邊框。"
|
||||
},
|
||||
"portrait": {
|
||||
"title": "風格化頭像",
|
||||
"prompt": "生成一個友善 AI 夥伴的風格化頭像，柔和光線、細節豐富但親切，現代插畫風格。"
|
||||
},
|
||||
"edit": {
|
||||
"title": "編輯圖片",
|
||||
"prompt": "幫我編輯一張圖片。先請我上傳或指定要編輯的圖片，然後生成編輯後的結果。"
|
||||
}
|
||||
}
|
||||
},
|
||||
"header": {
|
||||
@ -93,7 +127,23 @@
|
||||
"inputAria": "訊息輸入框",
|
||||
"sendHint": "Enter 送出 · Shift+Enter 換行",
|
||||
"send": "送出訊息",
|
||||
"stop": "停止回覆",
|
||||
"attachImage": "附加圖片",
|
||||
"imageMode": {
|
||||
"label": "圖片生成",
|
||||
"toggle": "切換圖片生成模式",
|
||||
"placeholder": "描述或編輯圖片…",
|
||||
"aspectAria": "圖片畫幅",
|
||||
"aspectLabel": "圖片畫幅",
|
||||
"aspect": {
|
||||
"auto": "自動",
|
||||
"1_1": "方形 1:1",
|
||||
"3_4": "直式 3:4",
|
||||
"9_16": "故事版 9:16",
|
||||
"4_3": "橫式 4:3",
|
||||
"16_9": "寬螢幕 16:9"
|
||||
}
|
||||
},
|
||||
"encoding": "處理中…",
|
||||
"remove": "移除附件",
|
||||
"normalizedSizeHint": "{{orig}} → {{current}}（已自動壓縮）",
|
||||
|
||||
@ -126,13 +126,15 @@ export async function listSlashCommands(
|
||||
arg_hint?: string;
|
||||
};
|
||||
const body = await request<{ commands: Row[] }>(`${base}/api/commands`, token);
|
||||
return body.commands.map((command) => ({
|
||||
command: command.command,
|
||||
title: command.title,
|
||||
description: command.description,
|
||||
icon: command.icon,
|
||||
argHint: command.arg_hint ?? "",
|
||||
}));
|
||||
return body.commands
|
||||
.filter((command) => !["/stop", "/restart"].includes(command.command))
|
||||
.map((command) => ({
|
||||
command: command.command,
|
||||
title: command.title,
|
||||
description: command.description,
|
||||
icon: command.icon,
|
||||
argHint: command.arg_hint ?? "",
|
||||
}));
|
||||
}
|
||||
|
||||
export async function updateSettings(
|
||||
|
||||
@ -2,6 +2,7 @@ import type {
|
||||
ConnectionStatus,
|
||||
InboundEvent,
|
||||
Outbound,
|
||||
OutboundImageGeneration,
|
||||
OutboundMedia,
|
||||
} from "./types";
|
||||
|
||||
@ -181,12 +182,21 @@ export class NanobotClient {
|
||||
}
|
||||
}
|
||||
|
||||
sendMessage(chatId: string, content: string, media?: OutboundMedia[]): void {
|
||||
sendMessage(
|
||||
chatId: string,
|
||||
content: string,
|
||||
media?: OutboundMedia[],
|
||||
options?: { imageGeneration?: OutboundImageGeneration },
|
||||
): void {
|
||||
this.knownChats.add(chatId);
|
||||
const frame: Outbound =
|
||||
media && media.length > 0
|
||||
? { type: "message", chat_id: chatId, content, media, webui: true }
|
||||
: { type: "message", chat_id: chatId, content, webui: true };
|
||||
const frame: Outbound = {
|
||||
type: "message",
|
||||
chat_id: chatId,
|
||||
content,
|
||||
...(media && media.length > 0 ? { media } : {}),
|
||||
...(options?.imageGeneration ? { image_generation: options.imageGeneration } : {}),
|
||||
webui: true,
|
||||
};
|
||||
this.queueSend(frame);
|
||||
}
|
||||
|
||||
|
||||
@ -150,6 +150,11 @@ export interface OutboundMedia {
|
||||
name?: string;
|
||||
}
|
||||
|
||||
export interface OutboundImageGeneration {
|
||||
enabled: true;
|
||||
aspect_ratio?: string | null;
|
||||
}
|
||||
|
||||
export type Outbound =
|
||||
| { type: "new_chat" }
|
||||
| { type: "attach"; chat_id: string }
|
||||
@ -158,6 +163,7 @@ export type Outbound =
|
||||
chat_id: string;
|
||||
content: string;
|
||||
media?: OutboundMedia[];
|
||||
image_generation?: OutboundImageGeneration;
|
||||
/** Marks messages sent by the embedded WebUI, without changing the
|
||||
* generic websocket protocol for other clients. */
|
||||
webui?: true;
|
||||
|
||||
@ -84,6 +84,18 @@ describe("webui API helpers", () => {
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
commands: [
|
||||
{
|
||||
command: "/stop",
|
||||
title: "Stop current task",
|
||||
description: "Cancel the active task.",
|
||||
icon: "square",
|
||||
},
|
||||
{
|
||||
command: "/restart",
|
||||
title: "Restart nanobot",
|
||||
description: "Restart the bot process.",
|
||||
icon: "rotate-cw",
|
||||
},
|
||||
{
|
||||
command: "/history",
|
||||
title: "Show conversation history",
|
||||
|
||||
@ -7,6 +7,7 @@ import { ThreadComposer } from "@/components/thread/ThreadComposer";
|
||||
import { resources } from "@/i18n";
|
||||
|
||||
const QUICK_ACTION_KEYS = ["plan", "analyze", "brainstorm", "code", "summarize", "more"];
|
||||
const IMAGE_QUICK_ACTION_KEYS = ["icon", "sticker", "poster", "product", "portrait", "edit"];
|
||||
|
||||
describe("webui i18n", () => {
|
||||
it("switches UI copy and document locale through the language switcher", async () => {
|
||||
@ -54,6 +55,11 @@ describe("webui i18n", () => {
|
||||
expect(action.title).toBeTruthy();
|
||||
expect(action.prompt).toBeTruthy();
|
||||
}
|
||||
for (const key of IMAGE_QUICK_ACTION_KEYS) {
|
||||
const action = empty.imageQuickActions[key as keyof typeof empty.imageQuickActions];
|
||||
expect(action.title).toBeTruthy();
|
||||
expect(action.prompt).toBeTruthy();
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@ -102,4 +102,26 @@ describe("MessageBubble", () => {
|
||||
expect(video).toHaveAttribute("src", "/api/media/sig/payload");
|
||||
expect(container.querySelector("video[controls]")).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("renders assistant image media as a larger generated result", () => {
|
||||
const message: UIMessage = {
|
||||
id: "a-image",
|
||||
role: "assistant",
|
||||
content: "done",
|
||||
createdAt: Date.now(),
|
||||
media: [
|
||||
{
|
||||
kind: "image",
|
||||
url: "/api/media/sig/image",
|
||||
name: "generated.png",
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const { container } = render(<MessageBubble message={message} />);
|
||||
|
||||
const imageButton = screen.getByRole("button", { name: /view image/i });
|
||||
expect(imageButton).toHaveClass("h-56", "sm:h-72");
|
||||
expect(container.querySelector("img")).toHaveClass("object-contain");
|
||||
});
|
||||
});
|
||||
|
||||
@ -120,6 +120,33 @@ describe("NanobotClient", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("includes image generation options in outbound messages", () => {
|
||||
const client = new NanobotClient({
|
||||
url: "ws://test",
|
||||
reconnect: false,
|
||||
socketFactory: (url) => new FakeSocket(url) as unknown as WebSocket,
|
||||
});
|
||||
client.connect();
|
||||
lastSocket().fakeOpen();
|
||||
|
||||
client.sendMessage(
|
||||
"chat-img",
|
||||
"draw a banner",
|
||||
undefined,
|
||||
{ imageGeneration: { enabled: true, aspect_ratio: "16:9" } },
|
||||
);
|
||||
|
||||
expect(lastSocket().sent).toContain(
|
||||
JSON.stringify({
|
||||
type: "message",
|
||||
chat_id: "chat-img",
|
||||
content: "draw a banner",
|
||||
image_generation: { enabled: true, aspect_ratio: "16:9" },
|
||||
webui: true,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("re-attaches known chats after a reconnect", async () => {
|
||||
const client = new NanobotClient({
|
||||
url: "ws://test",
|
||||
|
||||
@ -91,4 +91,116 @@ describe("ThreadComposer", () => {
|
||||
expect(onSend).not.toHaveBeenCalled();
|
||||
expect(screen.queryByRole("listbox", { name: "Slash commands" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("sends image generation mode with automatic aspect ratio", () => {
|
||||
const onSend = vi.fn();
|
||||
render(
|
||||
<ThreadComposer
|
||||
onSend={onSend}
|
||||
placeholder="Type your message..."
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Toggle image generation mode" }));
|
||||
expect(screen.getByPlaceholderText("Describe or edit an image…")).toBeInTheDocument();
|
||||
|
||||
const input = screen.getByLabelText("Message input");
|
||||
fireEvent.change(input, { target: { value: "Draw a friendly robot" } });
|
||||
fireEvent.click(screen.getByRole("button", { name: "Send message" }));
|
||||
|
||||
expect(onSend).toHaveBeenCalledWith(
|
||||
"Draw a friendly robot",
|
||||
undefined,
|
||||
{ imageGeneration: { enabled: true, aspect_ratio: null } },
|
||||
);
|
||||
});
|
||||
|
||||
it("shows a stop button while streaming", () => {
|
||||
const onStop = vi.fn();
|
||||
render(
|
||||
<ThreadComposer
|
||||
onSend={vi.fn()}
|
||||
onStop={onStop}
|
||||
isStreaming
|
||||
placeholder="Type your message..."
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Stop response" }));
|
||||
|
||||
expect(onStop).toHaveBeenCalledTimes(1);
|
||||
expect(screen.queryByRole("button", { name: "Send message" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("lets users select a concrete image aspect ratio", () => {
|
||||
const onSend = vi.fn();
|
||||
render(
|
||||
<ThreadComposer
|
||||
onSend={onSend}
|
||||
placeholder="Type your message..."
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Toggle image generation mode" }));
|
||||
fireEvent.click(screen.getByRole("button", { name: "Image aspect ratio" }));
|
||||
expect(screen.getByRole("listbox", { name: "Image aspect ratio" }).className).toContain(
|
||||
"bottom-full",
|
||||
);
|
||||
fireEvent.mouseDown(screen.getByRole("option", { name: "Wide 16:9" }));
|
||||
|
||||
const input = screen.getByLabelText("Message input");
|
||||
fireEvent.change(input, { target: { value: "Draw a banner" } });
|
||||
fireEvent.click(screen.getByRole("button", { name: "Send message" }));
|
||||
|
||||
expect(onSend).toHaveBeenCalledWith(
|
||||
"Draw a banner",
|
||||
undefined,
|
||||
{ imageGeneration: { enabled: true, aspect_ratio: "16:9" } },
|
||||
);
|
||||
});
|
||||
|
||||
it("opens the hero image aspect menu downward", () => {
|
||||
render(
|
||||
<ThreadComposer
|
||||
onSend={vi.fn()}
|
||||
placeholder="Ask anything..."
|
||||
variant="hero"
|
||||
imageMode
|
||||
/>,
|
||||
);
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Image aspect ratio" }));
|
||||
|
||||
expect(screen.getByRole("listbox", { name: "Image aspect ratio" }).className).toContain(
|
||||
"top-full",
|
||||
);
|
||||
});
|
||||
|
||||
it("dismisses the image aspect menu on outside click, escape, and wheel", () => {
|
||||
render(
|
||||
<div>
|
||||
<button type="button">outside</button>
|
||||
<ThreadComposer
|
||||
onSend={vi.fn()}
|
||||
placeholder="Type your message..."
|
||||
imageMode
|
||||
/>
|
||||
</div>,
|
||||
);
|
||||
|
||||
const aspectButton = screen.getByRole("button", { name: "Image aspect ratio" });
|
||||
fireEvent.click(aspectButton);
|
||||
expect(screen.getByRole("listbox", { name: "Image aspect ratio" })).toBeInTheDocument();
|
||||
|
||||
fireEvent.pointerDown(screen.getByRole("button", { name: "outside" }));
|
||||
expect(screen.queryByRole("listbox", { name: "Image aspect ratio" })).not.toBeInTheDocument();
|
||||
|
||||
fireEvent.click(aspectButton);
|
||||
fireEvent.keyDown(document, { key: "Escape" });
|
||||
expect(screen.queryByRole("listbox", { name: "Image aspect ratio" })).not.toBeInTheDocument();
|
||||
|
||||
fireEvent.click(aspectButton);
|
||||
fireEvent.wheel(screen.getByRole("listbox", { name: "Image aspect ratio" }), { deltaY: 120 });
|
||||
expect(screen.queryByRole("listbox", { name: "Image aspect ratio" })).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
@ -250,6 +250,64 @@ describe("ThreadShell", () => {
|
||||
expect(onNewChat).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps the first landing message when new chat history is still empty", async () => {
|
||||
const client = makeClient();
|
||||
const onCreateChat = vi.fn().mockResolvedValue("chat-new");
|
||||
vi.stubGlobal(
|
||||
"fetch",
|
||||
vi.fn(async () => ({
|
||||
ok: false,
|
||||
status: 404,
|
||||
json: async () => ({}),
|
||||
})),
|
||||
);
|
||||
|
||||
const { rerender } = render(
|
||||
wrap(
|
||||
client,
|
||||
<ThreadShell
|
||||
session={null}
|
||||
title="nanobot"
|
||||
onToggleSidebar={() => {}}
|
||||
onCreateChat={onCreateChat}
|
||||
/>,
|
||||
),
|
||||
);
|
||||
|
||||
fireEvent.change(screen.getByLabelText("Message input"), {
|
||||
target: { value: "first message should stay" },
|
||||
});
|
||||
fireEvent.click(screen.getByRole("button", { name: "Send message" }));
|
||||
|
||||
await waitFor(() => expect(onCreateChat).toHaveBeenCalledTimes(1));
|
||||
|
||||
await act(async () => {
|
||||
rerender(
|
||||
wrap(
|
||||
client,
|
||||
<ThreadShell
|
||||
session={session("chat-new")}
|
||||
title="Chat chat-new"
|
||||
onToggleSidebar={() => {}}
|
||||
onCreateChat={onCreateChat}
|
||||
/>,
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
await waitFor(() =>
|
||||
expect(client.sendMessage).toHaveBeenCalledWith(
|
||||
"chat-new",
|
||||
"first message should stay",
|
||||
undefined,
|
||||
),
|
||||
);
|
||||
await waitFor(() =>
|
||||
expect(screen.getByText("first message should stay")).toBeInTheDocument(),
|
||||
);
|
||||
expect(screen.queryByText("What can I do for you?")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("sends quick action prompts from the empty thread landing", async () => {
|
||||
const client = makeClient();
|
||||
const onNewChat = vi.fn().mockResolvedValue("chat-a");
|
||||
@ -566,6 +624,30 @@ describe("ThreadShell", () => {
|
||||
expect(screen.queryByRole("listbox", { name: "Slash commands" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("switches welcome quick actions when image mode is enabled", async () => {
|
||||
const client = makeClient();
|
||||
render(
|
||||
wrap(
|
||||
client,
|
||||
<ThreadShell
|
||||
session={null}
|
||||
title="nanobot"
|
||||
onToggleSidebar={() => {}}
|
||||
onNewChat={() => {}}
|
||||
/>,
|
||||
),
|
||||
);
|
||||
await act(async () => {});
|
||||
|
||||
expect(screen.getByText("Write code")).toBeInTheDocument();
|
||||
expect(screen.queryByText("Design an app icon")).not.toBeInTheDocument();
|
||||
|
||||
fireEvent.click(screen.getByRole("button", { name: "Toggle image generation mode" }));
|
||||
|
||||
expect(screen.getByText("Design an app icon")).toBeInTheDocument();
|
||||
expect(screen.queryByText("Write code")).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("surfaces a dismissible banner when the stream reports message_too_big", async () => {
|
||||
const client = makeClient();
|
||||
const onNewChat = vi.fn().mockResolvedValue("chat-a");
|
||||
|
||||
@ -134,6 +134,89 @@ describe("useNanobotStream", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it("suppresses redundant stream confirmation after assistant media", () => {
|
||||
const fake = fakeClient();
|
||||
const { result } = renderHook(() => useNanobotStream("chat-img-result", EMPTY_MESSAGES), {
|
||||
wrapper: wrap(fake.client),
|
||||
});
|
||||
|
||||
act(() => {
|
||||
fake.emit("chat-img-result", {
|
||||
event: "message",
|
||||
chat_id: "chat-img-result",
|
||||
text: "image ready",
|
||||
media_urls: [{ url: "/api/media/sig/image", name: "generated.png" }],
|
||||
});
|
||||
fake.emit("chat-img-result", {
|
||||
event: "message",
|
||||
chat_id: "chat-img-result",
|
||||
text: "message()",
|
||||
kind: "tool_hint",
|
||||
});
|
||||
fake.emit("chat-img-result", {
|
||||
event: "delta",
|
||||
chat_id: "chat-img-result",
|
||||
text: "发送成功",
|
||||
});
|
||||
fake.emit("chat-img-result", {
|
||||
event: "stream_end",
|
||||
chat_id: "chat-img-result",
|
||||
});
|
||||
fake.emit("chat-img-result", {
|
||||
event: "turn_end",
|
||||
chat_id: "chat-img-result",
|
||||
});
|
||||
});
|
||||
|
||||
expect(result.current.messages).toHaveLength(1);
|
||||
expect(result.current.messages[0].content).toBe("image ready");
|
||||
expect(result.current.messages[0].media).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("passes image generation options to the websocket client", () => {
|
||||
const fake = fakeClient();
|
||||
const { result } = renderHook(() => useNanobotStream("chat-img", EMPTY_MESSAGES), {
|
||||
wrapper: wrap(fake.client),
|
||||
});
|
||||
|
||||
act(() => {
|
||||
result.current.send(
|
||||
"draw a square icon",
|
||||
undefined,
|
||||
{ imageGeneration: { enabled: true, aspect_ratio: "1:1" } },
|
||||
);
|
||||
});
|
||||
|
||||
expect(fake.client.sendMessage).toHaveBeenCalledWith(
|
||||
"chat-img",
|
||||
"draw a square icon",
|
||||
undefined,
|
||||
{ imageGeneration: { enabled: true, aspect_ratio: "1:1" } },
|
||||
);
|
||||
});
|
||||
|
||||
it("stops the active turn without adding a user slash command bubble", () => {
|
||||
const fake = fakeClient();
|
||||
const { result } = renderHook(() => useNanobotStream("chat-stop", EMPTY_MESSAGES), {
|
||||
wrapper: wrap(fake.client),
|
||||
});
|
||||
|
||||
act(() => {
|
||||
result.current.send("long task");
|
||||
});
|
||||
expect(result.current.messages).toHaveLength(1);
|
||||
expect(result.current.isStreaming).toBe(true);
|
||||
|
||||
act(() => {
|
||||
result.current.stop();
|
||||
});
|
||||
|
||||
expect(fake.client.sendMessage).toHaveBeenLastCalledWith("chat-stop", "/stop");
|
||||
expect(result.current.isStreaming).toBe(false);
|
||||
expect(result.current.messages).toHaveLength(1);
|
||||
expect(result.current.messages[0].content).toBe("long task");
|
||||
});
|
||||
|
||||
it("keeps assistant buttons on complete messages", () => {
|
||||
const fake = fakeClient();
|
||||
const { result } = renderHook(() => useNanobotStream("chat-q", EMPTY_MESSAGES), {
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user