fix: multi-bug sprint across settings, insights, energy, jobs, diagnostics

specs/001-project-aether/features/41-guided-workflow-agent/plan.md

@@ -0,0 +1,153 @@
+# Implementation Plan: Guided Workflow Agent
+
+**Feature**: [spec.md](./spec.md)
+**Status**: Planned
+**Date**: 2026-03-12
+
+## Summary
+
+Wire the existing Feature 36 automation builder workflow into the system as a dedicated, guided agent. This involves three layers: (1) a backend API route and chat preset, (2) orchestrator intent routing, and (3) conversational streaming with step progress in the activity panel.
+
+## Technical Context
+
+**Existing infrastructure (Feature 36)**:
+- `src/graph/workflows/automation_builder.py` — LangGraph workflow with 6 nodes, HITL interrupt at preview
+- `src/graph/nodes/automation_builder.py` — Node implementations (gather_intent, validate_entities, check_duplicates, generate_yaml, validate_yaml, preview)
+- `src/graph/state/automation_builder.py` — `AutomationBuilderState` typed dict
+- `src/tools/automation_builder_tools.py` — check_entity_exists, find_similar_automations, validate_automation_draft
+- Registered in `src/graph/workflows/_registry.py` as `automation_builder`
+
+**What's missing**:
+- No API endpoint invokes the workflow
+- No chat preset for "Create Automation"
+- Orchestrator doesn't route automation intents to the builder
+- No streaming/activity integration
+- `preview_node` generates a `proposal_id` but doesn't actually call `seek_approval`
+
+## Constitution Check
+
+- **Safety First**: Automations go through HITL approval (interrupt_before=["preview"]) and the proposal pipeline. No deployment without user consent.
+- **Isolation**: YAML generation only; no script execution.
+- **Observability**: All nodes wrapped in `traced_node()`, MLflow spans.
+- **State**: PostgreSQL checkpointing via `PostgresCheckpointer`.
+
+## Architecture
+
+### Two entry points
+
+```
+Entry 1: Chat preset "Create Automation"
+    |
+    POST /v1/chat/completions
+    |-- agent="automation_builder"
+    |-- Bypasses orchestrator, goes directly to builder
+    v
+    AutomationBuilderWorkflow.stream()
+
+Entry 2: General chat with automation intent
+    |
+    POST /v1/chat/completions
+    |-- agent="auto" (default)
+    |-- Orchestrator.classify_intent()
+    |-- intent = "create_automation"
+    |-- Routes to AutomationBuilderWorkflow
+    v
+    AutomationBuilderWorkflow.stream()
+```
+
+### Streaming flow
+
+```
+User message
+    |
+    v
+gather_intent_node
+    |-- emit_job_status("Extracting automation intent...")
+    |-- If needs_clarification: stream question back, wait for user reply
+    v
+validate_entities_node
+    |-- emit_job_status("Validating entities...")
+    |-- If errors: stream suggestions, loop to gather_intent
+    v
+check_duplicates_node
+    |-- emit_job_status("Checking for duplicates...")
+    |-- If found: stream warning, ask user to confirm
+    v
+generate_yaml_node
+    |-- emit_job_status("Generating automation YAML...")
+    v
+validate_yaml_node
+    |-- emit_job_status("Validating YAML...")
+    |-- If errors (< 3 attempts): loop to generate
+    v
+preview_node
+    |-- emit_job_status("Preparing preview...")
+    |-- Stream YAML preview to user
+    |-- Call seek_approval() to create proposal
+    |-- Stream proposal link
+    v
+END
+```
+
+## Key Design Decisions
+
+1. **Wrap workflow as an agent**: Create `AutomationBuilderAgent` that wraps the compiled LangGraph graph with streaming, job events, and message formatting. This parallels how `ArchitectWorkflow` wraps the conversation graph.
+
+2. **Conversational multi-turn**: The HITL interrupt at `preview` pauses the graph. For clarification loops (needs_clarification, entity_errors), the gather_intent node already loops. We enhance it to stream the question as an SSE message and wait for the user's next message (resume from checkpoint).
+
+3. **Orchestrator routing**: Add `"create_automation"` as an intent in `OrchestratorAgent.classify_intent()`. When detected, set `target_agent = "automation_builder"` in the `TaskPlan`.
+
+4. **Proposal creation**: `preview_node` currently only generates a message. Enhance it to call `seek_approval()` to create an actual `AutomationProposal` in the database.
+
+5. **Reuse streaming infrastructure**: Use the same SSE streaming as the conversation workflow (`/v1/chat/completions` with `stream=true`).
+
+## How Users Can Create Automations
+
+### Method 1: Guided Agent (via preset)
+
+1. Open chat, select "Create Automation" preset from the workflow picker
+2. Describe the automation: "Turn off all lights at 11pm"
+3. Agent validates entities, checks for duplicates
+4. Agent shows YAML preview
+5. User approves -> proposal created
+6. Navigate to Proposals page to deploy
+
+### Method 2: Natural Language in Chat
+
+1. Open chat (general mode or any preset)
+2. Say: "Create an automation that turns off the lights when nobody is home"
+3. Orchestrator routes to automation builder automatically
+4. Same guided flow as Method 1
+
+### Method 3: Manual via Architect
+
+1. Open chat, select any preset
+2. Tell the Architect exactly what you want:
+   - "Use seek_approval to create an automation with this YAML: ..."
+   - "Create a proposal for a time-triggered automation that..."
+3. Architect generates YAML inline and uses `seek_approval` to create a proposal
+4. This path is free-form — no step-by-step guidance
+
+### Method 4: Registry Page
+
+1. Go to the Registry page, Automations tab
+2. Use the "Ask Architect" inline assistant
+3. Say: "Create a new automation that locks the door when I leave"
+4. Architect generates and proposes via `seek_approval`
+5. Navigate to Proposals page to review and deploy
+
+## Files to Create
+
+- `src/agents/automation_builder/agent.py` — `AutomationBuilderAgent` wrapper (stream, resume, job events)
+- `src/agents/automation_builder/__init__.py` — Module init
+- `tests/unit/test_automation_builder_agent.py` — Agent wrapper tests
+- `tests/unit/test_orchestrator_routing.py` — Intent routing tests
+
+## Files to Modify
+
+- `src/graph/nodes/automation_builder.py` — Enhance `preview_node` to call `seek_approval`; add step status emissions
+- `src/agents/orchestrator.py` — Add `create_automation` intent, route to builder
+- `src/api/routes/openai_compat/handlers.py` — Handle `agent="automation_builder"` in stream handler
+- `src/api/routes/workflows.py` — Add "Create Automation" preset
+- `ui/src/api/client/conversations.ts` — No changes (uses existing stream API)
+- `ui/src/pages/chat/WorkflowPresetSelector.tsx` — Picks up new preset from API automatically

specs/001-project-aether/features/41-guided-workflow-agent/spec.md

@@ -0,0 +1,83 @@
+# Feature Specification: Guided Workflow Agent
+
+**Feature Branch**: `feat/41-guided-workflow-agent`
+**Created**: 2026-03-12
+**Status**: Draft
+**Depends on**: Feature 36 (NL Automation Builder — state, nodes, tools exist but are not exposed)
+**Input**: User request: "I want the workflows to be guided with a dedicated agent."
+
+## Problem Statement
+
+Feature 36 implemented the automation builder workflow (LangGraph graph, nodes, tools, state) but it was never wired to an API endpoint, chat preset, or UI. Users currently create automations through free-form chat with the Architect agent, which requires them to know HA automation structure. There is no guided, step-by-step experience and no way to invoke the automation builder at all.
+
+Two gaps need filling:
+
+1. **Guided mode** — a dedicated agent that walks users through automation creation step by step (trigger, entities, conditions, actions), with live validation and previews at each stage.
+2. **Manual instruction** — users who prefer free-form chat should be able to say "create an automation that..." to the Architect, which routes the intent to the automation builder workflow automatically.
+
+## User Scenarios & Testing
+
+### User Story 1 — Guided Automation Creation via Dedicated Agent (Priority: P1)
+
+As a user, I want to select "Create Automation" from the chat preset picker and be guided through each step (describe intent, confirm entities, review conditions, preview YAML, approve/deploy) interactively.
+
+**Acceptance Scenarios**:
+
+1. **Given** I select the "Create Automation" preset, **When** I type "turn off lights at 10pm", **Then** the agent extracts intent, validates entities, and presents a step-by-step summary before generating YAML.
+2. **Given** the agent asks me to confirm entities, **When** I correct an entity name, **Then** it re-validates and proceeds with the corrected entity.
+3. **Given** the agent generates YAML, **When** I see the preview, **Then** I can approve (creating a proposal), request modifications, or cancel.
+4. **Given** I approve, **Then** a proposal is created via the existing proposal pipeline and I'm directed to the proposals page.
+
+### User Story 2 — Architect Routes to Automation Builder (Priority: P1)
+
+As a user, I want to tell the Architect "create an automation that turns off lights when nobody is home" in general chat and have it automatically route to the automation builder workflow.
+
+**Acceptance Scenarios**:
+
+1. **Given** I send an automation-creation intent in general chat, **When** the orchestrator classifies the intent, **Then** it routes to the automation builder workflow (not free-form Architect).
+2. **Given** the automation builder handles my request, **When** it needs clarification, **Then** it asks within the same chat session.
+3. **Given** the builder generates valid YAML, **Then** it creates a proposal via `seek_approval` and informs me.
+
+### User Story 3 — Step Progress UI (Priority: P2)
+
+As a user, I want to see which step I'm on during guided automation creation (intent, entities, duplicates, YAML, preview) so I know the progress.
+
+**Acceptance Scenarios**:
+
+1. **Given** the automation builder is running, **When** each node completes, **Then** the activity panel shows the current step name and progress.
+2. **Given** validation errors occur, **When** the agent loops back, **Then** the step indicator shows the regression clearly.
+
+### Edge Cases
+
+- User starts guided flow then switches to a different chat session: state is checkpointed and can be resumed.
+- User describes something that isn't an automation (e.g. "tell me the weather"): agent recognizes this and redirects to Architect.
+- User requests a complex automation (multiple triggers, parallel actions): agent handles multi-step intent extraction.
+- The automation builder generates invalid YAML 3 times: agent gives up, presents best attempt, and suggests manual editing.
+
+## Requirements
+
+### Functional Requirements
+
+- **FR-001**: Automation builder workflow MUST be invocable via a chat workflow preset ("Create Automation").
+- **FR-002**: Orchestrator MUST route automation-creation intents to the automation builder (not Architect).
+- **FR-003**: Automation builder MUST create proposals via `seek_approval` (reusing existing proposal pipeline).
+- **FR-004**: Each step of the builder MUST stream status updates to the activity panel.
+- **FR-005**: Builder MUST support multi-turn clarification within a single chat session.
+- **FR-006**: Builder MUST checkpoint state to PostgreSQL for resumability.
+
+### Non-Functional Requirements
+
+- **NFR-001**: Step transitions should complete within 5 seconds for simple automations.
+- **NFR-002**: All nodes traced via MLflow (existing `traced_node` wrapper).
+
+### Key Entities
+
+- **AutomationBuilderState** (exists): Extended if needed for streaming step updates.
+- Reuses: AutomationProposal, HAEntity, workflow presets.
+
+## Success Criteria
+
+- **SC-001**: "Create Automation" preset appears in the chat workflow picker.
+- **SC-002**: Simple automations complete end-to-end (NL -> preview -> proposal) within 30 seconds.
+- **SC-003**: Orchestrator correctly routes 90%+ of automation-creation intents to the builder.
+- **SC-004**: Activity panel shows step progress during guided flow.

specs/001-project-aether/features/41-guided-workflow-agent/tasks.md

@@ -0,0 +1,90 @@
+# Tasks: Guided Workflow Agent
+
+**Feature**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md)
+
+---
+
+## Phase 1 — Wire the Automation Builder Agent
+
+- [ ] T4101 Create `src/agents/automation_builder/__init__.py` and `agent.py` with `AutomationBuilderAgent` class that:
+  - Compiles the existing `automation_builder` graph with PostgreSQL checkpointing
+  - Provides `start_session(user_message, session_id)` and `continue_session(user_message, session_id)` methods
+  - Streams node outputs as SSE-compatible messages
+  - Emits `emit_job_start` / `emit_job_status` / `emit_job_complete` events at each step
+  - Handles the HITL interrupt at preview (pauses and streams the YAML preview)
+
+- [ ] T4102 Enhance `preview_node` in `src/graph/nodes/automation_builder.py` to:
+  - Call `seek_approval()` from `src/tools/approval_tools.py` to create an actual `AutomationProposal`
+  - Include the `proposal_id` from the created proposal in the state
+  - Format a user-friendly preview message with YAML and a link to the proposals page
+
+- [ ] T4103 [P] Unit tests for `AutomationBuilderAgent` — mock LLM, verify streaming, verify job events, verify checkpoint resume
+
+**Checkpoint**: Agent compiles and can stream a simple automation flow end-to-end with mock LLM
+
+---
+
+## Phase 2 — Orchestrator Routing
+
+- [ ] T4104 Add `"create_automation"` intent to `OrchestratorAgent.classify_intent()` in `src/agents/orchestrator.py`:
+  - Add intent patterns: "create automation", "make automation", "set up automation", "automate", "when X then Y", "turn on/off ... at/when"
+  - When classified, set `target_agent = "automation_builder"` in `TaskPlan`
+
+- [ ] T4105 Update `src/api/routes/openai_compat/handlers.py` to handle `agent="automation_builder"`:
+  - When `agent` is `"automation_builder"` (explicit) or orchestrator routes to it (auto), invoke `AutomationBuilderAgent` instead of `ArchitectWorkflow`
+  - Reuse existing SSE streaming format
+
+- [ ] T4106 [P] Unit tests for orchestrator routing — verify "create an automation" classifies as `create_automation`, verify handler invokes correct agent
+
+**Checkpoint**: Saying "create an automation" in auto mode routes to the builder
+
+---
+
+## Phase 3 — Chat Preset and Activity Integration
+
+- [ ] T4107 Add "Create Automation" preset in `src/api/routes/workflows.py`:
+  - Name: "Create Automation"
+  - Description: "Guided step-by-step automation creation with live validation"
+  - Agents: `["automation_builder"]`
+  - Default agent: `automation_builder`
+
+- [ ] T4108 Add step status emissions to each node in `src/graph/nodes/automation_builder.py`:
+  - `gather_intent_node`: "Extracting automation intent..."
+  - `validate_entities_node`: "Validating {n} entities..."
+  - `check_duplicates_node`: "Checking for duplicates..."
+  - `generate_yaml_node`: "Generating YAML (attempt {n})..."
+  - `validate_yaml_node`: "Validating YAML..."
+  - `preview_node`: "Automation ready for review"
+
+- [ ] T4109 [P] Verify the WorkflowPresetSelector picks up the new preset automatically (no frontend changes needed — it reads from API)
+
+**Checkpoint**: "Create Automation" preset visible in chat, activity panel shows step progress
+
+---
+
+## Phase 4 — Multi-turn Clarification
+
+- [ ] T4110 Enhance `AutomationBuilderAgent` to support multi-turn clarification:
+  - When `needs_clarification=True`, stream the clarification question, then pause (checkpoint)
+  - On next user message, resume from checkpoint with the new message appended
+  - Same pattern for `entity_errors` — stream suggestions, wait for user correction
+
+- [ ] T4111 Add conversation threading: `AutomationBuilderAgent` uses `thread_id` (session ID) for LangGraph checkpointing so the user can pick up where they left off
+
+- [ ] T4112 [P] Integration test: multi-turn flow (user sends intent -> agent asks clarification -> user responds -> agent proceeds to YAML -> user approves)
+
+**Checkpoint**: Multi-turn guided flow works end-to-end
+
+---
+
+## Phase 5 — Polish and Documentation
+
+- [ ] T4113 Add help text to the "Create Automation" preset description explaining what users can do
+- [ ] T4114 Ensure error handling: LLM failures, invalid state, timeout — all surface user-friendly messages
+- [ ] T4115 Run `make ci-local` and fix any issues
+
+**Checkpoint**: Feature complete, CI green
+
+---
+
+`[P]` = Can run in parallel (different files, no dependencies)

src/api/routes/diagnostics.py

@@ -86,13 +86,23 @@ async def error_log() -> dict[str, Any]:
     entries = parse_error_log(raw_log)
     summary = get_error_summary(entries)
     by_integration = categorize_by_integration(entries)
-    known_patterns = analyze_errors(entries)
+    raw_patterns = analyze_errors(entries)
 
-    # Serialize by_integration: integration -> list of entry dicts
     serialized_by_int = {}
     for integration, int_entries in by_integration.items():
         serialized_by_int[integration] = [asdict(e) for e in int_entries]
 
+    _LEVEL_TO_SEVERITY = {"ERROR": "high", "CRITICAL": "high", "WARNING": "medium"}
+    known_patterns = [
+        {
+            "pattern": p.get("category", p.get("message", "")),
+            "severity": _LEVEL_TO_SEVERITY.get(p.get("level", ""), "low"),
+            "matched_entries": p.get("count", 0),
+            "suggestion": p.get("suggestion", ""),
+        }
+        for p in raw_patterns
+    ]
+
     return {
         "summary": summary,
         "by_integration": serialized_by_int,
@@ -172,9 +182,16 @@ async def recent_traces(limit: int = 50) -> dict[str, Any]:
             include_spans=False,
         )
 
+        from src.api.routes.jobs import _resolve_job_type
+
         items = []
         for t in traces:
             info = t.info
+            tags: dict[str, str] = getattr(info, "tags", {}) or {}
+            run_name = tags.get("mlflow.runName", "")
+            job_title = tags.get("ha.job_title", "")
+            title = job_title or (run_name.replace("_", " ").title() if run_name else "Unknown")
+
             items.append(
                 {
                     "trace_id": info.request_id,
@@ -183,6 +200,10 @@ async def recent_traces(limit: int = 50) -> dict[str, Any]:
                     else str(info.status),
                     "timestamp_ms": info.timestamp_ms,
                     "duration_ms": info.execution_time_ms,
+                    "run_name": run_name,
+                    "job_type": _resolve_job_type(run_name),
+                    "title": title,
+                    "conversation_id": tags.get("mlflow.trace.session", ""),
                 }
             )