Skip to content

v0.241.007#888

Merged
paullizer merged 176 commits intomainfrom
Staging
May 7, 2026
Merged

v0.241.007#888
paullizer merged 176 commits intomainfrom
Staging

Conversation

@paullizer
Copy link
Copy Markdown
Contributor

(v0.241.007)

New Feature

  • Improved Mobile UI Support

Bug Fixes

  • Uploaded File Preview Body XSS Hardening

    • Fixed the uploaded-file preview modal so stored file bodies no longer reach the preview pane through raw HTML sinks.
    • Plain-text previews now render as inert preformatted text, CSV-backed previews are built with DOM text nodes, and legacy HTML-backed table payloads now fall back to inert text instead of live markup.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened preview path.
    • (Ref: chat-input-actions.js, test_uploaded_file_preview_xss_fix.py, test_uploaded_file_preview_escaping.py, UPLOADED_FILE_PREVIEW_XSS_FIX.md)
  • Public Workspace Tag Color XSS Hardening

    • Fixed the public workspace tag surfaces so stored tag colors no longer reach folder-grid actions, tag badges, tag management rows, or selected-tag chips through inline handler or style interpolation.
    • Shared tag helper paths now normalize and validate tag colors on create and update across personal, group, and public routes, and previously stored invalid colors fall back to safe deterministic values on read.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened public tag rendering path.
    • (Ref: functions_documents.py, route_backend_documents.py, route_backend_group_documents.py, route_backend_public_documents.py, public_workspace.js, test_public_workspace_tag_color_xss_fix.py, test_public_workspace_tag_color_rendering.py, PUBLIC_WORKSPACE_TAG_COLOR_XSS_FIX.md)
  • Agent Template Gallery Actions Escaping

    • Fixed the agent template gallery so stored actions_to_load values no longer reach the recommended-actions row through a raw HTML sink.
    • Agent template helper paths now normalize actions_to_load consistently on read, create, and update flows, and invalid write payload shapes are rejected before they can persist.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened gallery path.
    • (Ref: agent_templates_gallery.js, functions_agent_templates.py, test_agent_template_gallery_actions_to_load_xss_fix.py, test_agent_template_gallery_actions_escaping.py, AGENT_TEMPLATE_GALLERY_ACTIONS_TO_LOAD_XSS_FIX.md)
  • Stored XSS Share, Activity, and Masking Hardening

    • Fixed the remaining stored-XSS share-modal flows so attacker-controlled user names, group names, descriptions, emails, and toast content no longer render through inline handlers or raw HTML sinks.
    • Hardened the group activity timeline and raw-activity modal so stored activity metadata and serialized activity JSON now render as inert text instead of executable markup.
    • Rebuilt masked-range rendering with DOM APIs and bound masking display names to the authenticated server-side user instead of trusting browser-supplied identity fields.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened sharing, activity, and masking paths.
    • (Ref: chat-toast.js, workspace-documents-sharing.js, group-documents-sharing.js, manage_group.js, chat-messages.js, route_backend_chats.py, test_stored_xss_share_activity_and_masking_fix.py, test_document_share_modal_escaping.py, STORED_XSS_SHARE_ACTIVITY_AND_MASKING_FIX.md)
  • Chat Scope Picker and Conversation Details XSS Hardening

    • Fixed the chat scope-lock picker so stored group and public workspace names no longer reach the locked-workspaces modal through raw HTML interpolation.
    • Hardened the conversation-details modal so attacker-controlled titles, context names, participant labels, document labels, semantic tags, classifications, and scope-lock names render as inert text, and invalid web-source values no longer produce active javascript: links.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the affected chat modal surfaces.
    • (Ref: chat-documents.js, chat-conversation-details.js, test_stored_xss_chat_scope_and_conversation_details_fix.py, test_chat_scope_lock_and_conversation_details_escaping.py, CHAT_SCOPE_LOCK_AND_CONVERSATION_DETAILS_XSS_FIX.md)
  • Chat Citation and Uploaded File Modal Filename XSS Hardening

    • Fixed the first-render chat citation modal so attacker-controlled document filenames returned from citation APIs no longer reach the modal header as raw HTML on the first open.
    • The uploaded-file preview modal now uses the same safe title-population path, closing the adjacent filename sink before it can regress into the same stored-XSS family.
    • Added focused functional and UI regression coverage plus versioned fix documentation for both modal title flows.
    • (Ref: chat-citations.js, chat-input-actions.js, test_stored_xss_chat_modal_filename_fix.py, test_chat_modal_filename_escaping.py, CITATION_AND_FILE_MODAL_FILENAME_XSS_FIX.md)
  • Stored XSS Agent and Member Rendering Hardening

    • Fixed the stored-XSS sink in chat message rendering so agent display names no longer reach the sender header, image header, or metadata drawer as raw HTML.
    • Public and group workspace member-management views now escape untrusted member display names and emails before rendering member rows, pending requests, ownership-transfer options, bulk-remove summaries, user-search results, and CSV validation previews, and the public member search no longer embeds untrusted values inside an inline onclick handler.
    • /api/userSearch now escapes Microsoft Graph OData filter literals before composing the $filter expression, so apostrophes in search input cannot break the backend Graph query.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened chat, workspace member-management, and Graph filter paths.
    • (Ref: chat-messages.js, manage_public_workspace.js, manage_group.js, route_backend_users.py, test_stored_xss_chat_workspace_rendering_fix.py, test_public_workspace_member_rendering_escaping.py, test_group_workspace_member_rendering_escaping.py, STORED_XSS_AGENT_AND_MEMBER_RENDERING_FIX.md)
  • Chat Selected Document Metadata Authorization Fix

    • Fixed chat selected-document metadata resolution so /api/chat, /api/chat/stream, and the selected tabular document helper no longer trust caller-supplied document ids after authentication.
    • Personal selected documents now resolve only for the owner or a legitimately shared user, group selected documents now honor authorized owner and shared-group access, and public selected documents now resolve only inside the caller's visible public workspaces.
    • Added focused regression coverage for the shared selected-document resolver and updated the existing all-scope tabular regression so the hardened lookup path stays covered.
    • (Ref: route_backend_chats.py, test_chat_selected_document_metadata_authorization.py, test_tabular_all_scope_group_source_context.py, CHAT_SELECTED_DOCUMENT_METADATA_AUTHORIZATION_FIX.md)
  • Control Center Public Workspace Members XSS Fix

    • Fixed a stored XSS in the Control Center public workspace members modal where stored member displayName and email values were rendered into an admin-facing HTML sink.
    • The members modal now builds the member row with DOM text nodes instead of injecting those fields through innerHTML, so malicious stored markup renders as inert text while the existing role badge styling remains unchanged.
    • Added focused regression coverage for the affected modal and documented the hardened sink under the current version line.
    • (Ref: workspace-manager.js, test_control_center_public_workspace_members_escaping.py, test_stored_xss_admin_rendering_fix.py, CONTROL_CENTER_PUBLIC_WORKSPACE_MEMBERS_XSS_FIX.md)
  • Plugin Log Recent Feed Admin Authorization Follow-Up

    • Fixed the adjacent plugin logging route so /api/plugins/invocations/recent now enforces the Admin role instead of exposing the cross-user recent invocation feed to any authenticated user.
    • Unauthenticated requests still return 401 Unauthorized, non-admin users now receive 403 Forbidden, and the admin response payload remains unchanged for legitimate troubleshooting flows.
    • Extended the focused plugin logging regression coverage so both admin-only plugin logging endpoints are exercised under unauthenticated, non-admin, and admin conditions.
    • (Ref: route_plugin_logging.py, test_plugin_logging_clear_logs_authorization.py, PLUGIN_LOG_RECENT_INVOCATIONS_ADMIN_FIX.md)
  • Public Workspace Details Projection Hardening

    • Fixed GET /api/public_workspaces/<workspace_id> so authenticated non-members no longer receive the full public workspace Cosmos document.
    • The route now returns a minimal public summary for non-members and a member-aware payload with explicit userRole and isMember fields for authorized workspace members, which preserves the manage-page UX without exposing manager lists, pending requests, or other member-only metadata.
    • Added focused functional and UI regression coverage to lock down the new payload contract and verify the public directory and non-member workspace page continue to behave correctly.
    • (Ref: route_backend_public_workspaces.py, functions_public_workspaces.py, manage_public_workspace.js, public_directory.js, test_security_authorization_hardening.py, test_public_workspace_projection_non_member_ui.py, PUBLIC_WORKSPACE_DETAILS_DISCLOSURE_FIX.md)
  • Approval Route Authorization Guard Consolidation

    • Hardened the approval detail, approve, and deny endpoints so both the admin and non-admin route variants now resolve requests through one shared authorization helper before returning approval data or executing destructive approval actions.
    • This reduces the chance of future drift between approval handlers while preserving the existing 403 Forbidden behavior for callers who are not allowed to view or approve a request.
    • Added focused regression coverage to ensure the approval routes continue using the shared authorization path.
    • (Ref: route_backend_control_center.py, functions_approvals.py, test_security_authorization_hardening.py)
  • Feedback Submission Ownership Enforcement

    • Fixed the user feedback submission route so caller-supplied conversationId and messageId values must resolve inside the authenticated user's own conversation before any feedback row is created.
    • Foreign conversation ids now return 403 Forbidden, missing assistant targets now return 404 Not Found, and invalid submissions no longer persist copied prompt or AI response content into the caller's feedback history.
    • Added focused regression coverage for owner success, foreign-conversation rejection before message lookup, and missing-target rejection without feedback persistence.
    • (Ref: route_backend_feedback.py, test_feedback_submission_authorization.py, FEEDBACK_AND_PLUGIN_LOG_ACCESS_CONTROL_FIX.md)
  • Plugin Log Clear Admin Authorization

    • Fixed the destructive plugin log clear endpoint so only administrators can wipe the shared in-memory plugin invocation history.
    • Unauthenticated requests still return 401 Unauthorized, non-admin authenticated users now receive 403 Forbidden, and admin behavior remains unchanged for legitimate maintenance flows.
    • Added focused regression coverage for unauthenticated, non-admin, and admin clear-log requests against the shared logger state.
    • (Ref: route_plugin_logging.py, test_plugin_logging_clear_logs_authorization.py, FEEDBACK_AND_PLUGIN_LOG_ACCESS_CONTROL_FIX.md)
  • Authorization State Confusion Settings Hardening

    • Completed the remaining settings-boundary hardening so active public workspace selection now validates server-side before it is persisted, instead of accepting arbitrary caller-supplied workspace ids through generic settings updates.
    • Public workspace selection routes now share the same validated helper path, and public prompt operations now resolve the active workspace through a canonical authorization check instead of trusting raw stored settings values.
    • The generic user-settings update route also now drops unsupported settings keys and returns a client error when a payload contains no valid settings keys, reducing the chance that authorization-sensitive state can bypass dedicated validators in future changes.
    • (Ref: functions_public_workspaces.py, route_backend_users.py, route_backend_public_workspaces.py, route_frontend_public_workspaces.py, route_backend_public_prompts.py, AUTHORIZATION_STATE_CONFUSION_SETTINGS_FIX.md)
  • Key Vault Plugin Secret Scope Enforcement

    • Fixed a plugin Key Vault authorization gap where well-formed full secret names could be stored or replayed across user, group, or global scopes and later resolved with the application's Key Vault identity.
    • Plugin secret save, runtime resolution, SQL connection-test resolution, and delete cleanup now verify that stored secret references match the expected scope and source before any Key Vault operation is attempted.
    • Added focused regression coverage and versioned fix documentation for the hardened plugin secret boundary.
    • (Ref: functions_keyvault.py, semantic_kernel_loader.py, route_backend_plugins.py, test_keyvault_plugin_secret_scope_enforcement.py, KEY_VAULT_PLUGIN_SECRET_SCOPE_ENFORCEMENT_FIX.md)
  • Log Analytics Query History User Scope Enforcement

    • Fixed the Log Analytics plugin so query history now binds to the authenticated user on the server instead of accepting an LLM-controlled user_id parameter.
    • Shared user-settings reads and writes now deny cross-user request access by default unless a reviewed privileged path explicitly opts into a cross-user bypass, and the Control Center admin flows have been updated to use that bypass intentionally.
    • Added focused regression coverage and versioned fix documentation for the plugin surface change and the shared user-settings authorization boundary.
    • (Ref: log_analytics_plugin.py, functions_settings.py, route_backend_control_center.py, test_log_analytics_plugin_user_scope_enforcement.py, LOG_ANALYTICS_PLUGIN_USER_SCOPE_ENFORCEMENT_FIX.md)
  • Personal Conversation Authorization

    • Closed personal-conversation authorization gaps so conversation deletion, chat file-content retrieval, and frontend conversation rendering verify ownership before returning or destroying data.
    • The chat message loader also now handles 403 Forbidden and 404 Not Found conversation-message responses explicitly, so the browser shows a controlled error state instead of assuming every message load succeeds.
    • Added focused functional and UI regression coverage plus a separate follow-up fix document under the current release line.
    • (Ref: route_backend_conversations.py, route_backend_documents.py, route_frontend_conversations.py, chat-messages.js, test_personal_conversation_followup_authorization.py, test_chat_messages_authorization_error.py, PERSONAL_CONVERSATION_AUTHORIZATION_FOLLOW_UP_FIX.md)
  • Personal Conversation Read Authorization Hardening

    • Fixed authenticated personal conversation read paths so message history and inline image retrieval now verify conversation ownership before returning content.
    • Requests that use leaked or foreign conversation identifiers now return 403 Forbidden instead of disclosing another user's transcript or image content, while the existing missing-resource response contracts remain unchanged.
    • Added focused regression coverage and versioned fix documentation for the hardened conversation read boundary.
    • (Ref: route_backend_conversations.py, test_conversations_read_ownership_authorization.py, PERSONAL_CONVERSATION_READ_AUTHORIZATION_FIX.md)
  • Broken Access Control IDOR Hardening

    • Closed the authenticated authorization gaps by enforcing personal conversation ownership in chat, binding tabular blob access to the current authorized request context, and binding fact-memory operations to that same canonical scope.
    • Request group and public workspace scope is now canonicalized before downstream processing so forged or stale scope identifiers do not survive into plugin execution or grounded-history fallback reuse.
    • Added focused regression coverage and versioned fix documentation for the hardened chat and plugin authorization boundary.
    • (Ref: route_backend_chats.py, tabular_processing_plugin.py, fact_memory_plugin.py, test_security_authorization_hardening.py, BROKEN_ACCESS_CONTROL_IDOR_HARDENING_FIX.md)
  • Stored XSS Admin Rendering Hardening

    • Closed the admin-side stored-XSS findings by escaping stored member and agent metadata before Control Center and Admin Settings HTML row rendering.
    • Control Center toast rendering now escapes message content by default and requires an explicit opt-in for the small number of admin success messages that intentionally include formatted HTML.
    • Added focused functional and UI regression coverage plus versioned fix documentation for the hardened admin rendering sinks.
    • (Ref: control_center.html, control-center.js, admin_agents.js, test_stored_xss_admin_rendering_fix.py, test_control_center_group_members_escaping.py, STORED_XSS_ADMIN_RENDERING_FIX.md)
  • Web Search Data Egress Hardening

    • Fixed the Bing-grounding web-search path so external web search now sends only the user's current message instead of a query derived from prior conversation context.
    • Updated the admin consent copy and user notice text to match the implemented behavior and warn that sensitive content pasted into the current message may still be sent when web search is used.
    • Reduced outbound web-search invocation metadata and added focused functional and UI regression coverage for the boundary and disclosure text changes.
    • (Ref: route_backend_chats.py, functions_settings.py, route_frontend_admin_settings.py, admin_settings.html, chats.html, test_web_search_current_message_only.py, test_web_search_notice_copy.py)
  • Authorization Boundary Hardening Across Search, Groups, Approvals, and History Fallback

    • Hardened several authenticated workflows that previously trusted caller-supplied identifiers or stale stored scope values, so active group selection, group-scoped prompt access, approval actions, and history-grounded follow-up reuse now revalidate the current user's authorization before proceeding.
    • Azure AI Search filter construction now escapes OData literals for document, user, group, shared, and public workspace identifiers, and the Control Center public workspace view now renders untrusted workspace metadata as inert text instead of raw HTML.
    • Added focused functional and UI regression coverage for the authorization and escaping paths, plus versioned fix documentation for the full hardening pass.
    • (Ref: functions_search.py, functions_group.py, route_backend_users.py, route_backend_group_prompts.py, route_backend_control_center.py, route_backend_chats.py, control-center.js, test_security_authorization_hardening.py, test_control_center_public_workspace_escaping.py)
  • Global Agent Scope Gate Fallback

    • Fixed per-user Semantic Kernel chats so selecting a global agent no longer silently falls back to the standard GPT model when personal agents are disabled for the tenant.
    • The per-user loader now treats global, personal, and group agent scopes separately, allowing valid global-agent selections to continue through agent invocation while keeping personal and group scope toggles enforced as configured.
    • Added regression coverage for the shared scope gate used by the per-user loader.
    • (Ref: semantic_kernel_loader.py, functions_agent_scope.py, test_global_agent_scope_gate.py, global agent request routing)

paullizer and others added 30 commits September 22, 2025 20:20
* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features
Updated else if for AUTHORITY
* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>
paullizer and others added 28 commits January 30, 2026 16:00
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* Update release_notes.md

* fixed sidebar race condition (#679)

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>
Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustio…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no…
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image …
* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API k…
* feat: Add ServiceNow integration documentation and bug fixes

- Add comprehensive ServiceNow integration guide with OAuth 2.0 setup
- Include OpenAPI specifications for Incident Management and Knowledge Base APIs
- Add agent instructions for ServiceNow support agent
- Fix GROUP_ACTION_OAUTH_SCHEMA_MERGING: Ensure additionalFields preserved during schema merge
- Fix GROUP_AGENT_LOADING: Improve group agent loading reliability
- Fix OPENAPI_BASIC_AUTH: Support basic authentication in OpenAPI actions
- Fix AZURE_AI_SEARCH_TEST_CONNECTION: Improve AI Search connection testing
- Update version to 0.236.012

* Removed the readme files for bug fix details

* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity a…
* Updated servicenow integration readme

* chore: Revert custom logo changes to upstream version

* chore: Revert terraform main.tf to upstream version

* Removed the two openai sample spec downloaed from servicennow site

* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned …
* Update docs/how-to/agents/ServiceNow/servicenow_agent_instructions.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️…
* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Checked in the bug fix detail readme to docs/explanation/fixes/v0.236.012

* Added version number to the feature readme files

* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
In…
* Added version number to document, and removed redudant import statement

* refactor: use _ for intentionally unused variable in AI Search test

- Changed 'indexes = list(...)' to '_ = list(...)'
- Follows Python convention for discarded return values
- AI Search connection test only needs to verify the API call succeeds

* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated imag…
* Removed azure_speech_managed_indeity_manual readme file since it is unrelated to this servicenow integration

* update version numbers to 0.236.012 in bug fix documentation

- OPENAPI_BASIC_AUTH_FIX.md: 0.235.026 → 0.236.012
- GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md: 0.235.028 → 0.236.012
- GROUP_AGENT_LOADING_FIX.md: 0.235.027 → 0.236.012
- AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md: 0.235.004 → 0.236.012"

* Update application/single_app/semantic_kernel_loader.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_now_knowledge_latest_spec_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/open_api_specs/sample_servicenow_incident_api_basicauth.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight m…
* Remvoed debug statements that might include senstive info

* Rollback Azure AI Search test connection fix for separate PR

Reverted route_backend_settings.py to origin/development version and removed
AZURE_AI_SEARCH_TEST_CONNECTION_FIX.md documentation. These changes will be
submitted in a dedicated PR to keep the ServiceNow integration PR focused.

* Update application/single_app/semantic_kernel_plugins/openapi_plugin_factory.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_AGENT_LOADING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/explanation/fixes/v0.236.012/GROUP_ACTION_OAUTH_SCHEMA_MERGING_FIX.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/how-to/agents/ServiceNow/SERVICENOW_OAUTH_SETUP.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Fix Azure AI Search test connection with managed identity

Replaced REST API approach with SearchIndexClient SDK to properly handle managed identity authentication in Azure public cloud. The SDK automatically handles token acquisition and endpoint construction, eliminating the 'search_resource_manager is not defined' error that occurred with the REST API approach.

* Corrected file folder name

* Corrected the version number to reference 0.236.012

* Removed unneeded folder and document

* Revert terraform main.tf to upstream/Development version

* updated the logging logic when running retention delete with archiving enabled (#642)

* Corrected version to 0.236.011 (#645)

* v0.237.001 (#649)

* Use Microsoft python base image

* Add python ENV vars

* Add python ENV vars

* Install deps to systme

* Add temp dir to image and pip conf support

* Add custom-ca-certificates dir

* Logo bug fix (#654)

* release note updating for github coplilot

* fixed logo bug issue

* added 2,3,4,5,6,14 days to rentention policy

* added retention policy time updates

* Rentention policy (#657)

* Critical Retention Policy Deletion Fix

* Create RETENTION_POLICY_NULL_LAST_ACTIVITY_FIX.md

* Added ServiceNow support for create and publish article.  Including readme for configuration for 2 agents.

* Replace actual servicenow instance name with generic name in the readme file

* Changed version number in ServiceNow readme files to 0.237.005 since this pull request has latest changes from v0.237.004 Development branch

* Enhance ServiceNow agent for managing new KB article creation

* Added readme and open ai specs and agent instructions to support ServiceNow asset management

* Remove any references to actual ServiceNow instances

* fixed retention policy runtime bug and sidebar bug (#672)

* Fix: Windows Unicode encoding issue for video uploads (#662)

- Added explicit UTF-8 encoding when reading file content on Windows
- Prevents UnicodeDecodeError when processing non-ASCII filenames
- Ensures consistent file handling across different operating systems

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md (#675)

Co-authored-by: vivche <vivche@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add custom subdomain support for OpenAI and Speech Service in Terraform (#558)

* Add custom subdomain support for OpenAI and Speech Service in Terraform

- Added custom_subdomain_name to OpenAI resource for managed identity authentication
- Created Speech Service resource with custom subdomain configuration
- Added RBAC role assignments for Speech Service (Managed Identity and App Service MI)
- Includes Cognitive Services Speech User and Speech Contributor roles
- Documentation: Azure Speech managed identity setup guide

* Update docs/how-to/azure_speech_managed_identity_manul_setup.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Chen, Vivien <Vivien.Chen+ecolab@ecolab.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* 0.237.006 (#676)

* Update chat-sidebar-conversations.js

* 0.237.006

* docs: Update release notes for ServiceNow integration and bug fixes

- Add comprehensive ServiceNow integration documentation entry
- Document 4 critical bug fixes:
  - OpenAPI Basic Authentication credential parsing fix
  - Group Action OAuth schema merging fix
  - Group Agent loading in per-user kernel mode fix
  - Azure AI Search test connection fix
- Move debug_print import to top-level in openapi_plugin_factory.py

Version: 0.237.005

* Update release_notes.md

* resolve conflict

* fixed sidebar race condition (#679)

* fix the version number in config.py

* Security: Restrict group agent loading to active group only

WHAT:
- Changed group agent loading from ALL user groups to ACTIVE group only
- Added security validation to prevent cross-group agent access
- Updated import from get_user_groups to require_active_group

WHY:
- Prevents users from accessing group agents outside their active group
- Fixes a security gap where selecting a group agent bypassed the active-group check
- Aligns behavior with the intended single-active-group security model

FILES:
- semantic_kernel_loader.py (approx. lines 11801280)
- GROUP_AGENT_LOADING_FIX.md

VERSION:
- 0.237.008

* Fixed an instruction error that caused semantic kernel to fall back to default agent

* Fixed! The issue was caused by duplicated code blocks (#683)

* Manage group frontend bug (#684)

* Fixed! The issue was caused by duplicated code blocks

* updated release notes

* initial feature add

* added tag endpoint

* return unused tags too

* Bicepfix (#690)

* v0.237.006 (#678)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and message pulse animations
HTML Templates: Added search modal, updated navigation icons
Backend Changes
route_backend_conversations.py:

/api/search_conversations - Full-text search with filters and pagination
/api/conversations/classifications - Get unique classification values
/api/user-settings/search-history - GET/POST/DELETE endpoints for search history
/api/conversations/{id}/pin - Toggle pin status
/api/conversations/{id}/hide - Toggle hide status
Bulk operations for pin/hide/delete
functions_settings.py: Search history management functions

* Message management (#553)

* added message masking

mask selected content of message or an entire message

* fixed citation border

* enabled streaming

* image gen with streaming

* added reasoning support

* added reasoning to agents

* agent support

* fixed key bug

* disable group create and fixed model fetch

* updated config

* fixed support for workspace search for streaming

* fix bug with sidebar update

* fixed gpt-5 vision processing bug

* metadata works with all messages now

* fixed debug_print bug

* added reasoning effort to agents and fixed agent validation

* fixed file metadata loading bug

* fixed llm streaming when working with group workspace data

* fixed cosmos container config error

* added delete message and fixed message threading

* retry bug fixes

* fixed message threading order

* moved message buttons to menu

* fixed bug for conversation history that included inactive threads

* added css styling for urls for dark mode

* fixed bug with newly created messages not showing metadata or deleting

* improved search times by 100x

* added token collect to messages supports models and agents

* added streaming for agents along with token collection

* added embedding token tracking

* added document creation/deletion and token tracking to activity log

* adding conversations to activity logs

* added activity log viewer with filters, search, and export

* added support for agents in edit and retry messages

* Configure Application from AZD Up command (#548)

* Add Cosmos DB post-configuration script and update requirements - Initial POC

* post deploy configure services in cosmosdb

* refactor to prevent post deploy configuration + begin support of key based auth.

* Add additional parameter validation for creating entra app

* Refactor Bicep modules for improved authentication and key management

- Added keyVault-Secrets.bicep module for storing secrets in Key Vault.
- Modified keyVault.bicep to remove enterprise app client secret handling and commented out managed identity role assignments.
- Removed openAI-existing.bicep and refactored openAI.bicep to handle model deployments dynamically.
- Added setPermissions.bicep for managing role assignments for various resources.
- Updated postconfig.py to reflect changes in environment variable handling for authentication type.

* Refactor Bicep modules to conditionally add settings based on authentication type and enable resource declarations for services

* initial support for VideoIndexer service

* Refactor Bicep modules to enhance VideoIndexer service integration and update diagnostic settings configurations

* move from using chainguard-dev builder image to python slim image.

* Updates to support post deployment app config

* Add post-deployment permissions script for CosmosDB and update authentication type handling

* fix typo in enhanced citation deployment config

* Refactor Dockerfile to use Python 3.13-slim and streamline build process

* restart web application after deployment settings applied

* remove setting for disableLocalAuth

* update to latest version of bicep deployment

* remove dead code

* code cleanup / formatting

* removed unnecessary content from readme.md

* fix token scope for commericial search service

* set permission correctly for lookup of openAI models

* fixes required to configure search with managed identity

* Adds Azure Billing Plugin in Community Customizations (#546)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add abp back in

* upd abp/seperate graph from query

* rmv missed merge lines

* fix for AL

* upd for consistency testing

* upd abp to community

* fix copilot findings #1

* fix plotting conflict

* fix exception handling

* fix static max function invokes

* rmv unneeded decl

* rmv unneeded imports

* fix grouping dimensions

* fix abp copilot suggestions #2

* simplify methods for message reload

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

* upd pipeline to include branch name in image

* add abp to deploy

* upd instructions name/rmv abp from deploy

* fix pipeline

* mov back to Comm Cust for main inclusion

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Security/container build (#549)

* upd dockerfile to google distroless

* add pipelines

* add modifications to container

* upd to build

* add missing arg

* add arg for major/minor/patch python version

* upd python paths and pip install

* add perms to /app for user

* chg back to root

* rmv python3

* rmv not built python

* add shared

* add path and home

* upd for stdlib paths

* fix user input filesystem path vulns

* fix to consecutive dots

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/speech managed identity (#543)

* Bugfix - deleted duplicate enable_external_healthcheck entry

* Feature - updated Speech Service to use Managed Identity in addition to the key, added MAG functionality via Azure Speech SDK since the Fast Transcription API is not available in MAG, updated Admin Setup Walkthrough so it goes to the right place in the settings when Next is clicked, updated Speech requirements in Walkthrough, rewrote Admin Configuration docs, updated/corrected Managed Identity roles in Setup Instructions Special docs.

* Update application/single_app/templates/admin_settings.html

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_settings.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update application/single_app/functions_documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Paul Lizer <paullizer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

* Control center (#567)

* added group status (active, locked, upload disabled, and inactive)

Adds Azure Billing Plugin in Community Customizations

* added bulk member upload via csv for groups

* add document metadata modified activity log tracking

* activity logging for members deleted from groups

* added group activity timeline

* added notification system

* added notifications for document upload to workspaces

* fixed badge sizing

* fixed url link

* fixed badge to not show with zero notifications

* Updated notification system

* Updated approval system

* updated approval workflow

* updated notification workflow

* Fixed set active bug on my public workspace page

* Added user retention policy, updated user profile page with dashboards, retention config, and more.

* adding speed to text for chat UI

* updated the speech wave form and input field

* updated to transcribe entire recording

* fixed bug creating new conversation with auto-send

* add mic permissions

* added stream token tracking

* Added public workspace reporting

* Updated AI search sizing analysis

* added management for public workspaces

* improved public workspace management includes stats and bulk actions

* updated groups dashboard for owners and admins with stats and bulk actions

* added voice for ai to talk with users in chats

* Auto Voice Response

* for speech service, added 429 randomized response pattern to prevent thunder herding

* updated admin settings for speech services and fixed dark mode for raw log viewing

* updated video extraction card

* Added Control Center Admin and Dashboard Reader roles

* updated feedback and safety decorators so admins work unless required then those roles must be used

* Updated and Validated logic for admin roles; control center, safety, and feedback

* added support for control center admin and dashboard reader

* Development (#566)

* Banner text color picker from Vivien (#555)

* Classification text color picker

* Line endings

* Remove opencensus

* Add flask instrumentation

* Add troubleshooting doc

* Add troubleshooting doc

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* updated tool tip to better inform user on status of ai response

* improve query parameters detection for swagger

* updated visual cue showing the ai is talking to the user

* moved duplicates to shared js

* replaced alert with toast.

* fixed and added log_event to exceptions

* added @user_required and improved swagger generation

* Update route_frontend_profile.py

* fixed swagger generation bug on affecting two apis

* returned keyvault to admin settings ui

* Fixed bug when running local js

---------

Co-authored-by: Ed Clark <107473135+clarked-msft@users.noreply.github.com>
Co-authored-by: Ed Clark <clarked@microsoft.com>
Co-authored-by: Bionic711 <13358952+Bionic711@users.noreply.github.com>

* Adding release notes

* fixed debug_debug_print

* Updated README

* Update README.md

* accepted changes

* removed files

* GitHub Actions workflow that runs Python compilation checks on all Python files in the single_app

* Upated to v0.235.002

* removed debug test file

* Updated to v0.235.003

* Update python-syntax-check.yml

* fixed disable group creation bug

* fixed bug not showing raw activity log for groups

* fixed control center access to not require admin role when enabling controlcenteradmin role

* fix documentation

* Update release_notes.md

* Update README.md

* added click restrictions to top items in control center

* bug fix - fixed group member select after search, group member removal, group member role update, and approve/reject request

* Retention execution activity log (#600)

* retention execution logging bug fix

* debug timer reset with admin save bug fix

* Create test_debug_logging_timer_preservation.py

* fixed file processing logic prevent runtime execution

* fixed bug processing execution against personal documents

* removed test logging

* Retention Policy Document Deletion Fix

* Improve execution logic for rentention policy

Now uses retention_policy_next_run timestamp - Compares current time against the stored next scheduled time. If current time >= next scheduled time, it runs.

Reduced check interval from 1 hour to 5 minutes - More responsive scheduling, ensures it catches the scheduled time promptly.

Better fallback logic - If next_run can't be parsed, falls back to checking last_run (23-hour threshold).

Runs immediately if never run before - If there's no last_run or next_run, it will execute on the first check.

* Updated branch flow

* added log_event to exceptions

* enforce-branch-flow

* v0.235.025 (#609)

* creating workflows

* fraud analyssis

* support agents

* update

* fix

* updated demo

* Swagger lite (#469)

* Development (#467)

* upgrade to v0.229.060 (#459)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* docs demos for public workspaces

* V0.229 bug fixes (v0.229.019) (#448)

* Development (#445)

* Update release notes to show support for GPT-5

* Documented support for gpt-image-1

* Update config.py

* remove documentation folder

* Documentation and message table support (#444)

* Develop demo docs and import markdown table support

* fixed enhanced citations for groups and public workspaces

* Updated to support showing public workspaces in scope

* Update config.py

* fix docs

* Updated RELEASE_NOTES

* video indexer config details, doc intel test button fix, move multimedia configs to search and extract

* improved header security

* updated versions

* moved

* Update EXTERNAL_HEALTH_CHECK_DUPLICATION_FIX.md

* added pdfs

* v0.229.019 bug fixes upgrade to v0.229.058 (#452)

* all urls in chat open in new tabs

* consolidated admin settings for improved navigation

* added left hand nav admin settings menus

* added left hand menu options for workspaces

* Added debug logging to video indexer processes

* readme and functional test

* Workspace Scope in Chat affects Prompts

* Create WORKSPACE_SCOPE_PROMPTS_FIX.md

* time based turn off for debug and file process logging

* improve saving in admin settings

* update to v0.229.058

* Update RELEASE_NOTES.md

* Update RELEASE_NOTES.md

* Popup modal for Health Check config

* Added Health Check config guide

* Chat page top nav bug (#458)

* initial fix

* fixed top nav chat up bug

* notes for v0.229.060

* file location fix

* Update config.py

* Update RELEASE_NOTES.md

* moved to correct location

* Fixed enhanced citations CSP bug

Simple Chat implemented improved security which negatively impacted enhanced citations.

* Updated release notes

* updated version and tests

* swagger support for all endpoints and added swagger search

* added wide screen support for chats when collapsing side bar

* v0.230.001 features

* adding support for xlsm, Macro Excel files.

* moved into features

* initial

* added readme

* removed html code

* Update config.py (#477)

Updated else if for AUTHORITY

* Initial Setup for Pages documentation (#479)

* setup folders and base files

* setting up files

* architecture diagrams

* updated to libdoc

* libdoc updates

* updating side bar

* removed loops

* editing side bar

* Created Simple Chat Jekyll theme

* Update config.py (#477) (#478)

Updated else if for AUTHORITY

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* Updating architectures

* Update README.md

---------

Co-authored-by: Patrick C Davis <82388365+Patrick-Davis-MSFT@users.noreply.github.com>

* initial

* added to base

* adding real data endpoints

* Update route_backend_control_center.py

* added individual charts

* fix for bug 485

* added document metrics

* added links to control center

* debug

* added date

* fixed bugs due to branch descrepancies

* added Azure SQL Driver Docker File

* added documentation for docker_fileSession updates

* Redis Managed Identity Azure Government Support Changes

* Stop tracking ignored folders

* updated gitignore

* added sort by to table for user management

* storage account size processing

* Front end now shows storage account sizing

* export user management list to csv

* adding group management

* fixing swagger generation

* fix

* Added inline dynamic property generation

* added YAML support

* Improved muiltform vs app/json detection

* added Control Center Admin role

ControlCenterAdmin

* ai search sizing is working for groups

* group refresh fixed

* added group data fix

* group table refresh

* updated export to include group docs

* adding public workspace management

* removed sample data and consolidated row generators

* Changed both caching helper functions to use the existing update_document() function from functions_documents.py instead of direct upsert.

* removed workflow, will work on that in different branch

* Document Set Fingerprinting, Scope-Aware Cache Key Generation, Event-Based Invalidation

I've successfully implemented Document Set Fingerprint + Event-Based Cache Invalidation with deterministic sorting and Score Normalization.

* added debug logging

* setup cache feature and ttl time to admin app settings

* removed cosmos level ttl

* Keyvault for secrets (#492)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/remove abp for pr (#510)

* add crude keyvault base impl

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* add ui trigger word and get agent helper

* adds safe calls for akv functions

* add keyvault to global actions loading

* rmv secret leak via logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add security tab UI + key vault UI

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Feature/group agents actions (#521)

* add crude keyvault base impl

* upd actions for MAG

* add settings to fix

* upd secret naming convention

* upd auth types to include conn string/basic(un/pw)

* fix method name

* add get agent helper

* add ui trigger word and get agent helper

* upd function imports

* upd agents call

* add desc of plugins

* fix for admin modal loading

* upd default agent handling

* rmv unneeded file

* rmv extra imp statements

* add new cosmos container script

* upd instructions for consistency of code

* adds safe calls for akv functions

* adds akv to personal agents

* fix for user agents boot issue

* fix global set

* upd azure function plugin to super init

* upd to clean imports

* add keyvault to global actions loading

* add plugin loading docs

* rmv secret leak via logging

* rmv displaying of token in logs

* fix not loading global actions for personal agents

* rmv unsupported characters from logging

* fix chat links in dark mode

* chg order of css for links in dark mode

* fix chat color

* add default plugin print logging

* rmv default check for nonsql plugins

* upd requirements

* add keyvault and dynamic addsetting ui

* fix for agents/plugins with invalid akv chars

* add imp to appins logging

* add security tab UI + key vault UI

* add keyvault settings

* fix for copilot findings.

* fix for resaving plugin without changing secret

* init azure billing plugin

* add app settings cache

* upd to azure billing plugin

* upd to msgraph plugin

* init community customizations

* add module

* add key vault config modal

* add logging and functions to math

* rmv extra telemetry, add appcache

* upd billing plugin

* add/upd key vault, admin settings, agents, max tokens

* Remove abp for pr

* disable static logging for development

* rmv dup import

* add note on pass

* added notes

* rmv dup decl

* add semicolon

* rmv unused variable add agent name to log

* add actions migration back in

* add notes and copilot fixes

* add group agents/actions

* add branch for testing/rmv old branch

* bug fixes, group agent modifications, rmv client validation

* rmv ajv

* upd from copilot

---------

Co-authored-by: Bionic711 <nadoyle@microsoft.com>

* Add cosmos activity logs container configuration

* incorporate branch updates Add 372 fix 489

* Support deployment via AZD UP (#530)

* Update devcontainer configuration for support of AZD

* Move to module based bicep files

* Add Azure deployment configuration and update Bicep modules for service outputs

* Enhance Azure deployment process by adding predeploy hooks for Docker image management and updating Bicep modules to include managed identity client ID and container registry outputs.

* Add deployment script for creating and storing Azure AD client secret in Key Vault

* Update Azure Dev CLI feature version to latest in devcontainer configuration

* Remove deprecated Bicep files and parameter configurations for cleaner deployment structure

* Refactor Bicep modules for improved diagnostics and role assignments

- Updated appService.bicep to conditionally import diagnostic settings based on enableDiagLogging parameter.
- Changed Azure Cosmos DB authentication type to managed identity and removed key-based authentication settings.
- Enhanced appServiceAuthentication.bicep by removing unnecessary parameters and configuring Key Vault reference for client secret.
- Modified appServicePlan.bicep to conditionally import diagnostic settings.
- Refactored azureContainerRegistry-existing.bicep to deploy role assignment to the ACR's resource group.
- Updated azureContainerRegistry.bicep to conditionally import diagnostic settings.
- Enhanced contentSafety.bicep with conditional diagnostic settings import.
- Updated cosmosDb.bicep to include a new database and container, and added role assignments for managed identity.
- Refactored documentIntelligence.bicep to conditionally import diagnostic settings.
- Enhanced enterpriseApplication.bicep by adding additional required resource access scopes.
- Updated keyVault.bicep to conditionally import diagnostic settings and adjusted enterprise app parameters.
- Refactored openAI.bicep to conditionally import diagnostic settings.
- Enhanced redisCache.bicep with conditional diagnostic settings import.
- Updated search.bicep to conditionally import diagnostic settings.
- Refactored speechService.bicep to conditionally import diagnostic settings.
- Enhanced storageAccount.bicep with conditional diagnostic settings import.
- Added main.parameters.json for parameter management.
- Introduced azureContainerRegistry-roleAssignment.bicep for managing ACR role assignments.

* Add custom subdomain names for document intelligence, OpenAI, and speech services

* Fix casing for hostingMode property in search service configuration

* Enhance storage account configuration by enabling hierarchical namespace and setting public access to 'None' for document containers

* Add enterprise app permissions module for resource access management

* Fixed ExternalApi configuration to valid guid and set value to a unique name

* Add Init Script to Configure Entra Application

* Fix spelling error

* fix failure in hostingMode value

* configure managed identity for contentSafety

* update readme to support new AZD deployment solution

* Video Indexer, Multi-Modal Enhancements, Scope Bug

## PR Summary: Video Indexer Multi-Modal Enhancements

### Overview
This PR introduces significant enhancements to video processing and image analysis capabilities, focusing on multi-modal AI features and improved metadata handling. **Version updated from 0.233.167 to 0.233.172**.

### 🎯 Key Features

#### 1. **Multi-Modal Vision Analysis for Images**
- Added AI-powered vision analysis for uploaded images using GPT-4 Vision or similar models
- Extracts comprehensive image insights including:
  - AI-generated descriptions
  - Object detection
  - Text extraction from images (OCR)
  - Detailed visual analysis
- New admin setting: `enable_multimodal_vision` to control feature availability
- Vision analysis results stored in document metadata and included in AI Search indexing
- Connection testing endpoint added for vision model validation

#### 2. **Enhanced Document Metadata Citations**
- Implemented metadata-based citations that surface document keywords, abstracts, and vision analysis
- New citation types displayed with distinct visual indicators:
  - **Keywords**: Tagged with `bi-tags` icon, labeled as "Metadata"
  - **Abstract**: Document summaries included as contextual citations
  - **Vision Analysis**: AI-generated image insights labeled as "AI Vision"
- Metadata content passed to AI models as additional context for more informed responses
- Special modal view for metadata citations (separate from standard document citations)

#### 3. **Image Message UI Improvements**
- Enhanced display for user-uploaded images vs AI-generated images
- Added "View Text" button for uploaded images with extracted content or vision analysis
- Collapsible info sections showing:
  - Extracted OCR text from Document Intelligence
  - AI Vision Analysis results
- Proper avatar distinction between uploaded and generated images
- Improved metadata tracking with `is_user_upload` flag

#### 4. **Video Indexer Configuration Updates**
- **BREAKING CHANGE**: Removed API key authentication support
- Now exclusively uses **Managed Identity authentication** for Video Indexer
- Updated admin UI documentation to guide managed identity setup:
  - Enable system-assigned managed identity on App Service
  - Assign "Video Indexer Restricted Viewer" role
  - Configure required ARM settings (subscription ID, resource group, account name)
- Improved validation for required Video Indexer settings
- Enhanced error messaging for missing configuration

#### 5. **Search Scope Improvements**
- Fixed search behavior when `document_scope='all'` to properly include group documents
- Added `active_group_id` to search context when document scope is 'all' and groups are enabled
- Conditional group index searching - only queries group index when `active_group_id` is present
- Prevents unnecessary searches and potential errors when groups aren't in use

#### 6. **Image Context in Conversation History**
- Enhanced conversation history to include rich image context for AI models
- Extracts and includes:
  - OCR text from Document Intelligence (up to max content length)
  - AI Vision analysis (description, objects, text)
  - Structured prompt formatting for multimodal understanding
- **Important**: Base64 image data excluded from conversation history to prevent token overflow
- Only metadata and extracted insights passed to models for efficient token usage

### 🔧 Technical Improvements

#### Backend Changes
- **route_backend_chats.py**: 
  - Added metadata citation extraction logic (~150 lines)
  - Enhanced conversation history building for image uploads
  - Improved search argument handling for group contexts
  
- **functions_documents.py**:
  - New `analyze_image_with_vision_model()` function for AI vision analysis
  - Enhanced `get_document_metadata_for_citations()` integration
  - Vision analysis now runs BEFORE chunk saving to include insights in AI Search indexing
  - Removed redundant blob storage for vision JSON (stored in document metadata)
  
- **route_backend_settings.py**:
  - New `_test_multimodal_vision_connection()` endpoint for testing vision models
  - Supports both APIM and direct Azure OpenAI endpoints
  - Test uses 1x1 pixel sample image for validation

- **functions_search.py**:
  - Added conditional logic for group search execution
  - Prevents empty `active_group_id` from causing search errors

#### Frontend Changes
- **chat-messages.js** (~275 lines changed):
  - Enhanced `appendMessage()` to handle uploaded image metadata
  - New `toggleImageInfo()` functionality for expandable image details
  - Improved citation rendering with metadata type indicators
  - Debug logging for image message processing
  
- **chat-citations.js** (~70 lines added):
  - New `showMetadataModal()` function for displaying keywords/abstracts/vision analysis
  - Enhanced citation click handling to detect metadata citations
  - Separate modal styling and behavior for metadata vs document citations

- **admin_settings.html**:
  - Complete redesign of Video Indexer configuration section
  - Removed all API key references
  - Added managed identity setup instructions with step-by-step guidance
  - Updated configuration display to show resource group and subscription ID

- **_video_indexer_info.html**:
  - Updated modal content to clarify managed identity requirement
  - Added warning banner about authentication type
  - Enhanced configuration display with ARM resource details

### 📊 Files Changed
- **16 files** modified
- **+1,063 insertions**, **-412 deletions**
- Net change: **+651 lines**

### 🧪 Testing Considerations
- Test multi-modal vision analysis with various image types
- Validate metadata citations appear correctly in chat responses
- Verify Video Indexer works with managed identity authentication
- Test search scope behavior with and without groups enabled
- Validate image upload UI shows extracted text and vision analysis
- Confirm conversation history properly handles image context without token overflow

### 🔐 Security & Performance
- Managed identity authentication improves security posture (no stored API keys)
- Image base64 data excluded from conversation history prevents token exhaustion
- Metadata citations add minimal overhead while providing rich context
- Vision analysis runs efficiently during document processing pipeline

### 📝 Configuration Required
Admins must configure:
1. Enable `enable_multimodal_vision` in admin settings
2. Select vision-capable model (e.g., `gpt-4o`, `gpt-4-vision-preview`)
3. For Video Indexer: Configure managed identity and ARM resource details
4. Enable `enable_extract_meta_data` to surface metadata citations

---

This PR significantly enhances the application's multi-modal capabilities, providing users with richer context from images and documents while maintaining efficient token usage and robust security practices.

* Conversation Management Features (#532)

New Features
1. Pin Conversations
Users can pin important conversations to keep them at the top of the list
Pinned conversations display a pin icon (📌) in the conversation header and details modal
Pin icon appears before the conversation title
Bulk pin/unpin operations available in multi-select mode
Pinned conversations always appear first, sorted by most recent activity
2. Hide Conversations
Users can hide conversations to declutter their workspace without deleting them
Hidden conversations display an eye-slash icon (👁️‍🗨️) in the conversation header and details modal
Eye-slash icon appears next to the pin icon (if pinned)
Bulk hide/unhide operations available in multi-select mode
Toggle visibility of hidden conversations using the eye icon in the sidebar
3. Two-Tier Conversation Search
Quick Search (Sidebar)
Instant title-based filtering of conversations
Search icon in sidebar activates inline search input
Real-time filtering as you type
Clear button to reset search
Expand button to open advanced search modal
Advanced Search (Modal)
Full-text search across all message content
Multiple filter options:
Date range (from/to)
Chat type (personal/group/public)
Classifications (multi-select)
Has uploaded files
Has generated images
Pagination (20 results per page)
Message snippets with highlighted search terms (50 chars before/after match)
Click to navigate directly to specific messages
Search history tracking (last 20 searches)
Clickable search history to repeat searches
4. Message Highlighting & Navigation
Search results highlight matched text in yellow (amber in dark mode)
Smooth scroll animation to navigate to specific messages
Pulse animation draws attention to the target message
Highlights persist for 30 seconds before auto-clearing
Works across conversation switches
5. Multi-Select Mode
Select multiple conversations for bulk operations
Visual checkboxes appear when entering selection mode
Bulk actions available:
Pin/unpin selected conversations
Hide/unhide selected conversations
Delete selected conversations
Selection mode accessible from conversation dropdown menu
Auto-exit after 30 seconds of inactivity
6. Enhanced Conversation Details Modal
Displays pin icon if conversation is pinned
Displays eye-slash icon if conversation is hidden
Shows both icons at the top of the modal (next to title)
Status section shows visual badges for pinned/hidden state
Comprehensive metadata display
Technical Implementation
Frontend Changes
chat-conversations.js: Core conversation management, quick search, pin/hide functionality
chat-search-modal.js (NEW): Advanced search modal implementation
chat-sidebar-conversations.js: Sidebar search synchronization, hidden conversation handling
chat-messages.js: Message highlighting, smooth scroll, search highlight persistence
chat-conversation-details.js: Updated to show pin/hidden icons in modal
chats.css: Styles for search highlights and messa…
// Use agent display name if available, otherwise show AI with model
if (agentDisplayName) {
senderLabel = agentDisplayName;
senderLabel = escapeHtml(agentDisplayName);

import { escapeHtml } from "./workspace-utils.js";
import { initializeTags, renderTagBadges, loadWorkspaceTags } from "./workspace-tags.js";
import { initializeTags, renderTagBadges, loadWorkspaceTags, currentView } from "./workspace-tags.js";
@paullizer paullizer merged commit c629438 into main May 7, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants