Python: Add Ollama text to image service - #14437
Shoumik Chakravarty (shoumikchakravarty-dev) wants to merge 3 commits into
Conversation
Adds OllamaTextToImage, a TextToImageClientBase implementation backed by the Ollama generate endpoint, which returns a base64 image payload for image generation models such as x/z-image-turbo. The service follows the existing per-modality layout of the Ollama connector: a new OLLAMA_IMAGE_MODEL_ID setting, an OllamaTextToImagePromptExecutionSettings class carrying width, height and steps, and exports from the connector package. The deprecated width and height arguments of generate_image are forwarded to the endpoint when the settings do not already specify them. A response without image data raises ServiceInvalidResponseError rather than returning empty bytes. Adds unit tests covering initialization, custom host and client, base64 decoding, get_image_content, size settings, the deprecated size arguments and their precedence, and the missing-image error path. Closes microsoft#13938
There was a problem hiding this comment.
🟢 Approval recommended
The implementation and tests are complete; only a minor documentation nit remains.
Pull request overview
Adds Ollama text-to-image support with configurable image generation settings and base64-decoded output.
Changes:
- Adds
OllamaTextToImageand public exports. - Adds image model configuration and generation settings.
- Adds comprehensive unit tests.
File summaries
| File | Summary |
|---|---|
python/tests/unit/connectors/ai/ollama/services/test_ollama_text_to_image.py |
Tests image generation behavior and error handling. |
python/tests/unit/connectors/ai/ollama/conftest.py |
Adds image model test configuration. |
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py |
Implements Ollama image generation. |
python/semantic_kernel/connectors/ai/ollama/ollama_settings.py |
Adds the image model setting. |
python/semantic_kernel/connectors/ai/ollama/ollama_prompt_execution_settings.py |
Adds image size and steps settings. |
python/semantic_kernel/connectors/ai/ollama/__init__.py |
Exports the new service. |
Review details
Suppressed comments (1)
python/semantic_kernel/connectors/ai/ollama/init.py:13
- The connector index in
python/semantic_kernel/connectors/ai/README.mdlists each provider's public modality service, including the existing OpenAI and Azure text-to-image clients, but does not list this new Ollama client. AddOllamaTextToImagethere so the exported service is discoverable and the index stays complete.
from semantic_kernel.connectors.ai.ollama.services.ollama_text_to_image import OllamaTextToImage
- Files reviewed: 6/6 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Adds the new service to the modality table in connectors/ai/README.md, next to the other Ollama services, so the exported client is discoverable alongside the OpenAI and Azure text to image clients.
There was a problem hiding this comment.
🔵 Needs a closer look
Fix typed-settings handling and keyword collisions, update the central settings reference, and add SDK-shaped response coverage.
Review details
Suppressed comments (5)
Previously missed (1) — in code that hasn't changed since the last review.
python/semantic_kernel/connectors/ai/ollama/ollama_settings.py:34
- The new
OLLAMA_IMAGE_MODEL_IDsetting is not documented in the repository's settings reference (python/samples/concepts/setup/ALL_SETTINGS.md), which currently lists the three existing Ollama services but not this new service. Please add the corresponding service/model-id/host row so users following the central setup guide can discover and configure the connector.
python/semantic_kernel/connectors/ai/README.md:51
- The settings reference at
python/samples/concepts/setup/ALL_SETTINGS.mdis linked from the setup documentation and lists each Ollama service, but this new service and itsOLLAMA_IMAGE_MODEL_IDenvironment variable are missing there. Add the service'sai_model_id/hostentry so users can discover the required configuration.
| | [`OllamaTextToImage`](./ollama/services/ollama_text_to_image.py) |
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py:104
- This unconditionally calls
from_prompt_execution_settings, so an already-typed settings object is repacked intoextension_databefore each request. If a caller clears a previously set field (for example,settings.width = None),pack_extension_data()leaves the old value inextension_dataand this conversion restores it, meaning the request ignores the caller's current settings. Preserve typed settings (or use the service helper) instead.
else OllamaTextToImagePromptExecutionSettings.from_prompt_execution_settings(settings)
python/semantic_kernel/connectors/ai/ollama/services/ollama_text_to_image.py:127
- Merging
kwargsintooptionsand then supplyingmodel,prompt, andstreamexplicitly makes any of those validAsyncClient.generatearguments fail withTypeError: multiple values for keyword argument(for example,generate_image(prompt, stream=False)). Since the method documentskwargsas endpoint arguments, reserve these request-control keys or build one dictionary and apply the fixed values before calling the client.
options = image_settings.prepare_settings_dict()
options.update(kwargs)
response_object = await self.client.generate(
python/tests/unit/connectors/ai/ollama/services/test_ollama_text_to_image.py:58
- All success and missing-image mocks here are mappings, but the Ollama SDK returns a
GenerateResponse, so the primarygetattr(response_object, "image")path in the new service is never exercised. Add a typed/SDK-shaped response case to verify the integration with the actual response object.
mock_generate.return_value = {"image": ENCODED_IMAGE}
- Files reviewed: 7/7 changed files
- Comments generated: 0 new
- Review effort level: Lite
Preserve already-typed execution settings instead of always converting them. The unconditional from_prompt_execution_settings call packed the settings into extension_data on every request, so a field the caller had cleared was restored from the stale packed value and the request ignored the current settings. Build a single request dict instead of merging kwargs into the settings dict and passing model, prompt and stream separately, which raised TypeError on duplicate keyword arguments when a caller passed one of those as a documented endpoint argument. The service-controlled keys now override settings and kwargs. Apply the deprecated width and height arguments to the request rather than to the settings object, so a caller's settings are no longer mutated, keeping the same precedence: settings win when they carry a size. List OllamaTextToImage and OLLAMA_IMAGE_MODEL_ID in ALL_SETTINGS.md next to the other Ollama services. Adds tests for a GenerateResponse-shaped result (with and without image data), which exercises the attribute path the SDK actually returns, plus regression tests for the settings repacking, the keyword collision, and settings mutation.
Motivation and Context
Closes #13938
Ollama supports image generation https://ollama.com/blog/image-generation through
POST /api/generate, which returns a base64-encodedimagefield for image models. Semantic Kernel has no Ollama text-to-image service, so these models cannot be used through theTextToImageClientBaseabstraction.Description
Adds
OllamaTextToImage, following the connector's existing one-service-per-modalityollama_text_to_image.pyimplementinggenerate_image()viaAsyncClient.generate()and returning thedecoded image bytes
OLLAMA_IMAGE_MODEL_IDsetting onOllamaSettings, matching the existing chat, text and embedding model IDs.OllamaTextToImagePromptExecutionSettingscarryingwidth,heightandsteps.connectors/ai/ollama/__init__.py.No new dependencies: the
ollamapackage SK already depends on exposesimageonGenerateResponse, andwidth,heightandstepsongenerate().Contribution Checklist