[AI] support on-device structured output by milaGGL · Pull Request #8395 · firebase/firebase-android-sdk

milaGGL · 2026-07-06T15:42:16Z

This PR introduces Structured Output support on device, allowing developers to generate strongly-typed objects using ML Kit's backend. It defines the public API (GenerateObjectResponse and generateObject()), handles the necessary Kotlin Symbol Processing (KSP) logic for schema translation.

How to use On-Device Structured Output

Define a data class annotated with @generable (and @Guide). Note: You must provide an empty companion object so the KSP processor can generate the necessary schema extensions.
Call generateObject

import com.google.firebase.ai.annotations.Generable
@Generable
data class Recipe(
    val recipeName: String,
    @Guide(description = "A list of ingredients", minItems = 2)
    val ingredients: List<String>,
    @Guide(description = "Preparation time in minutes", minimum = 1.0)
    val prepTimeMinutes: Int
) {
    companion object
}

val schema = Recipe.firebaseAISchema()
val prompt = "Give me a recipe for chocolate chip cookies"

val response = model.generateObject(schema, prompt)
val recipe: Recipe? = response.getObject()
recipe?.let {
    println("Recipe: ${it.recipeName}")
    println("Ingredients: ${it.ingredients.joinToString()}")
    println("Prep time: ${it.prepTimeMinutes} mins")
}

github-actions · 2026-07-06T15:42:34Z

📝 PRs merging into main branch

Our main branch should always be in a releasable state. If you are working on a larger change, or if you don't want this change to see the light of the day just yet, consider using a feature branch first, and only merge into the main branch when the code complete and ready to be released.

google-oss-bot · 2026-07-06T15:42:47Z

	1 Warning
⚠️	Did you forget to add a changelog entry? (Add the 'no-changelog' label to the PR to silence this warning.)

Generated by 🚫 Danger

gemini-code-assist · 2026-07-06T15:42:58Z

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

google-oss-bot · 2026-07-27T21:23:42Z

The public api surface has changed for the subproject ai-logic_firebase-ai-ondevice-interop:
error: Added class com.google.firebase.ai.ondevice.interop.GenerateObjectResponse [AddedClass]
error: Added method com.google.firebase.ai.ondevice.interop.GenerativeModel.generateObject(com.google.firebase.ai.ondevice.interop.GenerateContentRequest,kotlin.reflect.KClass,kotlin.coroutines.Continuation<? super com.google.firebase.ai.ondevice.interop.GenerateObjectResponse>) [AddedAbstractMethod]

Please update the api.txt files for the subprojects being affected by this change by running ./gradlew ${subproject}:generateApiTxtFile. Also perform a major/minor bump accordingly.

milaGGL · 2026-07-28T17:52:18Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for structured object generation (generateObject) in the on-device model interop and implementation layers, including updates to the KSP processor to generate ML Kit companion classes and support enum values in the @Guide annotation. Feedback on these changes highlights a type-safety issue in GenerateObjectResponse.kt regarding MutableList invariance, as well as an opportunity to replace unsafe non-null assertions (!!) with idiomatic safe calls in SchemaSymbolProcessorVisitor.kt.

milaGGL · 2026-07-28T20:09:41Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for structured object generation on-device by integrating with ML Kit's schema-based generation. It updates the KSP processor to generate companion classes (_MlKitCompanion) for @Generable classes, adds the generateObject API to the on-device interop and implementation layers, and updates GenerateObjectResponse to support lazy-loading and caching of deserialized instances. The review feedback highlights several important improvements: handling nested classes correctly when resolving companion class names, refining the caching logic in GenerateObjectResponse to prevent redundant deserialization of null values, safely escaping enum values using KotlinPoet's %S specifier, and propagating KDoc-based class and property descriptions to the generated companion annotations.

gemini-code-assist · 2026-07-28T20:11:57Z

+    schemaClass: kotlin.reflect.KClass<T>
+  ): com.google.firebase.ai.ondevice.interop.GenerateObjectResponse<T> =
+    try {
+      val companionClassName = "${schemaClass.java.name}_MlKitCompanion"


If schemaClass is a nested class (e.g., Outer.Inner), schemaClass.java.name will be Outer$Inner. However, KSP generates the companion class as a top-level class named Inner_MlKitCompanion in the same package. Using ${schemaClass.java.name}_MlKitCompanion will resolve to Outer$Inner_MlKitCompanion, which does not exist and will cause a runtime IllegalArgumentException. Construct the companion class name using the package name and the simple name of the class to support nested classes correctly.

val packageName = schemaClass.java.name.substringBeforeLast('.', "") val simpleName = schemaClass.java.simpleName val companionClassName = if (packageName.isEmpty()) "${simpleName}_MlKitCompanion" else "$packageName.${simpleName}_MlKitCompanion"

gemini-code-assist · 2026-07-28T20:11:57Z

 public class GenerateObjectResponse<T : Any>
 internal constructor(
  public val response: GenerateContentResponse,
-  internal val schema: JsonSchema<T>
+  internal val schema: JsonSchema<T>?,
+  internal var instances: MutableList<T?>?
 ) {

+  internal constructor(


Introduce a resolvedIndices set to track which candidate indices have already been resolved. This is necessary to distinguish between an unresolved candidate and a candidate that has been resolved to null (e.g., due to empty text or deserialization failure), preventing redundant deserialization attempts.

Suggested change

public class GenerateObjectResponse<T : Any>

internal constructor(

public val response: GenerateContentResponse,

internal val schema: JsonSchema<T>

internal val schema: JsonSchema<T>?,

internal var instances: MutableList<T?>?

) {

internal constructor(

public class GenerateObjectResponse<T : Any>

internal constructor(

public val response: GenerateContentResponse,

internal val schema: JsonSchema<T>?,

internal var instances: MutableList<T?>?

) {

private val resolvedIndices = HashSet<Int>()

internal constructor(

gemini-code-assist · 2026-07-28T20:11:57Z

+      // 1. Fast Path / Cache Hit: Return immediately if already resolved or in memory
+      insts?.getOrNull(candidateIndex)?.let {
+        return it
+      }


Check if the candidate index has already been resolved or if we are in on-device mode (where schema is null and instances are pre-populated), and return the cached value directly. This avoids skipping the cache when the resolved value is null.

Suggested change

// 1. Fast Path / Cache Hit: Return immediately if already resolved or in memory

insts?.getOrNull(candidateIndex)?.let {

return it

}

// 1. Fast Path / Cache Hit: Return immediately if already resolved or in memory

if (insts != null && (schema == null || candidateIndex in resolvedIndices)) {

return insts.getOrNull(candidateIndex)

}

gemini-code-assist · 2026-07-28T20:11:57Z

+      if (candidateIndex < currentInstances.size) {
+        currentInstances[candidateIndex] = deserialized
+      }


Mark the candidate index as resolved when saving the deserialized value to the cache.

Suggested change

if (candidateIndex < currentInstances.size) {

currentInstances[candidateIndex] = deserialized

}

if (candidateIndex < currentInstances.size) {

currentInstances[candidateIndex] = deserialized

resolvedIndices.add(candidateIndex)

}

gemini-code-assist · 2026-07-28T20:11:57Z

+            .indent()
+            .addStatement("values = listOf(")
+            .indent()
+            .addStatement(guideValues.enumValues.joinToString { "\"$it\"" })


Using joinToString { "\"$it\"" } to embed raw string values directly into the generated code block can result in invalid Kotlin code if any enum value contains double quotes or other special characters. Use KotlinPoet's %S format specifier to safely escape and format the string values.

Suggested change

.addStatement(guideValues.enumValues.joinToString { "\"$it\"" })

.addStatement(

guideValues.enumValues.joinToString { "%S" },

*guideValues.enumValues.toTypedArray()

)

gemini-code-assist · 2026-07-28T20:11:57Z

+    val packageName = classDeclaration.packageName.asString()
+    val companionClassName = "${classDeclaration.simpleName.asString()}_MlKitCompanion"


Extract the property KDocs once at the beginning of the function so they can be used as fallbacks for property descriptions in the generated companion class.

Suggested change

val packageName = classDeclaration.packageName.asString()

val companionClassName = "${classDeclaration.simpleName.asString()}_MlKitCompanion"

val packageName = classDeclaration.packageName.asString()

val companionClassName = "${classDeclaration.simpleName.asString()}_MlKitCompanion"

val propertyDocs = extractPropertyKdocs(classDeclaration.docString ?: "")

gemini-code-assist · 2026-07-28T20:11:58Z

+    val generableAnn =
+      classDeclaration.annotations.firstOrNull { it.shortName.getShortName() == "Generable" }
+    val classDesc = getStringFromAnnotation(generableAnn, "description")


Class descriptions documented via KDocs are not propagated to the generated companion class's @Generable annotation. Since ML Kit's on-device structured output engine relies on these annotations on the companion class to build the schema, any KDoc-based class descriptions will be lost on-device. We should fall back to the base KDoc description if the annotation description is missing.

Suggested change

val generableAnn =

classDeclaration.annotations.firstOrNull { it.shortName.getShortName() == "Generable" }

val classDesc = getStringFromAnnotation(generableAnn, "description")

val generableAnn =

classDeclaration.annotations.firstOrNull { it.shortName.getShortName() == "Generable" }

val classDesc = getStringFromAnnotation(generableAnn, "description") ?: extractBaseKdoc(classDeclaration.docString ?: "")

gemini-code-assist · 2026-07-28T20:11:58Z

+      val guideAnn = property.annotations.firstOrNull { it.shortName.getShortName() == "Guide" }
+      if (guideAnn != null) {
+        val guideValues =
+          getGuideValuesFromAnnotation(guideAnn, getStringFromAnnotation(guideAnn, "description"))


Use the extracted property KDocs as a fallback description for the @Guide annotation on the companion class property, and ensure the @Guide annotation is generated if a description exists even if the original property lacked a @Guide annotation.

Suggested change

val guideAnn = property.annotations.firstOrNull { it.shortName.getShortName() == "Guide" }

if (guideAnn != null) {

val guideValues =

getGuideValuesFromAnnotation(guideAnn, getStringFromAnnotation(guideAnn, "description"))

val guideAnn = property.annotations.firstOrNull { it.shortName.getShortName() == "Guide" }

val propDescription = getStringFromAnnotation(guideAnn, "description") ?: propertyDocs[propName]

if (guideAnn != null || !propDescription.isNullOrEmpty()) {

val guideValues =

getGuideValuesFromAnnotation(guideAnn, propDescription)

initial implementation

a0ffcd5

milaGGL added 8 commits July 6, 2026 16:09

remove objectResponse from Candidate

64ef2ef

update naming, the on-device GenerateObjectResponse, and ObjectSource

35e128a

update GenerateObjectResponse

60b3c12

update the name of shadowClass property

8d670e8

update Generable

565be05

remove local dependency

b12b96d

remove debug log

e4fc45a

Merge branch 'main' into mila-support-structured-ouput-locally

f98bd5c

update api txt file

e3dbf21

milaGGL marked this pull request as ready for review July 28, 2026 14:25

milaGGL changed the title ~~initial implementation~~ [AI] support on-device structured output Jul 28, 2026

milaGGL added 2 commits July 28, 2026 12:52

Fix ConvertersTest by removing brittle ML Kit default assertions

7e1f92b

Update SchemaSymbolProcessorVisitor.kt

cda79ef

gemini-code-assist Bot reviewed Jul 28, 2026

View reviewed changes

Comment thread ai-logic/firebase-ai/src/main/kotlin/com/google/firebase/ai/type/GenerateObjectResponse.kt Outdated

Comment thread ...-ai-ksp-processor/src/main/kotlin/com/google/firebase/ai/ksp/SchemaSymbolProcessorVisitor.kt Outdated

Address code review feedback, add dependency on genai-schema

1944371

firebase deleted a comment from google-oss-bot Jul 28, 2026

Merge branch 'main' into mila-support-structured-ouput-locally

2981a63

gemini-code-assist Bot reviewed Jul 28, 2026

View reviewed changes

		val packageName = classDeclaration.packageName.asString()
		val companionClassName = "${classDeclaration.simpleName.asString()}_MlKitCompanion"

Uh oh!

Conversation

milaGGL commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 6, 2026

📝 PRs merging into main branch

Uh oh!

google-oss-bot commented Jul 6, 2026

Uh oh!

gemini-code-assist Bot commented Jul 6, 2026

Uh oh!

google-oss-bot commented Jul 27, 2026

Uh oh!

milaGGL commented Jul 28, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

milaGGL commented Jul 28, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

milaGGL commented Jul 6, 2026 •

edited

Loading