Skip to content

InvalidArgument error: "resource labels is incomplete, missing (instance_id)" when exporting built-in metrics #1481

@shutootaki

Description

@shutootaki

After upgrading to google-cloud-spanner v3.60.0+, the built-in OpenTelemetry metrics feature (enabled by default) fails to export metrics to Cloud Monitoring with the following error:

InvalidArgument: One or more TimeSeries could not be written:
  timeSeries[N] (metric.type="spanner.googleapis.com/internal/client/operation_latencies", ...):
  the set of resource labels is incomplete, missing (instance_id)

Impact: This error does not affect Spanner operations (queries succeed with HTTP 200), but it:

  • Generates significant log noise
  • Floods error tracking systems (e.g., Sentry) with thousands of errors
  • Prevents built-in metrics from being collected in Cloud Monitoring

Environment

  • google-cloud-spanner version: 3.62.0 (also affects 3.60.0, 3.61.0)
  • Python version: 3.13.8
  • OS: Linux (Cloud Run)
  • Previous working version: 3.59.x and earlier (metrics were opt-in)

Minimal Reproduction

from google.cloud import spanner

# With google-cloud-spanner >= 3.60.0, built-in metrics are enabled by default
client = spanner.Client(project="my-project")
instance = client.instance("my-instance")
database = instance.database("my-database")

with database.snapshot() as snapshot:
    results = snapshot.execute_sql("SELECT 1")
    list(results)  # Execute query

# After ~60 seconds (metrics export interval), the following error appears in logs:
# google.api_core.exceptions.InvalidArgument: ... missing (instance_id)

Expected Behavior

All required resource labels, including instance_id, are set, and the error does not occur.


Actual Behavior

Metrics export fails with InvalidArgument error. The resource labels in the error message show:

resource.labels={"location": "asia-northeast1", "instance_config": "unknown", "client_hash": "XXXXXX"}

Stack Trace

File "opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
    self._exporter.export(
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 357, in export
    self._batch_write(time_series_list, timeout_millis)
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 154, in _batch_write
    self.client.create_service_time_series(
File "google/cloud/monitoring_v3/services/metric_service/client.py", line 1937, in create_service_time_series
    rpc(
File "google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
File "google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: One or more TimeSeries could not be written:
    ... the set of resource labels is incomplete, missing (instance_id)

Metadata

Metadata

Assignees

Labels

api: spannerIssues related to the googleapis/python-spanner API.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions