-
Notifications
You must be signed in to change notification settings - Fork 100
Open
Labels
api: spannerIssues related to the googleapis/python-spanner API.Issues related to the googleapis/python-spanner API.
Description
After upgrading to google-cloud-spanner v3.60.0+, the built-in OpenTelemetry metrics feature (enabled by default) fails to export metrics to Cloud Monitoring with the following error:
InvalidArgument: One or more TimeSeries could not be written:
timeSeries[N] (metric.type="spanner.googleapis.com/internal/client/operation_latencies", ...):
the set of resource labels is incomplete, missing (instance_id)
Impact: This error does not affect Spanner operations (queries succeed with HTTP 200), but it:
- Generates significant log noise
- Floods error tracking systems (e.g., Sentry) with thousands of errors
- Prevents built-in metrics from being collected in Cloud Monitoring
Environment
- google-cloud-spanner version: 3.62.0 (also affects 3.60.0, 3.61.0)
- Python version: 3.13.8
- OS: Linux (Cloud Run)
- Previous working version: 3.59.x and earlier (metrics were opt-in)
Minimal Reproduction
from google.cloud import spanner
# With google-cloud-spanner >= 3.60.0, built-in metrics are enabled by default
client = spanner.Client(project="my-project")
instance = client.instance("my-instance")
database = instance.database("my-database")
with database.snapshot() as snapshot:
results = snapshot.execute_sql("SELECT 1")
list(results) # Execute query
# After ~60 seconds (metrics export interval), the following error appears in logs:
# google.api_core.exceptions.InvalidArgument: ... missing (instance_id)Expected Behavior
All required resource labels, including instance_id, are set, and the error does not occur.
Actual Behavior
Metrics export fails with InvalidArgument error. The resource labels in the error message show:
resource.labels={"location": "asia-northeast1", "instance_config": "unknown", "client_hash": "XXXXXX"}
Stack Trace
File "opentelemetry/sdk/metrics/_internal/export/__init__.py", line 550, in _receive_metrics
self._exporter.export(
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 357, in export
self._batch_write(time_series_list, timeout_millis)
File "google/cloud/spanner_v1/metrics/metrics_exporter.py", line 154, in _batch_write
self.client.create_service_time_series(
File "google/cloud/monitoring_v3/services/metric_service/client.py", line 1937, in create_service_time_series
rpc(
File "google/api_core/gapic_v1/method.py", line 131, in __call__
return wrapped_func(*args, **kwargs)
File "google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: One or more TimeSeries could not be written:
... the set of resource labels is incomplete, missing (instance_id)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
api: spannerIssues related to the googleapis/python-spanner API.Issues related to the googleapis/python-spanner API.