BigQueryReadClient::ReadRows fails with UNKNOWN: Error occurred when fetching oauth2 token after CreateReadSession succeeds

[dharanad]

Client google-cloud-cpp v2.22.0

What Happened

Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: BigQuery ReadRows failed for stream projects/rising-woods-374602/locations/us/sessions/CAISDG80QTJWdmc5UnpMSRoCamMaAm55/streams/CM8RGgJqYxoCbnkoAg: UNKNOWN: Error occurred when fetching oauth2 token. (rows delivered by this data source before the failure: 0)

I want to understand the failure mode here.

1. Under what conditions does this occur?
2. Is this a client side or service side issue ? how do I confirm which one it is?

Is UNKNOWN error type retried by the default BigQueryReadConnection retry policy? If a transient token fetch failure lands as UNKNOWN, it looks like it would bypass retry and kill the read outright. Is that expected, and is there a supported way to make token-fetch failures retryable ?

[hamedrabah]

This failure is happening on the client side, before the BigQuery Storage service sees the ReadRows RPC. In v2.22.0, BigQueryReadAuthDecorator::ReadRows calls auth_->ConfigureContext(*context); a token-fetch failure is returned as a streaming RPC error before the child client is invoked.

The default BigQuery Read retry traits only retry UNAVAILABLE, so an UNKNOWN status from credential acquisition is treated as permanent. I would check which ADC source the process selected, then test connectivity from that same process/container to its token source (metadata server or OAuth endpoint), including DNS, proxy, CA bundle, and clock.

I would also upgrade from v2.22.0 before adding custom policy. A custom BigQueryReadRetryPolicyOption can retry this narrowly, but treating every UNKNOWN as transient is risky. If you add an outer retry, limit it to establishing the stream before any row has been delivered.

[ChangkeunJ]

That string isn't ours. At v2.22.0, git grep "fetching oauth2 token" over the whole tree returns nothing. It's gRPC core, in src/core/lib/security/credentials/oauth2/oauth2_credentials.cc:

    } else {
      auto err = GRPC_ERROR_CREATE_REFERENCING(
          "Error occurred when fetching oauth2 token.", &error, 1);
      pending_request->result = grpc_error_to_absl_status(err);
    }

and the macro hardcodes the code:

#define GRPC_ERROR_CREATE_REFERENCING(desc, errs, count)                      \
  grpc_status_create(absl::StatusCode::kUnknown, desc, DEBUG_LOCATION, count, \
                     errs)

UNKNOWN there isn't a mapping of the underlying failure, it's a literal. That's the whole reason it doesn't arrive as UNAVAILABLE. That exact text also pins your gRPC at 1.66 or older, since the site is gone in 1.67. For reference, v2.22.0's Bazel pin is gRPC 1.62.0, and the code above is identical in 1.60 through 1.66.

@hamedrabah is right that it's client side, but the mechanism is a little different and the difference matters for debugging. If you're on default ADC, the auth decorator isn't in your stub stack at all. From bigquery_read_stub_factory.cc:

  if (auth->RequiresConfigureContext()) {
    stub = std::make_shared<BigQueryReadAuth>(std::move(auth), std::move(stub));
  }

and unified_grpc_credentials.cc maps default ADC to

    void visit(GoogleDefaultCredentialsConfig const&) override {
      result = std::make_unique<GrpcChannelCredentialsAuthentication>(
          grpc::GoogleDefaultCredentials());
    }

whose RequiresConfigureContext() returns false. And PopulateGrpcOptions sets UnifiedCredentialsOption to MakeGoogleDefaultCredentials() when you haven't set one, so that is the default path. ConfigureContext never gets called and this library does no token fetching there. ExternalAccountConfig lands on the same strategy. Only ServiceAccountConfig and ImpersonateServiceAccountConfig install strategies that fetch tokens inside google-cloud-cpp, and those surface our own error text, not this string. So seeing the gRPC string already tells you which side of the boundary you're on.

Conditions: gRPC caches the token and only refetches when it's within GRPC_SECURE_TOKEN_REFRESH_THRESHOLD_SECS, which is 60, of expiry, and it gives the fetch itself a deadline of Now() + 60s. So CreateReadSession succeeding proves nothing about the next call, it can have been served entirely from cache. I wouldn't commit to a single story for why ReadRows is the one that fetches. Cache expiry is one. For a multi-stream read the one I'd bet on is a burst of concurrent ReadRows calls all hitting the metadata server at once and getting throttled. Either way it's the fetch failing, not the RPC.

The reason you can't tell those apart from the status is that on this gRPC a transport failure and a non-200 collapse into the same code and the same text, since parse_server_response also returns GRPC_CREDENTIALS_ERROR for response->status != 200.

Your logs do separate them, and it's one or the other, never both. Transport failures go through

  GRPC_LOG_IF_ERROR("oauth_fetch", error);

Non-200s do not, because grpc_log_if_error is error.ok() ? true : grpc_log_error(...), and on a 403 the HTTP exchange itself completed so error is OK. Those log this instead, from inside parse_server_response:

  if (response->status != 200) {
    gpr_log(GPR_ERROR, "Call to http server ended with error %d [%s].",
            response->status,
            null_terminated_body != nullptr ? null_terminated_body : "");

Both are GPR_ERROR and gRPC's default min_severity_to_print is GPR_LOG_SEVERITY_ERROR, so both print without enabling anything. Grep stderr for oauth_fetch and for Call to http server. A hit on the second gives you the HTTP status and the response body, which means the token endpoint answered and you're looking at identity, permissions or throttling. A hit on the first means DNS, proxy, egress, CA bundle, or the 60s fetch deadline. Either way the BigQuery Storage backend never saw the RPC.

Another free discriminator: if your ADC resolves to a service account JSON key, gRPC's google default creds build self-signed JWTs via grpc_service_account_jwt_access_credentials_create_from_auth_json_key and never contact a token endpoint, so this error can't come from there. make_default_call_creds tries the ADC file before it ever probes the metadata server, so getting this means you're on GCE/GKE metadata, a gcloud auth application-default login refresh token, or an external account, which derives from grpc_oauth2_token_fetcher_credentials.

On retry, the default is BigQueryReadLimitedTimeRetryPolicy(std::chrono::minutes(30)) and the traits are:

  static inline bool IsPermanentFailure(google::cloud::Status const& status) {
    return status.code() != StatusCode::kOk &&
           status.code() != StatusCode::kUnavailable;
  }

so yes, UNKNOWN is permanent. But there's a wrinkle in the resume loop that explains why your message says 0 rows specifically. From resumable_streaming_read_rpc.h:

    while (!retry_policy->IsExhausted() &&
           (has_received_data_ || retry_policy->OnFailure(last_status))) {

has_received_data_ short-circuits OnFailure. Once a single response has been delivered the stream resumes on any status code, UNKNOWN included. It's only the very first Read() where the code is consulted, and there OnFailure(UNKNOWN) is false, so you get the status straight back with no backoff and no attempt. This isn't a general "UNKNOWN kills reads" problem, it's specifically fatal at stream start.

Resumption itself is offset-based, so retrying is safe. The whole updater is:

void BigQueryReadReadRowsStreamingUpdater(
    ::google::cloud::bigquery::storage::v1::ReadRowsResponse const& response,
    ::google::cloud::bigquery::storage::v1::ReadRowsRequest& request) {
  request.set_offset(request.offset() + response.row_count());
}

Supported way to widen it, no internal headers needed:

namespace bqs = ::google::cloud::bigquery_storage_v1;
using ::google::cloud::Status;
using ::google::cloud::StatusCode;

class RetryUnknownPolicy : public bqs::BigQueryReadRetryPolicy {
 public:
  explicit RetryUnknownPolicy(int maximum_failures)
      : maximum_failures_(maximum_failures) {}

  bool IsPermanentFailure(Status const& s) const override {
    auto const c = s.code();
    return c != StatusCode::kOk && c != StatusCode::kUnavailable &&
           c != StatusCode::kUnknown;
  }
  bool OnFailure(Status const& s) override {
    if (IsPermanentFailure(s)) return false;
    ++failures_;
    return !IsExhausted();
  }
  bool IsExhausted() const override { return failures_ > maximum_failures_; }
  std::unique_ptr<bqs::BigQueryReadRetryPolicy> clone() const override {
    return std::make_unique<RetryUnknownPolicy>(maximum_failures_);
  }

 private:
  int maximum_failures_;
  int failures_ = 0;
};

That's the same shape as TraitBasedRetryPolicy plus LimitedErrorCountRetryPolicy, which is what the stock policies are built on, just with UNKNOWN added to the transient set. clone() returning a fresh instance is load-bearing: retry_policy() in bigquery_read_connection_impl.cc clones the option, and ResumableStreamingReadRpc::Read clones the prototype again every time it enters the retry loop, relying on the state being reset.

Scope it to the one call rather than the connection, ReadRows has an overload with a trailing Options:

auto rows = client.ReadRows(
    request, google::cloud::Options{}
                 .set<bqs::BigQueryReadRetryPolicyOption>(
                     RetryUnknownPolicy(3).clone()));

Set it connection-wide instead and CreateReadSession and SplitReadStream start retrying UNKNOWN too, since they read the same option, and UNKNOWN from gRPC's transport and TLS layers covers plenty you don't want to retry. Worth knowing too that on this gRPC a genuinely permanent auth failure, a 403 from the metadata server or a revoked key, produces this identical status and text, so a time-based policy will burn its whole budget on an identity that is never going to work. A small error-count policy is safer than widening the 30-minute default.

A newer gRPC helps, but less than you'd hope, so I wouldn't skip the policy on that basis. 1.67 did replace the error site, and a non-200 or unparseable response now becomes absl::UnavailableError("error parsing oauth2 token"), which the stock traits retry. But the transport-error branch there is just

    if (!error.ok()) {
      self->on_done_(std::move(error));
      return;
    }

and httpcli.cc builds those errors with GRPC_ERROR_CREATE, which is hardcoded kUnknown the same way. So connect failures, TLS failures and the fetch deadline still land as UNKNOWN on 1.67 through 1.76. The deliberate mapping first appears in a release in 1.78.0:

  if (!error.ok()) {
    // TODO(roth): It shouldn't be necessary to explicitly set the
    // status to UNAVAILABLE here.  Once the HTTP client code is
    // migrated to stop using legacy grpc_error APIs to create
    // statuses, we should be able to just propagate the status as-is.
    self->on_done_(absl::UnavailableError(StatusToString(error)));
    return;
  }
  if (self->response_.status != 200) {
    grpc_status_code status_code =
        grpc_http2_status_to_grpc_status(self->response_.status);
    if (status_code != GRPC_STATUS_UNAVAILABLE) {
      status_code = GRPC_STATUS_UNAUTHENTICATED;
    }

So check what you actually link against, because google-cloud-cpp's own pin is behind that. v2.22.0 pins 1.62.0 and even v3.8.0 pins 1.74.1. Bumping google-cloud-cpp alone won't get you the deliberate mapping.

Last thing, the Error Source: RUNTIME / INVALID_STATE / rows delivered by this data source framing isn't from google-cloud-cpp, none of those strings are in the tree. Worth confirming whatever wrapper produced it isn't layering its own retry semantics on top of the ones above.