optable-targeting new features: non-blocking early network call + hid + id5

[justadreamer]

Type of changes

• module update

What's the context?

Two related improvements to the optable-targeting module: the Optable API call no longer blocks the auction pipeline, and the Targeting API request now carries the parameters needed for mobile in-app resolution (hid, bundle/ver, ID5).

Latency: non-blocking early network call

The API call starts at raw-auction-request and runs in parallel with the rest of the auction. The result is consumed per bidder at bidder-request, where enrichment is actually injected, so the pipeline never waits on the network.

Stored requests and stored imps are merged after raw-auction-request, so for requests whose bidders or site/app live there (f.e. Prebid Mobile SDK traffic) the call is started by processed-auction-request on the merged request instead. Bidders are read from both imp.ext.prebid.bidder and the old-style imp.ext.<bidder>.

Traffic controls

• enrichment-percentage / bidder-enrichment-percentages: sampling rate (0-100) applied per bidder. A bidder with no entry falls back to the account default; 0 disables it entirely.
• enrich-web / enrich-app: select which traffic sources are enriched.

Sampling applies to the split plan only. The legacy processed-auction-request hook enriches the whole bid request before bidders split out, so it cannot sample per bidder.

Targeting API request parameters

• hid-prefixes (new config entry): comma-separated id prefixes to emit as hid=, in the given order (e.g. "e,p,i6,a,g,c"). Independent of id-prefix-order, and selective rather than merely ordering: only the listed prefixes are sent as hid.
• bundle / ver: emitted from bidRequest.app when present.
• id5_signature in: read from user.ext.optable, forwarded as a Targeting API query attribute alongside email/phone/zip/vid.
• id5_signature out: returned in ext.prebid.passthrough.optable.id5_signature. PBS cannot persist the signature between requests, so it round-trips through the client, which caches it and replays it on the next auction.

Backwards compatibility

No breaking changes. The processed-auction-request hook stays in the plan. When the raw-auction-request and bidder-request hooks are present it no longer blocks and only starts the call for stored-request traffic; otherwise it keeps the previous synchronous behavior.

Configuration (recommended)

hooks:
  modules:
    optable-targeting:
      enrichment-percentage: 100
      bidder-enrichment-percentages:
        appnexus: 75
        pubmatic: 0
      enrich-web: true
      enrich-app: true
      id-prefix-order: "e,v,c"
      hid-prefixes: "a,c,i6"

  execution-plan:
    endpoints:
      "/openrtb2/auction":
        stages:
          raw-auction-request:
            groups:
              - timeout: 50
                hook-sequence:
                  - module-code: "optable-targeting"
                    hook-impl-code: "optable-targeting-raw-auction-request-hook"
          processed-auction-request:
            groups:
              - timeout: 50
                hook-sequence:
                  - module-code: "optable-targeting"
                    hook-impl-code: "optable-targeting-processed-auction-request-hook"
          bidder-request:
            groups:
              - timeout: 500
                hook-sequence:
                  - module-code: "optable-targeting"
                    hook-impl-code: "optable-targeting-bidder-request-hook"
          auction-response:
            groups:
              - timeout: 10
                hook-sequence:
                  - module-code: "optable-targeting"
                    hook-impl-code: "optable-targeting-auction-response-hook"

The bidder-request group timeout bounds the wait on the API call, so it must accommodate round-trip latency to the Optable edge. The raw-auction-request and processed-auction-request timeouts only cover dispatch.

To migrate from the legacy plan, keep the processed-auction-request hook and add the raw-auction-request and bidder-request hooks.

Test plan

• Unit tests
• Semi-automated integration tests against the live Optable edge, covering enrichment with eids, per-bidder sampling (including a bidder pinned at 0%), ad server targeting keywords, ID5 signature passthrough, and the legacy execution plan

Quality check

• Are your changes following our code style guidelines?
• Are there any breaking changes in your code? No
• Does your test coverage exceed 90%?
• Are there any erroneous console logs, debuggers or leftover code in your changes? No

cc: @jrosendahl-opt @softcoder594

[Lightwood13]

Wrong class for logger, should be OptableTargetingFlowResolver instead of OptableTargetingProcessedAuctionRequestHook.

[Lightwood13]

moduleContext.shouldSkipEnrichment is only set to true in the hasRawAuctionRequestHook. If that hook has been executed, than this code is unreachable because of the if (moduleContext.isEarlyNetworkCallEnabled()) check in the OptableTargetingProcessedAuctionRequestHook.call(). Therefore this if statement is redundant and can be removed.

[Lightwood13]

This function can be inlined.

[Lightwood13]

Please put callTargetingAPITimestamp near optableTargetingExecutionTime to keep related fields together for better readability. I also recommend renaming it, so it's clear that it's a start timestamp for targeting (and not, for example, an end timestamp). For example, optableTargetingStartTimestamp.

[Lightwood13]

This field is always set to System.currentTimeMillis() - moduleContext.getCallTargetingAPITimestamp(). Please create a helper method recordTargetingExecutionTime that will set it to this value and update all the usages. failWithExecutionTime can be updated to failAndRecordExecutionTime with no parameters and OptableTargetingFlowResolver.calcAPICallExecutionTime can be removed.

[Lightwood13]

Add .filter(it -> it.length > 0), otherwise an exception will be thrown if array is empty after splitting.

[Lightwood13]

The cleanRequestOnFail boolean flag makes logic complicated and harder to read. First of all, it's not only about cleaning the request or not - in hasStoredRequestOrImps if this flag is true, it skips the whole validation. Looking at usages, this flag is passed as false in raw auction request hook and true in processed auction request hook, so it's a way to distinguish between different behaviors in those two hooks. This flag is currently only used in the validation at the start of this function:

if (!isTrafficSourceValid(moduleContext, properties, cleanRequestOnFail, bidRequest)
        || hasStoredRequestOrImps(moduleContext, cleanRequestOnFail, bidRequest)) {
    return noEnrichment(cleanRequestOnFail, moduleContext);
}

Instead of using a boolean flag and creating complicated logic, I think it's better to move this validation out of this function directly into the hooks that call it. This introduces a little bit of code duplication, but the code is much easier to read and understand what is happening in each branch.

Example implementation

Raw auction request hook:

if (!PropertiesValidator.isTrafficSourceValid(bidRequest, properties)
        || hasStoredRequest(bidRequest)
        || hasStoredImps(bidRequest)) {

    moduleContext.setEarlyCallInitializationCompleted(false);
    return success(moduleContext);
}

return optableTargetingFlowResolver.resolveAsyncOptableTargetingFlow(
        moduleContext, payload, invocationContext, properties);

Proccessed auction request hook:

if (!PropertiesValidator.isTrafficSourceValid(auctionRequestPayload.bidRequest(), properties)) {
    moduleContext.setShouldSkipEnrichment(true);
    moduleContext.setEarlyCallInitializationCompleted(false);
    return update(AuctionRequestCleaner.instance(), moduleContext);
}

return optableTargetingFlowResolver.resolveAsyncOptableTargetingFlow(
        moduleContext, auctionRequestPayload, invocationContext, properties);

After this refactoring, helper functions containing this boolean flag are inlined and no longer necessary and can be removed, namely hasStoredRequestOrImps, isTrafficSourceValid and noEnrichment.

[Lightwood13]

Please apply masking to user and device before extracting sensitive data (like IP or user agent). This code is now being called from raw auction request hook, before masking is applied, so it has to be applied here manually. Example code: https://docs.prebid.org/prebid-server/developers/add-a-module-java.html#interaction-with-activity-infrastructure

[Lightwood13]

In the current implementation, the processed auction request hook is used to handle both the old sync flow and the new async flow, which complicates its implementation. Please split it into two separate hooks - one for the old flow and one for the new flow. Old users can keep using the old legacy hook without any changes in the config, while the new async flow requires to use the new hook in the execution plan.

This way, there won’t be any checks to try to understand which flow we’re currently executing - each of the hooks will handle exactly one flow, sync or async respectively. if (moduleContext.isEarlyNetworkCallEnabled()) can then be removed - if branch goes into the new hook, else branch goes into the legacy hook. You can then also simplify OptableTargetingFlowResolver.resolveOptableTargetingFlow, removing checks for the presence of the raw auction request hook. That function will only be called in the legacy sync flow, which doesn't care about the raw auction request hook.