[dynamic control] Add pipeline config initialization from file

[jackshirazi]

Description:

Adding a method to read a pipeline initialization file in yaml (preferred) or json format.

Existing Issue(s):

#2546

Testing:

Added

Documentation:

Added

Outstanding items:

Wiring up the policy pipeline:

• Generic: message -> provider -> policy -> policy handler -> implementer
• eg change sampling rate message -> OpampPolicyProvider -> TraceSamplingRatePolicy -> PolicyStore -> TraceSamplingRatePolicyImplementer

Steps needed for the wiring:

• configuring the pipeline
  • DONE in this PR
• providers reading policies, eg OpampPolicyProvider, FilePolicyProvider, etc
  • DONE: OpampPolicyProvider
• implementers applying policies, eg TraceSamplingRatePolicyImplementer
  • DONE: TraceSamplingRatePolicyImplementer
• policy structures (eg TraceSamplingRatePolicy) that the provider converts messages into
  • DONE: TraceSamplingRatePolicy
• registering config to policy structures (eg "trace_sampling_rate_policy" registered to TraceSamplingRatePolicy)
• initializing policy classes (eg TraceSamplingRatePolicy needs to install a custom sampler)
  • DONE TraceSamplingRatePolicy initialization
• activate configured providers (eg start OpampPolicyprovider reading from it's source)
• register implementers for policies (eg a new TraceSamplingRatePolicy is applied by a TraceSamplingRatePolicyImplementer)
• link the provider to processing policies and applying implementers

[Copilot]

Pull request overview

Adds support in the dynamic-control module to initialize the policy pipeline configuration from an external YAML (preferred) or JSON file, selected via OpenTelemetry ConfigProperties.

Changes:

• Added PolicyInitConfig.readFromConfigProperties(ConfigProperties) to load init configuration from a YAML/JSON file path configured via properties.
• Added unit tests covering YAML-vs-JSON precedence, fallback when YAML property is blank, and failure scenarios (missing/unparseable files).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
dynamic-control/src/main/java/io/opentelemetry/contrib/dynamic/policy/registry/PolicyInitConfig.java	Adds config-property driven file loading (YAML preferred, JSON supported) with logging on failure.
dynamic-control/src/test/java/io/opentelemetry/contrib/dynamic/policy/registry/PolicyInitConfigTest.java	Adds coverage for selection logic and error handling behavior.

[trask]

can we use declarative config api instead? (and rely on the java agent declarative config bridge for the system property support)

[jackshirazi]

I'm not sure that gives any benefit. The system property doesn't provide the pipeline configuration, it points to a file that provides pipeline configuration. We'd need to change the property to be an ugly string encoded yaml to use that. json would work because that can be a single string for the full config, but yaml would be ugly.

jsonconfig='{"sources": [{"kind": "opamp", "format": "jsonkeyvalue", "location": "elastic", "mappings": [ { "sourceKey": "sampling_rate", "policyType": "trace_sampling_rate_policy" }, ] } ]}'

yamlconfig=sources:\n  - kind: opamp\n    format: jsonkeyvalue\n    location: elastic\n    mappings:\n      - sourceKey: sampling_rate\n        policyType: trace_sampling_rate_policy

Even to define a small pipeline this is painful

The yaml is deliberately setup so that when the telemetry policy otep is accepted and we then add policy to declarative config, it slots right in, eg

policy:
 sources:
   - kind: opamp
     format: jsonkeyvalue
     location: elastic
     mappings:
       - sourceKey: sampling_rate
         policyType: trace_sampling_rate_policy

The node under policy is exactly what this is supporting. To use the declarative config api just to get the file path seems pointless?

[trask]

The yaml is deliberately setup so that when the telemetry policy otep is accepted and we then add policy to declarative config, it slots right in, eg

policy:
 sources:
   - kind: opamp
     format: jsonkeyvalue
     location: elastic
     mappings:
       - sourceKey: sampling_rate
         policyType: trace_sampling_rate_policy

does it make sense to slot it in now, even if the location under declarative config changes later when it gets an official location?

[jackshirazi]

We can do that for the declarative config, since this is experimental it would be fine. But how would you define the config when declarative config is not used.

List<DeclarativeConfigProperties> sources = config.getStructuredList("sources");
if (sources == null) {//true when not using declarative config
  //how to get the structure? read a file? use the ugly stringified file?

[trask]

could this be a declarative config only feature? (similar to rule based sampling, some of the new method instrumentation, ..)

[jackshirazi]

Allowing the structure to be read by failing over to how this PR does it would work. So it would look like this. I need to add the additional methods if this plan looks okay

PolicyInitConfig policyPipeline;
List<DeclarativeConfigProperties> sources = config.getStructuredList("sources");
if (sources == null) {//true when not using declarative config
  policyPipeline = PolicyInitConfig.readFromConfigProperties();
} else {
  policyPipeline = convertToPipeline(sources);
}

[jackshirazi]

I've done a different implementation in #2766 (I'll close this PR when this discussion is complete or shifts to that).

I think having only declarative config supported imposes a very high burden here. You already need to include this as an extension, insisting that you must also wire everything up using declarative config in order to use this makes it a huge shift instead of the smaller one I'd like to support

[trask]

I think having only declarative config supported imposes a very high burden here.

yeah, I understand. at the same time, the whole reason we (in Java at least) have been looking forward to declarative configuration is that it allows us to have a unified story for these kinds of complex configurations, instead of having lots of different config files that are pointed to by various configuration properties (e.g. having separate metric view yaml configuration file, rule-based sampling configuration file, jmx configuration file, opamp policy configuration file, methods instrumentation configuration file, etc etc..)

[jackshirazi]

I've made the declarative config the default in the new PR, with support for experimental config files only if declarative config is not used.

Without the config file failover, it just forces deployments that don't use declarative config to use another coded extension to create the pipeline which seems like imposing a lot of additional pain to avoid including an option and a method