Repository navigation
[Bug] The global topic policy config override the local topic policy config #21202
Description
Search before asking
- I searched in the issues and found nothing similar.
Version
2.10.5
Minimal reproduce step
-
create a Topic
./pulsar-admin --admin-url http://127.0.0.1:8881 topics create-partitioned-topic -p 2 persistent://geo/geo/geo_3 -
set local topic policy (This problem exists in all configurations, max-message-size is used here to demonstrate)
./pulsar-admin --admin-url http://127.0.0.1:8880 topicPolicies set-max-message-size persistent://geo/geo/geo_3 -m 10000 -
set global topic policy
./pulsar-admin --admin-url http://127.0.0.1:8880 topicPolicies set-max-message-size persistent://geo/geo/geo_3 -g -m 10 -
check the topic policy
./pulsar-admin --admin-url http://127.0.0.1:8880 topicPolicies get-max-message-size persistent://geo/geo/geo_3
result: 10000./pulsar-admin --admin-url http://127.0.0.1:8880 topicPolicies get-max-message-size persistent://geo/geo/geo_3 -g
result: 10 -
verify the configuration to take effect
./pulsar-client --url pulsar://127.0.0.1:6650 produce -m 0123456780123456789 persistent://geo/geo/geo_3
What did you expect to see?
2023-09-18T20:25:03,301+0800 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced
What did you see instead?
2023-09-18T20:25:33,934+0800 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Exceed maximum message size
org.apache.pulsar.client.api.PulsarClientException$NotAllowedException: Exceed maximum message size
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:1048) ~[org.apache.pulsar-pulsar-client-api-2.10.4.jar:2.10.4]
at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:90) ~[org.apache.pulsar-pulsar-client-original-2.10.4.jar:2.10.4]
at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:309) ~[org.apache.pulsar-pulsar-client-tools-2.10.4.jar:2.10.4]
at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:241) ~[org.apache.pulsar-pulsar-client-tools-2.10.4.jar:2.10.4]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:184) ~[org.apache.pulsar-pulsar-client-tools-2.10.4.jar:2.10.4]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:220) ~[org.apache.pulsar-pulsar-client-tools-2.10.4.jar:2.10.4]
2023-09-18T20:25:33,937+0800 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced
Anything else?
as PIP-92 described:
After adding the global topic policy, the topic applied priority is:
Local cluster topic > policy Global topic policy > Namespace policy > Broker default configuration
the global topic policy config shouldn't override the local topic policy config.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
[-][Bug] the global topicPolicies config override the local topicPolicies config[/-][+][Bug] the global topic policy config override the local topic policy config[/+][-][Bug] the global topic policy config override the local topic policy config[/-][+][Bug] The global topic policy config override the local topic policy config[/+]mattisonchao commentedon Sep 20, 2023
It should be a known issue.
I guess the root cause should be global and local topic policies using the same key. it will be compacted by the compactor.
Could you please share the idea of fixing it?
chenhongSZ commentedon Sep 20, 2023
@mattisonchao I think there could be multiple issues here。
In my test case, the error was caused by the local topic policy and global topic policy using the same variable in the
PolicyHierarchyValueclass (topicValue). It should be divided into two variables.github-actions commentedon Oct 21, 2023
The issue had no activity for 30 days, mark with Stale label.
merlimat commentedon Oct 24, 2026
Thanks @chenhongSZ for the report and the analysis. This was fixed by #24254:
PolicyHierarchyValuenow keeps the local and global topic-level values separately (topicValueandtopicGlobalValue), and the local topic policy takes precedence over the global one, as PIP-92 describes. #24286 also fixed local and global topic policies overwriting each other after compaction of the__change_eventstopic.I'm closing this as fixed. If it still happens on a currently supported release (4.0.x, 4.2.x or 5.0.x), please reopen with the steps.