Broker

[sindhushreem25]

We have pulsar 2.10.x running in production .
We have around 3 broker running ,230 topics and 16 bundles .
seeing high latency on the pulsar producer stats like 3000ms to 7000ms.
broker metrics shows the pulsar write latency to be less then 5ms. producers are not hitting max pending messages and batch size is 3 msg .Sharing the produce stats below :
2025-07-04 12:40:36.832 [pulsar-timer-6-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://public/default/-partition-1] [-175-224582] Pending messages: 15
--- Publish throughput: 66.55 msg/s --- 0.49 Mbit/s
--- Latency: med: 3000.000 ms - 95pct: 4741.000 ms - 99pct: 5997.000 ms - 99.9pct: 6000.000 ms - max: 6404.000 ms
--- BatchSize: med: 2.000 - 95pct: 65.000 - 99pct: 68.000 - 99.9pct: 69.000 - max: 69.000
--- MsgSize: med: 1930.000 bytes - 95pct: 64713.000 bytes - 99pct: 65368.000 bytes - 99.9pct: 65465.000 bytes - max: 65465.000 bytes
--- Ack received rate: 69.33 ack/s
--- Failed messages: 0 --- Pending messages: 15

[persistent://public/default/-partition-0] [-175-224582] Pending messages: 88
--- Publish throughput: 103.66 msg/s --- 0.76 Mbit/s
--- Latency: med: 2735.000 ms - 95pct: 5846.000 ms - 99pct: 5998.000 ms - 99.9pct: 6753.000 ms - max: 6753.000 ms
--- BatchSize: med: 3.000 - 95pct: 59.000 - 99pct: 69.000 - 99.9pct: 69.000 - max: 69.000
--- MsgSize: med: 3066.000 bytes - 95pct: 57746.000 bytes - 99pct: 65327.000 bytes - 99.9pct: 65510.000 bytes - max: 65510.000 bytes
--- Ack received rate: 110.80 ack/s
--- Failed messages: 0 --- Pending messages: 88

2025-07-04 12:40:36.833 [pulsar-timer-6-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [persistent://public/default/-partition-2] [-175-224582] Pending messages: 0
--- Publish throughput: 105.25 msg/s --- 0.78 Mbit/s
--- Latency: med: 4.000 ms - 95pct: 252.000 ms - 99pct: 261.000 ms - 99.9pct: 430.000 ms - max: 961.000 ms
--- BatchSize: med: 3.000 - 95pct: 63.000 - 99pct: 69.000 - 99.9pct: 69.000 - max: 69.000
--- MsgSize: med: 2878.000 bytes - 95pct: 59276.000 bytes - 99pct: 65241.000 bytes - 99.9pct: 65469.000 bytes - max: 65528.000 bytes
--- Ack received rate: 105.25 ack/s
--- Failed messages: 0 --- Pending messages: 0

Can anyone help on what could be the issue for high latency on producer side .