Had a 3 year old one this week. A loop that builds a list of messages to send to a queue for another service to consume then it calls BatchPublish.
Only Batch Publish was inside the loop so instead of sending n messages, it sends 1+2+3… +n
We never noticed before because n was never more then 100 and the consuming service is idempotent so the duplicate messages don’t cause issues. I think it’s (n(n-1))/2. So n=100 is 4950. That’s only 4 minutes work. Also that code only runs at 1am.
Recently n has been hitting 1000 which produces 499500 messages and it takes a few hours to clear and triggers an alarm for delayed processing.
Had a 3 year old one this week. A loop that builds a list of messages to send to a queue for another service to consume then it calls BatchPublish.
Only Batch Publish was inside the loop so instead of sending n messages, it sends 1+2+3… +n
We never noticed before because n was never more then 100 and the consuming service is idempotent so the duplicate messages don’t cause issues. I think it’s (n(n-1))/2. So n=100 is 4950. That’s only 4 minutes work. Also that code only runs at 1am.
Recently n has been hitting 1000 which produces 499500 messages and it takes a few hours to clear and triggers an alarm for delayed processing.