Throttling
On this page
Intent
Ensure that a given client is not able to access service resources more than the assigned limit.
Explanation
Real-world example
A young human and an old dwarf walk into a bar. They start ordering beers from the bartender. The bartender immediately sees that the young human shouldn’t consume too many drinks too fast and refuses to serve if enough time has not passed. For the old dwarf, the serving rate can be higher.
In plain words
Throttling pattern is used to rate-limit access to a resource.
Control the consumption of resources used by an instance of an application, an individual tenant, or an entire service. This can allow the system to continue to function and meet service level agreements, even when an increase in demand places an extreme load on resources.
Programmatic Example
BarCustomer class presents the clients of the Bartender API. CallsCount tracks the number of
calls per BarCustomer.
1public class BarCustomer {
2
3 @Getter
4 private final String name;
5 @Getter
6 private final int allowedCallsPerSecond;
7
8 public BarCustomer(String name, int allowedCallsPerSecond, CallsCount callsCount) {
9 if (allowedCallsPerSecond < 0) {
10 throw new InvalidParameterException("Number of calls less than 0 not allowed");
11 }
12 this.name = name;
13 this.allowedCallsPerSecond = allowedCallsPerSecond;
14 callsCount.addTenant(name);
15 }
16}
17
18@Slf4j
19public final class CallsCount {
20 private final Map<String, AtomicLong> tenantCallsCount = new ConcurrentHashMap<>();
21
22 public void addTenant(String tenantName) {
23 tenantCallsCount.putIfAbsent(tenantName, new AtomicLong(0));
24 }
25
26 public void incrementCount(String tenantName) {
27 tenantCallsCount.get(tenantName).incrementAndGet();
28 }
29
30 public long getCount(String tenantName) {
31 return tenantCallsCount.get(tenantName).get();
32 }
33
34 public void reset() {
35 tenantCallsCount.replaceAll((k, v) -> new AtomicLong(0));
36 LOGGER.info("reset counters");
37 }
38}
Next, the service that the tenants are calling is introduced. To track the call count, a throttler timer is used.
1public interface Throttler {
2
3 void start();
4}
5
6public class ThrottleTimerImpl implements Throttler {
7
8 private final int throttlePeriod;
9 private final CallsCount callsCount;
10
11 public ThrottleTimerImpl(int throttlePeriod, CallsCount callsCount) {
12 this.throttlePeriod = throttlePeriod;
13 this.callsCount = callsCount;
14 }
15
16 @Override
17 public void start() {
18 new Timer(true).schedule(new TimerTask() {
19 @Override
20 public void run() {
21 callsCount.reset();
22 }
23 }, 0, throttlePeriod);
24 }
25}
Bartender offers the orderDrink service to the BarCustomers. The customers probably don’t
know that the beer serving rate is limited by their appearances.
1class Bartender {
2
3 private static final Logger LOGGER = LoggerFactory.getLogger(Bartender.class);
4 private final CallsCount callsCount;
5
6 public Bartender(Throttler timer, CallsCount callsCount) {
7 this.callsCount = callsCount;
8 timer.start();
9 }
10
11 public int orderDrink(BarCustomer barCustomer) {
12 var tenantName = barCustomer.getName();
13 var count = callsCount.getCount(tenantName);
14 if (count >= barCustomer.getAllowedCallsPerSecond()) {
15 LOGGER.error("I'm sorry {}, you've had enough for today!", tenantName);
16 return -1;
17 }
18 callsCount.incrementCount(tenantName);
19 LOGGER.debug("Serving beer to {} : [{} consumed] ", barCustomer.getName(), count+1);
20 return getRandomCustomerId();
21 }
22
23 private int getRandomCustomerId() {
24 return ThreadLocalRandom.current().nextInt(1, 10000);
25 }
26}
Now it is possible to see the full example in action. BarCustomer young human is rate-limited to 2
calls per second and the old dwarf to 4.
1public static void main(String[] args) {
2 var callsCount = new CallsCount();
3 var human = new BarCustomer("young human", 2, callsCount);
4 var dwarf = new BarCustomer("dwarf soldier", 4, callsCount);
5
6 var executorService = Executors.newFixedThreadPool(2);
7
8 executorService.execute(() -> makeServiceCalls(human, callsCount));
9 executorService.execute(() -> makeServiceCalls(dwarf, callsCount));
10
11 executorService.shutdown();
12 try {
13 executorService.awaitTermination(10, TimeUnit.SECONDS);
14 } catch (InterruptedException e) {
15 LOGGER.error("Executor service terminated: {}", e.getMessage());
16 }
17}
18
19private static void makeServiceCalls(BarCustomer barCustomer, CallsCount callsCount) {
20 var timer = new ThrottleTimerImpl(1000, callsCount);
21 var service = new Bartender(timer, callsCount);
22 // Sleep is introduced to keep the output in check and easy to view and analyze the results.
23 IntStream.range(0, 50).forEach(i -> {
24 service.orderDrink(barCustomer);
25 try {
26 Thread.sleep(100);
27 } catch (InterruptedException e) {
28 LOGGER.error("Thread interrupted: {}", e.getMessage());
29 }
30 });
31}
An excerpt from the example’s console output:
18:46:36.218 [Timer-0] INFO com.iluwatar.throttling.CallsCount - reset counters
18:46:36.218 [Timer-1] INFO com.iluwatar.throttling.CallsCount - reset counters
18:46:36.242 [pool-1-thread-2] DEBUG com.iluwatar.throttling.Bartender - Serving beer to dwarf soldier : [1 consumed]
18:46:36.242 [pool-1-thread-1] DEBUG com.iluwatar.throttling.Bartender - Serving beer to young human : [1 consumed]
18:46:36.342 [pool-1-thread-2] DEBUG com.iluwatar.throttling.Bartender - Serving beer to dwarf soldier : [2 consumed]
18:46:36.342 [pool-1-thread-1] DEBUG com.iluwatar.throttling.Bartender - Serving beer to young human : [2 consumed]
18:46:36.443 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:36.443 [pool-1-thread-2] DEBUG com.iluwatar.throttling.Bartender - Serving beer to dwarf soldier : [3 consumed]
18:46:36.544 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:36.544 [pool-1-thread-2] DEBUG com.iluwatar.throttling.Bartender - Serving beer to dwarf soldier : [4 consumed]
18:46:36.645 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
18:46:36.645 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:36.745 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:36.745 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
18:46:36.846 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:36.846 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
18:46:36.947 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
18:46:36.947 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:37.048 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
18:46:37.048 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:37.148 [pool-1-thread-1] ERROR com.iluwatar.throttling.Bartender - I'm sorry young human, you've had enough for today!
18:46:37.148 [pool-1-thread-2] ERROR com.iluwatar.throttling.Bartender - I'm sorry dwarf soldier, you've had enough for today!
Class diagram
Applicability
The Throttling pattern should be used:
- When service access needs to be restricted not to have high impact on the performance of the service.
- When multiple clients are consuming the same service resources and restriction has to be made according to the usage per client.