How to Use the Time Limiter > FocusMe Documentation

Leaky Bucket

The following image perfectly illustrates the leaky bucket algorithm.

The image shows the usage of leaky bucket algorith (Image credit: [1])

The image shows the usage of leaky bucket algorithm in traffic shaping. If we map it our limiting requests to server use case, water drops from the faucets are the requests, the bucket is the request queue and the water drops leaked from the bucket are the responses. Just as the water dropping to a full bucket will overflow, the requests arrive after the queue becomes full will be rejected.

As we can see, in the leaky bucket algorithm, the requests are processed at an approximately constant rate, which smooths out bursts of requests. Even though the incoming requests can be bursty, the outgoing responses are always at a same rate.

A simple implementation for demonstration purpose:

public class LeakyBucket extends RateLimiter { private long nextAllowedTime; private final long REQUEST_INTERVAL_MILLIS; protected LeakyBucket(int maxRequestPerSec) { super(maxRequestPerSec); REQUEST_INTERVAL_MILLIS = 1000 / maxRequestPerSec; nextAllowedTime = System.currentTimeMillis(); } @Override boolean allow() { long curTime = System.currentTimeMillis(); synchronized (this) { if (curTime >= nextAllowedTime) { nextAllowedTime = curTime + REQUEST_INTERVAL_MILLIS; return true; } return false; } } }

Video

Configuration

The following example creates two different rate limiters for an API service, to enforce different levels of service (free or paid):

  • YAML
  • XML
  • PHP

1 2 3 4 5 6 7 8 9 10 11 12 # config/packages/rate_limiter.yaml framework: rate_limiter: anonymous_api: # use ‘sliding_window’ if you prefer that policy policy: ‘fixed_window’ limit: 100 interval: ’60 minutes’ authenticated_api: policy: ‘token_bucket’ limit: 5000 rate: { interval: ’15 minutes’, amount: 500 }

Note

The value of the interval option must be a number followed by any of the units accepted by the PHP date relative formats (e.g. 3 seconds, 10 hours, 1 day, etc.)

In the anonymous_api limiter, after making the first HTTP request, you can make up to 100 requests in the next 60 minutes. After that time, the counter resets and you have another 100 requests for the following 60 minutes.

In the authenticated_api limiter, after making the first HTTP request you are allowed to make up to 5,000 HTTP requests in total, and this number grows at a rate of another 500 requests every 15 minutes. If you don’t make that number of requests, the unused ones don’t accumulate (the limit option prevents that number from being higher than 5,000).

Gotchas and Good Practices When Implementing Time Limiting

Usually, we deal with two kinds of operations – queries (or reads) and commands (or writes). It is safe to time-limit queries because we know that they don’t change the state of the system. The searchFlights() operation we saw was an example of a query operation.

Commands usually change the state of the system. A bookFlights() operation would be an example of a command. When time-limiting a command we have to keep in mind that the command is most likely still running when we timeout. A TimeoutException on a bookFlights() call for example doesn’t necessarily mean that the command failed.

We need to manage the user experience in such cases – perhaps on timeout, we can notify the user that the operation is taking longer than we expected. We can then query the upstream to check the status of the operation and notify the user later.

Web

Visit family.microsoft.com. Sign into your Family Safety account. Find your family member’s name and click More options > Screen time. Select Apps and games tab. Turn on App and game limits toggle. Click the app or game you want to set limits on. Set the amount of time your family member can spend on the app or game each day and when they’re allowed to use it. Use the same limits every day or tailor a schedule for each day of the week.

Using the Resilience4j TimeLimiter Module

TimeLimiterRegistry, TimeLimiterConfig, and TimeLimiter are the main abstractions in resilience4j-timelimiter.

TimeLimiterRegistry is a factory for creating and managing TimeLimiter objects.

TimeLimiterConfig encapsulates the timeoutDuration and cancelRunningFuture configurations. Each TimeLimiter object is associated with a TimeLimiterConfig.

TimeLimiter provides helper methods to create or execute decorators for Future and CompletableFuture Suppliers.

Let’s see how to use the various features available in the TimeLimiter module. We will use the same example as the previous articles in this series. Assume that we are building a website for an airline to allow its customers to search for and book flights. Our service talks to a remote service encapsulated by the class FlightSearchService.

The first step is to create a TimeLimiterConfig:

This creates a TimeLimiterConfig with default values for timeoutDuration (1000ms) and cancelRunningFuture (true).

Let’s say we want to set a timeout value of 2s instead of the default:

We then create a TimeLimiter:

We want to asynchronously call FlightSearchService.searchFlights() which returns a List<Flight>. Let’s express this as a Supplier<CompletionStage<List<Flight>>>:

We can then decorate the Supplier using the TimeLimiter:

Finally, let’s call the decorated asynchronous operation:

Here’s sample output for a successful flight search that took less than the 2s timeoutDuration we specified:

And this is sample output for a flight search that timed out:

The timestamps and thread names above show that the calling thread got a TimeoutException even as the asynchronous operation completed later on the other thread.

We would use decorateCompletionStage() if we wanted to create a decorator and re-use it at a different place in the codebase. If we want to create it and immediately execute the Supplier<CompletionStage>, we can use executeCompletionStage() instance method instead:

Sliding Window

Sliding window counter is similar to fixed window counter but it smooths out bursts of traffic by adding a weighted count in previous window to the count in current window. For example, suppose the limit is 10 per minute. There are 9 requests in window [00:00, 00:01) and 5 reqeuests in window [00:01, 00:02). For a requst arrives at 00:01:15, which is at 25% position of window [00:01, 00:02), we calculate the request count by the formula: 9 x (1 - 25%) + 5 = 11.75 > 10. Thus we reject this request. Even though both windows don’t exceed the limit, the request is rejected because the weighted sum of previous and current window does exceed the limit.

This is still not accurate becasue it assumes that

This is still not accurate becasue it assumes that the distribution of requests in previous window is even, which may not be true. But compares to fixed window counter, which only guarantees rate within each window, and sliding window log, which has huge memory footprint, sliding window is more practical.

A simple implementation for demonstration purpose:

import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.atomic.AtomicInteger; public class SlidingWindow extends RateLimiter { // TODO: Clean up stale entries private final ConcurrentMap<Long, AtomicInteger> windows = new ConcurrentHashMap<>(); protected SlidingWindow(int maxRequestPerSec) { super(maxRequestPerSec); } @Override boolean allow() { long curTime = System.currentTimeMillis(); long curWindowKey = curTime / 1000 * 1000; windows.putIfAbsent(curWindowKey, new AtomicInteger(0)); long preWindowKey = curWindowKey 1000; AtomicInteger preCount = windows.get(preWindowKey); if (preCount == null) { return windows.get(curWindowKey).incrementAndGet() <= maxRequestPerSec; } double preWeight = 1 (curTime curWindowKey) / 1000.0; long count = (long) (preCount.get() * preWeight + windows.get(curWindowKey).incrementAndGet()); return count <= maxRequestPerSec; } }

In this article we have learned several rate limiting algorithms and their simple implementations. In next post, we will analyze how rate limiting is implemented in Google guava library.

[1] Leaky Bucket & Tocken Bucket – Traffic shaping [2] How to Design a Scalable Rate Limiting Algorithm [3] An alternative approach to rate limiting [4] Better Rate Limiting With Redis Sorted Sets

Tags

Leave a Reply

Your email address will not be published. Required fields are marked *

Adblock
detector
Go up