Let’s start by defining what Rate Limiter actually is.

A rate limiter is a tool that the developers eventually will need in their web facing application. It does not matter if it’s a commercial application or an open source app because you need to protect it from abusive use or seasons with exceptionally high traffic.

For example your application may experience an attack called DDoS(Distributed Denial of Service) which is that one or more users are sending a large number of request to your application in a short period of time and as a result your application may cease to be available to legitimate users. Here is where a rate limiter is very useful because you can add rules that detect this and block those malicious requests, examples of rules could be checking by: IP address, location, JWT token or other attributes.

Another situation which is NOT an attack but it is very possible to occur is in high traffic sites that for example in some seasons like Christmas or Thanks Giving the traffic may be higher than expected and the infrastructure is not being able to handle it, in this case we may need to implement a rate limiter to reject requests when our infrastructure reached the threshold that we defined, allowing more requests might compromise our infrastructure.

By now we already covered what a rate limiter is and described a couple use cases, but obviously there might be more scenarios because as everything in software development it depends on your use case.

For this article we are going to focus in a possible solution when the infrastructure is overloaded, the scenario will be this one:

The rate limiter is going to be activated when the CPU usage of our spring-boot-solr-movies-search-service is over 80% in the last 30 seconds.

In order to generate high traffic to the spring boot application we are going to use Apache JMeter which is a tool to create load tests. At the end of this article you will find a section called Rate Limiter in Action which includes a link to the github repository that has all the code.

Let’s start for the high level view of the system that we are going to use in this article:

Openresty

It is a web platform that run a NGINIX server but with some additional features such as enhaced version of LuajIT, one of the cool things is that allows us to manipulate the request prior to reaching our services.

We use this container as our API Gateway, if you are not familiar with an API Gateway it is basically the entrypoint to your API, in this step is where you normally validate if the incoming request should reach or not the backend service.

In the example that we are reviewing in this article the api gateway validates if denies access to the backend service and returns a HTTP 503 response code or allow the request to reach the backend service.

The gateway decides to ALLOW or REJECT by checking if Valkey has this key: spring_boot_solr_movies_search_service_high_cpu_usage as true.

Here is a code snippet with what I just described.

            access_by_lua_block {
                local valkey = require "resty.redis";
                local val = valkey:new()

                val:set_timeout(1000)

                local ok, err = val:connect("valkey.dns.podman",6379)

                if not ok then
                    ngx.status = ngx.HTTP_INTERNAL_SERVER_ERROR
                    ngx.say('{"error": "Error while trying to connect to valkey: ' .. err .. '"}')
                    return ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
                end

                local context_path = ngx.var.uri

                local route_map = {}
                    route_map["/api/movies"] = "http://spring_boot_mongodb_movies_service_pool"
                    route_map["/api/movies/search"] = "http://spring_boot_solr_movies_search_service_pool"
                    route_map["/api/movies/semantic-search"] = "http://spring_boot_solr_movies_search_service_pool"
                    route_map["/api/movies/recommendations"] = "http://spring_boot_solr_movies_search_service_pool"


                local matched_backend_service = route_map[context_path]

                if matched_backend_service then

                     local valkey_keys_map = {}
                     valkey_keys_map["/api/movies"] = "spring_boot_mongodb_movies_service_high_cpu_usage"
                     valkey_keys_map["/api/movies/search"] = "spring_boot_solr_movies_search_service_high_cpu_usage"
                     valkey_keys_map["/api/movies/semantic-search"] = "spring_boot_solr_movies_search_service_high_cpu_usage"
                     valkey_keys_map["/api/movies/recommendations"] = "spring_boot_solr_movies_search_service_high_cpu_usage"


                    local matched_valkey_key = valkey_keys_map[context_path]

                    local res, err = val:get(matched_valkey_key)

                    if res and res ~= ngx.null and res == "true" then
                        ngx.status = 503
                        ngx.say('{"error": "Service Unavailable: ' .. context_path .. '"}')
                        val:set_keepalive(1000,200)
                        return ngx.exit(503)
                    end

                    ngx.var.target_backend = matched_backend_service

                else
                    ngx.status = 404
                    ngx.say('{"error": "The following path: ' .. context_path .. ' not found in API Gateway"}')
                    val:set_keepalive(1000,200)
                    return ngx.exit(404)
                end

                val:set_keepalive(10000,200)
            }

Script file path: spring-boot-examples-pb/spring-boot-search-platform-example/openresty/nginx.conf.

As you can see in the code snippet basically what we do is create a valkey instance so Openresty can connect to valkey and retrieve the keys based on the context path.

You may wonder why do I choose Openresty and not simply go with Spring Cloud Gateway. Here is why:

• Memory Usage: Here Openresty is superior to the Spring Cloud Gateway. Spring Cloud Gateway requires the JVM which introduces heavy memory usage, on the other side Openresty runs on top NGINIX and LuaJIT which does not consume a lot of memory.

spring-boot-solr-movies-search-service

This is the spring boot application that is used as the backend service. The application provides the following endpoints:

• /api/movies/search
• /api/movies/semantic-search
• /api/movies/recommendations

The endpoints are very self explanatory, I will not make a detailed description of those endpoints because they are out of the scope of this article, but if you are interested in terms such as semantic search, RAG Systems. I have created some articles about that, the link to those documents are in the previous listed endpoints.

prometheus-podman-exporter

Basically this container is going to help us to collect the metrics from the other containers that are being used in the docker-compose.yml file.

It is important to mention that we are running podman rootless, because we are following the least privilege principle, it is a security principle that states that you only grant the minimum permissions needed to a user, program to perform a specific task.

The metrics collected here will be used by prometheus. In order to allow podman-exporter collect the metrics we need to make the following configurations:

   podman-exporter:
    container_name: podman-exporter
    image: quay.io/navidys/prometheus-podman-exporter:v2.0.0
    security_opt:
      - "label=type:container_runtime_t"
    userns_mode: keep-id
    ports:
      - "9882:9882"
    environment:
      - CONTAINER_HOST=unix:///run/user/501/podman/podman.sock
      - HOME=/tmp
    volumes:
      - /run/user/501/podman/podman.sock:/run/user/501/podman/podman.sock:ro
    networks:
      - search_network

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/docker-compose.yaml

The critical configurations are these ones:

• security_opt
• userns_mode
• volumes
• environment

All four together are what allow podman-exporter securely access the host’s rootless podman socket.

One more thing here if you take a look to the environment and the volumes they are using the user 501, this change from system to system in order to know the user in your computer, you can run the following command:

        podman system connection list

The output of this command will let you know if you are running podman in rootless mode. The rootless mode uses the core user.

valkey

It is the open source version of REDIS. Valkey is a high performance key value datastore which is great for distributed environments. In our use case we only have one instance of spring-boot-solr-movies-search-service but in real production microservices environments you will normally have several instances.

Prometheus/alertManager

I will explain these 2 components together because they work very close. Prometheus acts as our metrics hub, it will pull the metrics from the different components and will define the alerts using something called promQL which is a query language that allows you to work with the metrics.

For example here we can see the alert associated to the data from the podman-exporter

- name: podman-exporter
 rules:
   - alert: spring_boot_solr_movies_search_service_high_cpu_usage
 expr: sum(rate(podman_container_cpu_seconds_total[30s])) by (id) * 100 * on(id) podman_container_info{name="spring-boot-search-platform-example-spring-boot-solr-movies-search-service-1"} > 80
 for: 30s
 labels:
   container_stats: spring_boot_solr_movies_search_service_cpu_usage
   instance: '{{ $labels.instance }}'
   severity: critical
 annotations:
   summary: "spring-boot-solr-movies-search-service CPU usage higher than 80%"
   description: "The instance *{{ $labels.instance }}* CPU usage has been higher than 80% for more than 30 seconds"

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/prometheus/alert.yml

The alert will be triggered if the CPU usage from the spring-boot-solr-movies-search-service is higher than 80% in the last 30 seconds, this alert will be send to the alertManager and will decide how to handle it.

Here are the configurations used by the alertmanager

global:
 smtp_smarthost: 'mailpit:1025'
 smtp_from: 'search-platform@example.com'
 smtp_require_tls: false

route:
 group_by: ['service','severity']
 group_wait: 5s
 group_interval: 30s
 repeat_interval: 4h
 receiver: 'email-receiver'

 routes:
   - receiver: 'webhook-receiver'
     matchers:
       - container_stats =~ "spring_boot_mongodb_movies_service_cpu_usage|spring_boot_solr_movies_search_service_cpu_usage"
templates:
 - '/etc/alertmanager/templates/*.tmpl'

receivers:
 - name: 'email-receiver'
   email_configs:
     - to: 'support-team@example.com'
       send_resolved: true
       require_tls: false
       headers:
         subject: '[{{ .Status | toUpper }}] Service Alert: {{ .CommonLabels.alertname }}'
       html: '{{ template "email.custom" . }}'
 - name: 'webhook-receiver'
   webhook_configs:
     - url: 'http://rate-limit-service:5050/api/alerts'
       send_resolved: true

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/prometheus-alert-manager/alertmanager.yml

For this alert there are 2 receivers

• Email
• Webhook

This is great for real PRODUCTION scenarios because it notifies the support team but also activates the rate limiter through the webhook, basically what the webhook does is that it sends an http request to this endpoint: http://rate-limit-service:5050/api/alerts, which is a spring boot application.

This interaction between the prometheus/alertmanager using the webhook to notify the rate-limit-service applicaiton is the well-known event driven approach. This approach is by far more efficient than the method polling data approach.

If you are interested in learning more about the process of receiving the alerts by email I wrote the following article:

From Metrics to Alerts: Monitoring Spring Boot App with Prometheus and Alertmanager

MailPit

It is an email testing tool which is great for this case given that we are doing local testing in this article, for other environments such as DEV, QA and PROD you will need another approach like a real SMTP server.

rate-limit-service

This is a normal spring boot REST API that uses this maven dependency:

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-redis-reactive</artifactId>
</dependency>

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/rate-limit-service/pom.xml

This maven dependency has the tools that we need to communicate with Valkey by creating ReactiveRedistTemplate as we can see here

@Bean
public ReactiveRedisTemplate<String, Boolean> reactiveRedisTemplate(ReactiveRedisConnectionFactory reactiveRedisConnectionFactory) {
   StringRedisSerializer keySerializer = new StringRedisSerializer();
   GenericToStringSerializer valueSerializer = new GenericToStringSerializer<>(Boolean.class);




   RedisSerializationContext serializationContext = RedisSerializationContext
           .<String, Boolean>newSerializationContext()
           .key(RedisSerializationContext.SerializationPair.fromSerializer(keySerializer))
           .value(RedisSerializationContext.SerializationPair.fromSerializer(valueSerializer))
           .hashKey(RedisSerializationContext.SerializationPair.fromSerializer(keySerializer))
           .hashValue(RedisSerializationContext.SerializationPair.fromSerializer(keySerializer))
           .build();


   return new ReactiveRedisTemplate<>(reactiveRedisConnectionFactory, serializationContext);
}

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/rate-limit-service/src/main/java/org/learning/spring/rate_limit_service/config/ValkeyConfig.java

The application in the AlertService class will process the alerts that it receives from the Prometheus/Alert Manager and will create the rate limit key(s) in valkey using the alertName as key.

Here is how the key is created in valkey:

    public void handleAlert(PrometheusAlertRequestBody prometheusAlertRequestBody) {


   prometheusAlertRequestBody.getAlerts().forEach(alert -> {
       if (alert.getStatus().equals("firing")) {


           Boolean rateLimitEnabled = reactiveRedisTemplate.opsForValue().set(alert.getLabels().getAlertName(), true, Duration.ofMinutes(4))
                   .block();


          if (rateLimitEnabled != null && rateLimitEnabled){
              LOGGER.info("Enable rate limit for: {}", alert.getLabels().getAlertName());
          }else{
              LOGGER.error("Error while trying to activate the rate limit for alert {}", alert.getLabels().getAlertName());
          }
       } else if (alert.getStatus().equals("resolved")) {
           LOGGER.info("Deactivate rate limit for alert: {}", alert.getLabels().getAlertName());
       }
   });


}

File Path: spring-boot-examples-pb/spring-boot-search-platform-example/rate-limit-service/src/main/java/org/learning/spring/rate_limit_service/service/AlertService.java

We can see that if the alert is in firing status it is going to create the rate limit key with TTL(Time to live) of 4 Minutes, this means that during 4 minutes Openresty will respond with HTTP 503 codes blocking the access to spring-boot-solr-movies-search-service.

As a separate note if you are curious about the OpenAPI specification for the alerManager you can find it here:

https://prometheus.io/docs/alerting/latest/alerts_api/

Apache JMeter

As mentioned earlier in this article JMeter it is a load test tool that we will use to simulate 1000 concurrent users progressively during 5 minutes. Here you can see the configurations:

You can find the load test file here:

Apache JMeter will send many http requests to this endpoint:

http://openresty:80/api/movies/search

HTTP Method: POST

HTTP Request Body:

{
  "query": "Future",
  "filters": [
    {
      "field": "genres",
      "operator": "IN",
      "values": ["Comedy","Drama"]
    },
    {
      "field": "title",
      "operator": "EQUALS",
      "value": "Future"
    },
    {
      "field": "tomatoes.critic.rating",
      "operator": "RANGE",
      "from": "7",
      "to": "8"
    }
  ]
}

Grafana

It is a visualization tool that is used in this system to help us see the traffic and CPU usage associated with the spring-boot-solr-movies-search-service application. If you want to know more about the configuration needed for grafana and how to create dashboards you can take a look to the following article that I created:

From Metrics to Dashboards: Monitoring Spring Boot App with Prometheus and Grafana

Rate Limiter In Action

We are going to see how we can test this.

1. Open a terminal and clone this project:

https://github.com/mpedroza-mx/spring-boot-examples-pb

2. Navigate to this folder: spring-boot-search-platform-example and execute the following command:

     podman compose --profile MONITORING --profile=LOAD_TEST  up --build -d

3. Once all containers are up and running you can check the spring-boot-solr-movies-search-service CPU and traffic and see the rate limit in action, for this you will need to connect to prometheus, grafana and mailpit. You should see something like this:

In the previous image we can see that there was a peak in the traffic and CPU usage but then at some point it ended abruptly that it is when the rate limiter was activated. We can also see that prometheus triggered the alert and mailpit received the notification email.

If you want to see this locally you can access these urls:

• Grafana: http://localhost:3000
• Prometheus: http://localhost:9090/alerts
• MailPit: http://localhost:8025

Note: If you are asked for a user and password to connect to your local grafana use admin/admin

4. When the load test is completed we now can check the index.html file in the following path:

The report should look something like this:

We can see that there are a lot of errors because Openresty rejected all the requests after the CPU usage for the spring-boot-solr-movies-search-service reached 80%.

Conclusion

• We learned in this article that there is more than one reason why our applications may need a rate limiter.
• We learned that as part of a good rate limit strategy we need to apply observability features in order to know what is happening with the application.
• We learned one of the best strategies to react to specific events, and that it is using webhooks which allows us to follow the event-driven approach.

I hope that this article give you some ideas for your use case or you find it interesting

👋 Thanks for reading!

Tools used in this tutorial:

• macOs 26.5.2
• openjdk 21.0.10
• Spring Boot 4.0.0.
• mongod v8.0.26
• Apache Solr 10.0.0
• podman version 5.8.1
• Docker Compose version 5.3.1
• Grafana 13.0
• prometheus v3.13.1
• alertmanager v0.33.1
• podman-exporter v2.0.0
• mailpit v1.30
• jmeter 5
• valkey 9.1.2
• openresty 1.31-alpine

Computer specs used with this application

• Apple M4 Pro
• Memory 24GB

Note: The configurations and code shown in this example are intended for learning and local development, not for production environments.