MockMotor
Vladimir Dyuzhev
MockMotor Creator
Load Testing: How to Keep the Request Rate Constant When the Code Does Graceful Degradation
CPU usage increases. Graceful degradation turns off some logic. RT decreases. The load generator increases the request rate. CPU usage increases. ...
When the load generating tool doesn’t have “constant rate” mode, and only allows setting the number of threads and pauses between the requests, the actual request rate can become times more than what we aim at.
Imagine the code has a graceful degradation feature based on CPU usage. A high load hits the application, causing a CPU usage increase.
The graceful degradation feature turns off some features, causing a reduction in RT.
The reduction in RT means the generator sends more requests per minute.
For example, if a generator thread has 1-second pause between requests, and the average RT is 4 seconds, the thread will send 60/(4+1) = 12 requests per minute. But if the RT drops to 3s due to graceful degradation, the thread will send 60/(3+1) = 15 requests per minute. That is 25% over expected volume!
More requests per minute means more CPU usage and further graceful degradation.
This loop repeats until the degradation reaches its maximum. By that time, the inbound volume can be times more than you planned.
Solution: Proxy the Load Traffic via MockMotor Service
Proxy the traffic via a MockMotor service that forwards it to the application.
Set an RT in the forward reaction that is below the expected average.
If the service RT is above the configured delay, that delay is not in effect and does not affect the service statistics.
However, when degradation kicks in and the RT falls below the delay configured in the MM reaction, MM adds a delay to maintain the RT at the configured level.
The RT never goes too low, and the request rate never goes too high.