Pricing Docs Blog See Demo Download
Vladimir Dyuzhev, author of MockMotor

Vladimir Dyuzhev
MockMotor Creator

How to Record HTTP/JSON Traffic with MockMotor

MockMotor can act as a MITM to record and mock the traffic.

Tweet This

MockMotor saves you from tedious manual work.

One of the ways it does that is the automatic creation of mocks for passing traffic.

What is Forward Functionality?

A forward mock response is not producing a reply payload on its own. Instead, it forwards the request to an external URL, i.e. to a real backend.

In the request flow, MockMotor forwards URI, parameters, payload and headers down to the backend. That makes most authentications schemes (e.g. Basic HTTP or JWT) work without any special configuration.

In the response flow, MockMotor sends the response payload and response headers provided by the backend, to the caller.

Why Forwarding?

Forwarding greatly enlarges what you can do to test your applications while reducing the required efforts. Essentially, you only need to spend time mocking the responses you're focusing on, and you leave the rest to the live implementation.

  • You can mock only selected few operations, relegating the rest to the real backend.
  • You can modify the responses before sending them back to the consumer (e.g. adding new fields).
  • You can record the traffic for later use as a pure mock service (i.e. without forwarding).

Let's see how we can do the recording.

Record Functionality

Forward responses, by default, only send the traffic to a live backend.

However, when the Record is enabled, the response is not only sent back to the consumer but is also recorded as a fully configured mock.

Here the request from the client is forwarded to the Yandex API. When the reply is received, it is saved as a mock response that matches the original request, and then the response is sent to the client. Next time a request with the same parameters comes, it is served from the recorded mock response and is not forwarded to live Yandex API.

The recorded mock uses as many of the request parameters as possible to provide the exact match: SOAPAction or first element's name, Relative URI, parameters in the URI, names and values of HTTP query parameters. The mock also saves the response values: HTTP Status, payload, response time, content-type.

Recording Yandex Weather API

The Yandex Weather API is widely used by applications that deal with weather in Russia and its neighbours. Its input is an HTTP GET request with two query parameters: lat for latitude and lon for longitude:

https://api.weather.yandex.ru/v1/forecast?lat=59.939015&lon=30.315804

The response contains a massive amount of information about the weather in the requested geographical point, in JSON format:

{
   "now_dt": "2020-08-03T18:49:09.355Z",
   "fact":    {
      "polar": false,
      "accum_prec":       {
         "1": 3.7,
         "3": 4.2,
         "7": 23.3
      },
      "temp": 19,
      "icon": "ovc",
      "pressure_mm": 756,
      "wind_dir": "w",
      "source": "station",
      "feels_like": 21,
      "wind_gust": 6.3,
      "condition": "overcast",
      "temp_water": 16,
      "uv_index": 0,
      "pressure_pa": 1008,
      "humidity": 87,
      "season": "summer",
      "wind_speed": 1,
      "soil_moisture": 0.2,
      "daytime": "n",
      "soil_temp": 18,
      "obs_time": 1596479524
   },
   "now": 1596480549,
   "info":    {
      "nr": true,
      "ns": true,
      "f": true,
      "def_pressure_mm": 759,
      "_h": false,
      "lon": 30.315804,
      "nsr": true,
      "n": true,
      "tzinfo":       {
...
I consider this an HTTP service and not a REST one. For a modern truly honestly REST service I'd expect the parameters to be in the URI segments, like this:

https://api.weather.yandex.ru/v1/forecast/lat/59.939015/lon/30.315804

However, this distinction is debatable. We're not here to have an ivory-tower discussion, so let's just mock what we've got, ok?

Working Around the Limits

In the test mode, Yandex API only allows 5000 requests a day. It seems like a lot, but if you generate maps with weather markers, 5000 requests can all be used up before lunch.

The solution is, of course, to mock the responses.

The number of locations used in test scenarios is limited. We can mock them all and then execute tens of thousands of requests - without any scowl from Yandex.

Let's Be Lazy

Let's create the mocks the most effortless way - make MockMotor record the responses for us.

Create a Service

First, we need to create a test service that mocks the Yandex Weather API. I just did that:

The mock service's relative URI is /yandex-weather.

Create a Forward Response

A forward mock response is a special mock response that contains no own response payload. Instead, it is configured to forward the request to the live backend.

How a forwarding mock should look?

  • It should match any HTTP operation.
  • It should use Javascript (because the response payload is JSON).
  • It should have the Yandex API URL set in the Forward field.
  • It should have Record enabled.
In the forwarding response, we shouldn't have any response payload, HTTP status, content-type, attachments or delay set. They are used for altering the response properties, so clear those fields.

Below is the forward response. Unrelated properties are hidden:

Note the Forward URL field that contains the target system hostname. Anything in the request path after the mock service URI is appended to the forward URL. That includes the relative path and query parameters.

E.g. a client calling the mock service with this URL:

https://127.0.0.1:20075/yandex-weather/v1/forecast?&lat=59.939015&lon=30.315804

MockMotor then removes the mock service URL (https://127.0.0.1:20075/yandex-weather) to figure out the relative URL:

/v1/forecast?&lat=59.939015&lon=30.315804

Then MockMotor appends this relative URI to the forward URL (https://api.weather.yandex.ru), and calls Yandex with this:

https://api.weather.yandex.ru/v1/forecast?lat=59.939015&lon=30.315804

At this point, the brand new service has the only response - the forwarding one:

Execute a Request for Weather

Now, let's execute our first request.

Since we do not have any local mocks yet, the request will be handled by the forwarding mock. It sends the request to the actual Yandex backend. When Yandex responds, MockMotor records the response properties as a new mock.

Let's find the weather in St. Petersburg. This city is my childhood home, and also a home of Russian Tsars, the birthplace of the 1917 revolution and the location of lots of other interesting things, but it is often rainy there, so it worth checking.

The latitude and longitude of St. Petersburg is 59.939012,30.315804. (To be precise, this is a location of the Palace Square - the very heart of SPb).

GET http://127.0.0.1:7080/ExamplesForBlog/yandex-weather/v1/forecast?&lat=59.939012&lon=30.315804 HTTP/1.1
Accept-Encoding: gzip,deflate
X-Yandex-API-Key: 3ae58b34-....-....-....-da111b3d6fe7
Host: 127.0.0.1:7080
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.1.1 (java 1.5)

Oh great, we've got a response! Lots of data in it (2 degrees Celsius - cold, as usual):

{
   "now": 1553914074,
   "now_dt": "2019-03-30T02:47:54.740Z",
   "info":    {
      "f": true,
      "n": true,
      "nr": true,
      "ns": true,
      "nsr": true,
      "p": true,
      "lat": 59.939012,
      "lon": 30.315804,
      "tzinfo":       {
         "name": "Europe/Moscow",
         "abbr": "MSK",
         "offset": 10800,
         "dst": false
      },
      "def_pressure_mm": 759,
      "def_pressure_pa": 1012,
      "_h": false,
      "url": "https://yandex.ru/pogoda/?lat=59.939012&lon=30.315804"
   },
   "fact":    {
      "temp": 2,
      "feels_like": -2,
      "temp_water": 0,
	...

Let's check what MockMotor recorded.

Review the Recorded Response

Let's take a look at the service's responses again.

You can see the newly recorded mock. It is placed ahead of the forward mock, so any new matched traffic is handled by mock. That is, if we have another request with the same lat and lon, it is responded by the mock, and not the live backend.

You can notice that MockMotor recorded:

  • HTTP operation GET
  • Relative URI /v1/forecast
  • Query parameter lon value
  • Query parameter lat value
  • Response time 564ms

Looking inside of the recorded response, you can see it has recorded the response payload, HTTP status and content type as well:

Test It

Let's test our new mock. We execute the same request for Palace Square once again. Now it should be replied from mock, and don't eat into our daily call limit.

And indeed, you can see X-MockMotor-Delay: 564 header in the response, telling us that the response was generated locally.

You can also see that the call count next to the mock response has incremented:

Repeat

To add more locations, just call the mock service with more combinations of lat and lon.

After running for some time, the mock service handles any predefined test locations from your test scenarios, not leaking any calls to the actual Yandex service.

Please Share

Was this post useful? Then please share! Tweet This