DEV Community: Ana Schwendler

Usando VCR para simular requisições

Ana Schwendler — Fri, 26 Nov 2021 11:46:36 +0000

Nos últimos meses, tenho trabalhado com muitas requisições externas de API. Como faço parte da equipe de logística da Marley Spoon, nosso desafio é garantir que nossas requisições sejam integradas a tempo e que tudo seja entregue da melhor forma possível.

Parte dessa tarefa, como desenvolvedora de software, é garantir que o código que foi escrito seja bem testado e faça o que deve ser feito, de forma correta e eficiente. Nosso projeto tem muitas integrações com parceiros de logística. Quando fazemos uma integração esses outros sistemas, não queremos realizar requisições reais toda vez que executamos nossos testes. Essa ação não só gera apenas requisições desnecessárias, mas também pode gerar problemas com limites de requisições com API externas além de outros efeitos colaterais inconvenientes, como testes tomando mais tempo para rodar. Para evitar isso, fingimos fazer requisições reais e há uma ferramenta que ajuda a simular essas chamadas: ao registrar dados reais de uma requisição real, somos capazes de fazer o esboço(stub) de uma resposta de serviço web externo. O arquivo salvo é chamados de "fita cassete (cassette tape)" e o nome da ferramenta a que me refiro é VCR.

Neste post vou explicar como tenho aprendido a usá-lo e por que acho que é útil.

O que é uma simulação de requisição de teste?

Como este artigo não é sobre a técnica de simulação de requisições, vou apenas explicar brevemente e adicionar um link para um artigo útil sobre o assunto. Neste artigo(em inglês) podemos observar que simulação de testes de requisição pode ser definido como:

Simular testes de requisição significa uma versão simulada de um serviço externo ou interno que pode substituir o real, ajudando seus testes a serem executados de forma mais rápida e confiável. Quando sua implementação interage com as propriedades de um objeto, em vez de sua função ou comportamento, uma imitação pode ser usada.

… e é exatamente para isso que usamos o VCR.

O que é VCR?

De acordo com a documentação:

Grave as interações HTTP do seu conjunto de testes e reproduza-as durante futuras execuções de teste para testes rápidos, determinísticos e precisos.

Que significa que, quando você executar o teste pela primeira vez com a sintaxe do VCR, ele registrará o que aconteceu e, na próxima vez que você executar o teste novamente, terá a versão gravada pronta para substituir a solicitação real.

Como usar o VCR?

Eu tenho um exemplo básico aqui.

Basicamente, sempre que você quiser testar uma parte do código que requer uma requisição externa, você usa o método .use_cassette para expor que você deseja que o VCR lide com isso como uma fita cassete. Se já houver dados com uma interação HTTP pré-gravada, o VCR o usará. Se não, o VCR cria automáticamente uma "fita cassete" (dessa vez fazendo uma requisição real) baseado na requisição feita no teste.

Mas ainda assim, se o que você quer realizar requisições reais todas as vezes que seus testes rodam, VCR também oferece diferente modos de gravação, que podem ser conferidos aqui.

Exemplo:

require 'rubygems'
require 'test/unit'
require 'vcr'

VCR.configure do |config|
  config.cassette_library_dir = "fixtures/vcr_cassettes"
  config.hook_into :webmock
end

class VCRTest < Test::Unit::TestCase
  def test_example_dot_com
    VCR.use_cassette("synopsis") do
      response = Net::HTTP.get_response(URI('http://www.iana.org/domains/reserved'))
      assert_match /Example domains/, response.body
    end
  end
end

Aqui estamos requisitando o uso de uma fita chamada synopsis, que simula a requisição para a iana.org.

Se o teste estiver rodando pela primeira vez, VCR irá gerar um arquivo, armazenado em fixtures/vcr_cassettes, que nesse exemplo se chamará synopsis.yml.

Após rodar o teste, o arquivo fixtures/vcr_cassettes/synopsis.yml se parecerá com (eu removi algumas partes por que o arquivo é grande):

--------
http_interactions:
- request:
    method: get
    uri: http://www.iana.org/domains/reserved
    body:
      encoding: US-ASCII
      string: ''
    headers:
      Accept-Encoding:
      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
      Accept:
      - "*/*"
      User-Agent:
      - Ruby
  response:
    status:
      code: 200
      message: OK
    headers:
      Date:
      - Fri, 18 Jun 2021 08:06:40 GMT
      Server:
      - Apache
      Vary:
      - Accept-Encoding
      Last-Modified:
      - Thu, 21 May 2020 22:41:39 GMT
      X-Frame-Options:
      - SAMEORIGIN
      Expires:
      - Fri, 18 Jun 2021 09:56:45 GMT
      Referrer-Policy:
      - origin-when-cross-origin
      X-Content-Type-Options:
      - nosniff
      Age:
      - '595'
      Content-Type:
      - text/html; charset=UTF-8
      Cache-Control:
      - public, max-age=21603
      Content-Security-Policy: /*cropped security policy*/
      Transfer-Encoding:
      - chunked
    body:
      encoding: ASCII-8BIT
      string: !binary |-
        /*cropped binary string*/
    http_version:
  recorded_at: Fri, 18 Jun 2021 08:06:40 GMT
recorded_with: VCR 5.0.0

Então, a partir de agora sua suíte de testes usará o arquivo salvo. Isso ainda vai fazer seus testes rodarem mais rápido, pois agora temos o arquivo armazenado e não precisamos fazer a requisição real a aplicação externa, o que sempre pode tomar muito mais tempo.

Exemplos da vida real

Um exemplo de projeto que usa VCR é esse cliente site-search-ruby da elastic: https://github.com/elastic/site-search-ruby

No contexto de Search, quando procuramos todos os DocumentTypes no mecanismo, eles usam um arquivo chamado engine_search para simular uma requisição:

https://github.com/elastic/site-search-ruby/blob/master/spec/client_spec.rb

Esse é o arquivo usado nesta instância: https://github.com/elastic/site-search-ruby/blob/master/spec/fixtures/vcr/engine_search.yml

Conclusão

VCR parece ser uma ferramenta bastante confiável para simular requisições para API externas. Mas nós ainda precisamos estar conscientes de que API podem sempre mudar, e quando isso acontece, precisamos gravar nossos testes novamente. Esse processo deve ser simples e fácil de reproduzir, então para mais ficar sobre o uso do VCR, eu recomendo esse artigo.

Escondendo credenciais confidenciais em vcr_cassettes

Há uma opção de configuração disponível para filtrar dados sensíveis que podem ser usados para evitar que sejam gravados nos arquivos de cassete, a documentação está aqui. Isso é importante, por que não queremos credenciais secretas expostas publicamente nos nossos repositórios armazenados na internet.

References

https://github.com/vcr/vcr

https://github.com/elastic/site-search-ruby

https://circleci.com/blog/how-to-test-software-part-i-mocking-stubbing-and-contract-testing/

https://fabioperrella.github.io/10_tips_to_help_using_the_VCR_gem_in_your_ruby_test_suite.html

Using VCR to Mock Your Requests

Ana Schwendler — Fri, 18 Jun 2021 10:05:31 +0000

In the past months, I've been working with a lot of external API requests. Since I'm part of the Marley Spoon’s logistics team, our challenge is to make sure our requests are done in time and everything is delivered in good shape.

Part of this task, as a software developer, is making sure the code I write is well tested, and does what it is supposed to do. Our software has many integrations with our logistics partners. When integrating with their systems, we don't want to hit external APIs with real requests every time we run our tests. This not only generates unnecessary requests that can cause problems with API rate limits and other unwanted side-effects. It also makes them run slower. To avoid that, we pretend to make real requests and there is a tool that helps "faking" these calls: by recording real data from a real network request, we are able to stub a third party’s web service response. The saved data is called a "cassette tape" and the name of the tool I refer to is VCR.

In this post I'll explain how I've been learning to use it and why I think it is useful.

What is a mock test?

Since this is not about the mocking technique, I'll just briefly introduce it and link to a useful article on the topic. In this post we can see mock testing defined as:

Mocking means creating a fake version of an external or internal service that can stand in for the real one, helping your tests run more quickly and more reliably. When your implementation interacts with an object’s properties, rather than its function or behavior, a mock can be used.

… and that is exactly the reason why we use VCR.

What is VCR?

According to their documentation:

Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.

Which means that when you first run your test with the VCR syntax, it will record what happened and next time you run the test again, you will have the recorded version ready to stand in for the real request.

How to use VCR?

I have a basic example here.

So basically, whenever you want to test a part of code that requires an external web request, you use the .use_cassette method to state that you want VCR to deal with that with a cassette file. If there already is data with a pre-recorded HTTP interaction, VCR will use it. If not, VCR will automatically create a cassette file (this time making a real request) based on the request made in the test.

But still, if you are looking forward to making real requests, VCR also offers different record modes, which can be checked here

Example:

require 'rubygems'
require 'test/unit'
require 'vcr'

VCR.configure do |config|
  config.cassette_library_dir = "fixtures/vcr_cassettes"
  config.hook_into :webmock
end

class VCRTest < Test::Unit::TestCase
  def test_example_dot_com
    VCR.use_cassette("synopsis") do
      response = Net::HTTP.get_response(URI('http://www.iana.org/domains/reserved'))
      assert_match /Example domains/, response.body
    end
  end
end

Here you are requesting to use a cassette called synopsis, which mocks a request to iana.org.

If this is running the first time, VCR will generate a "cassette" file, which will be stored at fixtures/vcr_cassettes, which for the example will be called synopsis.yml.

This is what fixtures/vcr_cassettes/synopsis.yml looks like (I've removed some parts as it was too big):

--------
http_interactions:
- request:
    method: get
    uri: http://www.iana.org/domains/reserved
    body:
      encoding: US-ASCII
      string: ''
    headers:
      Accept-Encoding:
      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
      Accept:
      - "*/*"
      User-Agent:
      - Ruby
  response:
    status:
      code: 200
      message: OK
    headers:
      Date:
      - Fri, 18 Jun 2021 08:06:40 GMT
      Server:
      - Apache
      Vary:
      - Accept-Encoding
      Last-Modified:
      - Thu, 21 May 2020 22:41:39 GMT
      X-Frame-Options:
      - SAMEORIGIN
      Expires:
      - Fri, 18 Jun 2021 09:56:45 GMT
      Referrer-Policy:
      - origin-when-cross-origin
      X-Content-Type-Options:
      - nosniff
      Age:
      - '595'
      Content-Type:
      - text/html; charset=UTF-8
      Cache-Control:
      - public, max-age=21603
      Content-Security-Policy: /*cropped security policy*/
      Transfer-Encoding:
      - chunked
    body:
      encoding: ASCII-8BIT
      string: !binary |-
        /*cropped binary string*/
    http_version:
  recorded_at: Fri, 18 Jun 2021 08:06:40 GMT
recorded_with: VCR 5.0.0

From now on your test suite will use this cassette file. It will also run faster, as it has the file in place and doesn't need to make the request which can take more time.

Examples in real life

One example of project that uses VCR is this site-search-ruby client from elastic: https://github.com/elastic/site-search-ruby

In their context of Search, when searching all DocumentTypes in the engine, they use a cassette called engine_search to mock a request:

https://github.com/elastic/site-search-ruby/blob/master/spec/client_spec.rb

This is the cassette file that is used in this call: https://github.com/elastic/site-search-ruby/blob/master/spec/fixtures/vcr/engine_search.yml

Conclusion

VCR seems to be a very reliable tool to fake requests to APIs. But we still need to be aware that sometimes APIs can change and with that we need to record our tests again. This process should be simple and easy to reproduce, so for more tips regarding VCR use, I recommend this blog post.

Hiding confidential credentials in vcr_cassettes

There is a configuration option available to filter sensitive data that can be used to prevent it from being written to the cassette files, the documentation is here. This is important, as you want to track your cassette YML files in your source control management tool. And secrets don’t belong there.

References

https://github.com/vcr/vcr

https://github.com/elastic/site-search-ruby

https://circleci.com/blog/how-to-test-software-part-i-mocking-stubbing-and-contract-testing/

https://fabioperrella.github.io/10_tips_to_help_using_the_VCR_gem_in_your_ruby_test_suite.html

Thanks Memuna for reviewing it! 🎉

Setting up tests for Django Plugins

Ana Schwendler — Sat, 04 Apr 2020 11:31:05 +0000

Did you have a good idea for a nice Django plugin? Don’t know how to set up tests for it without creating a whole Django app for your project? Here is the solution for you. This article is written after a pull request for Django Public Admin, using django-stdimage as a base.

Django Public Admin is a plugin that makes a public and read-only version of the Django Admin. It uses Poetry to manage packages and dependencies, pytest to run tests in addition to pytest-django.

What is Django

Django is a web framework written in Python, and it’s designed for developers to write reusable apps (apps in this context means application, which in this case is the project designed using Django):

“Reusability is the way of life in Python. […] Django itself is also a normal Python package. This means that you can take existing Python packages or Django apps and compose them into your own web project. You only need to write the parts that make your project unique. The idea of these apps.”

In other words, we can wrap reusable parts of any Django app and use them in other Django apps we write. This is the idea and for that we also need tests. At the end of the day, we have a reusable bit of code without the accompanying Django application.

So, how to test a part of a Django app without the full Django app? 🤔

Understanding what is missing

With the environment already configured, the first issue that appeared was that when running the tests outside of a Django app was that it was complaining that our Django was ImproperlyConfigured and that we need to set up the DJANGO_SETTINGS_MODULE:

You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

Setting up `tests/settings.py`

After talking to Ana Paula Gomes, she pointed out that she already saw this being implemented in a nice way on another project, and what was being made was to create a tests/settings.py file that will act as configuration for Django to run the tests. So for that, I created a file that looks like that:

import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'public_admin',
    'tests'
)

SECRET_KEY = 'foobar'

It does contain the default Django configuration variables as below:

BASE_DIR: This is pointed to the Django project directory.
INSTALLED_APPS: Which contains all the apps necessary to run your set up (don’t forget to add your project here public_admin in our target project, and tests ), and might vary depending on what you are testing.
SECRET_KEY: The SECRET_KEY setting must not be empty.

Setting up `setup.cfg`

After configuring up the settings.py we need to tell our application to use this file to run the tests, setting up the DJANGO_SETTINGS_MODULE variable in a special file called setup.cfg, in our case, we tell that when [tool:pytest] is called we should use this settings environment, this might look like that:

[tool:pytest]
DJANGO_SETTINGS_MODULE=tests.settings

And that is it, now you can set up your tests for Django Plugins without creating an app for testing using pytest.

DEV Community: Ana Schwendler

Usando VCR para simular requisições

O que é uma simulação de requisição de teste?

O que é VCR?

Como usar o VCR?

Exemplos da vida real

Conclusão

Escondendo credenciais confidenciais em vcr_cassettes

References

Using VCR to Mock Your Requests

What is a mock test?

What is VCR?

How to use VCR?

Examples in real life

Conclusion

Hiding confidential credentials in vcr_cassettes

References

Setting up tests for Django Plugins

What is Django

Understanding what is missing

Setting up tests/settings.py

Setting up setup.cfg

Setting up `tests/settings.py`

Setting up `setup.cfg`