DEV Community: Douglas Lise

Como adicionar análise sintática de código ruby em uma aplicação legada?

Douglas Lise — Wed, 04 Mar 2020 23:39:27 +0000

Como é conhecido na comunidade ruby, Rubocop é a ferramenta mais usada para análise estática de código.

Ele já vem com muitas regras padrões, o que é muito útil para novos projetos, dessa forma você pode seguir a maioria dos padrões e apenas personalizar alguns deles.

Mas como proceder em projetos antigos ou maiores, com muitos arquivos e com muitos problemas?

A gem Rubocop vem com um formatador automático que pode corrigir muitos problemas automaticamente em nosso código.

Além disso, ele pode criar um arquivo de configuração TODO que ignora todos os problemas já presentes, arquivo por arquivo. Assim, podemos começar a usar o rubocop apenas para novos arquivos ou novos problemas nos arquivos existentes e corrigir cada problema de forma incremental.

A recomendação é corrigir todos os problemas possíveis automaticamente e manter os outros ignorados, a serem corrigidos em uma etapa futura.

Você pode começar adicionando a gem ao Gemfile (ou simplesmente instalando-a, mas é recomendável mantê-la no Gemfile).

$> bundle add rubocop

Com isso, você pode simplesmente executar o rubocop e todos os problemas serão mostrados. Como este exemplo:

$> rubocop
...
test/test_helper.rb:5:7: C: Style/ClassAndModuleChildren: Use nested module/class definitions instead of compact style.
class ActiveSupport::TestCase
      ^^^^^^^^^^^^^^^^^^^^^^^
test/test_helper.rb:6:81: C: Metrics/LineLength: Line is too long. [82/80]
  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
                                                                                ^^

135 files inspected, 1659 offenses detected

A partir de agora podemos seguir por duas estratégias diferentes:

1. Corrigindo todos os problemas em uma única vez

A primeira estratégia é corrigir todos os problemas de uma única vez. Para isso, basta executar rubocop --auto-correct e todos os problemas corrigíveis são corrigidos e mostrados com o status " [Corrected] ".

$> rubocop --auto-correct
...
ENV['RAILS_ENV'] ||= 'test'
^
test/application_system_test_case.rb:1:1: C: [Corrected] Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.
require "test_helper"
^

135 files inspected, 1848 offenses detected, 1485 offenses corrected

Na próxima execução, apenas os problemas restantes serão mostrados.

$> rubocop
...
135 files inspected, 363 offenses detected

Agora podemos ignorar todos estes erros rodando o rubocop com a opção --auto-gen-config. Isso criará dois arquivos:

.rubocop_todo.yml: Este arquivo contém regras para ignorar todos os problemas atuais.
.rubocop.yml: O principal arquivo de configuração. Inicialmente apenas herdando / incluindo o arquivo TODO.

$> rubocop --auto-gen-config
...
135 files inspected, 133 offenses detected
Created .rubocop_todo.yml.

A próxima execução não deve retornar problemas, porque todos são ignorados em .rubocop_todo.yml:

$> rubocop
...
135 files inspected, no offenses detected

Agora você precisa apenas commitar todas as alterações e, opcionalmente, adicionar uma etapa no pipeline do CI para executar o rubocop e verificar seus arquivos de origem.

2. Corrigindo um problema de cada vez

A outra estratégia é fazer isso de forma incremental, usando um script, para separar cada correção de problema em um commit git diferente. Isso ajudará em uma futura identificação de alterações e o motivo pelo qual ela ocorreu.
Para fazer isso, basta executar este script e ele irá aplicar e commitar cada problema em um commit separado.

É importante criar esse script em um caminho ignorado pelo git, como tmp/script.rb por exemplo. Após criar o arquivo de script, basta chamá-lo executando ruby tmp/script.rb.

# tmp/script.rb
require "yaml"
require "rubocop"

puts("Criando configuração...")
system("rubocop --auto-gen-config")
system("rm .rubocop.yml") # OR git checkout .rubocop.yml if you want to start with some config

puts("Lendo a configuração de TODO para obter todos os problemas...")
todo = YAML.load_file(".rubocop_todo.yml")
system("rm .rubocop_todo.yml")

todo.each_with_index do |(cop, _cop_settings), index|
  puts("Corrigindo #{cop} (#{index + 1}/#{todo.count})...")
  if eval("RuboCop::Cop::#{cop.gsub("/", "::")}").new.support_autocorrect?
    files = `rubocop --auto-correct --force-default-config --only #{cop} --format files`.split("\n")
    system("git add #{files.join(" ")}")
    system("git commit -m \"Correção automática do Rubocop para #{cop}\" \
            --author \"Rubocop Auto Correct <rubocop@rubocop>\"")
  else
    puts("  Não suporta auto-correção")
  end
end

puts("Gerando o arquivo TODO final com problemas incorrigíveis...")
system("rubocop --auto-gen-config")

puts("Finalizado")

Poderá levar alguns minutos para ser executado, dependendo do número e tamanho dos seus arquivos fonte.

O Rubocop foi projetado para ser seguro, mas algumas correções automáticas podem causar algum comportamento indesejado, por isso é importante ter uma boa cobertura de testes. É possível executar os testes unitários em cada iteração deste script ou apenas executá-los no final do processo. Nesse caso, se ocorrer um erro, você poderá usar git bisect para descobrir qual problema introduziu o erro.

No final, o script gera commits como os abaixo:

$> git log --oneline
fad06ac (HEAD -> rubocop) Correção automática do Rubocop para Style/WordArray
22aca61 Correção automática do Rubocop para Style/ClassCheck
06b12a2 Correção automática do Rubocop para Style/BracesAroundHashParameters
6dcace8 Correção automática do Rubocop para Style/BlockDelimiters
a504890 Correção automática do Rubocop para Style/BlockComments
1fae73b Correção automática do Rubocop para Naming/RescuedExceptionsVariableName
10802ca Correção automática do Rubocop para Lint/UnusedMethodArgument
bd11ffc Correção automática do Rubocop para Lint/UnusedBlockArgument
b8e098d Correção automática do Rubocop para Layout/TrailingBlankLines
4cabbd7 Correção automática do Rubocop para Layout/Tab
f29e766 Correção automática do Rubocop para Layout/SpaceInsidePercentLiteralDelimiters
b451890 Correção automática do Rubocop para Layout/SpaceInsideHashLiteralBraces
ee9eae6 Correção automática do Rubocop para Layout/SpaceInsideBlockBraces
f6611fd Correção automática do Rubocop para Layout/SpaceInsideArrayLiteralBrackets
f858545 Correção automática do Rubocop para Layout/SpaceInLambdaLiteral
3aeed2e Correção automática do Rubocop para Layout/SpaceBeforeComment
8b40371 Correção automática do Rubocop para Layout/SpaceBeforeBlockBraces
444f7ae Correção automática do Rubocop para Layout/SpaceAroundEqualsInParameterDefault
42d4e69 Correção automática do Rubocop para Layout/SpaceAfterComma
1acc003 Correção automática do Rubocop para Layout/SpaceAfterColon
28ed714 Correção automática do Rubocop para Layout/MultilineOperationIndentation
3e08ad7 Correção automática do Rubocop para Layout/MultilineMethodCallIndentation
f4611e5 Correção automática do Rubocop para Layout/MultilineMethodCallBraceLayout
6cc68b6 Correção automática do Rubocop para Layout/MultilineHashBraceLayout
9f66fe9 Correção automática do Rubocop para Layout/LeadingCommentSpace
...

Independentemente da estratégia utilizada, é recomendável resolver todos os problemas que permaneceram no arquivo TODO e, incrementalmente, mover todos os ignorados que realmente precisam ser ignorados para arquivo principal rubocop.yml. Isso garantirá que você continue melhorando seu código ou, pelo menos, mantendo sua qualidade.

How to migrate all your Git repositores to a new computer?

Douglas Lise — Wed, 15 Jan 2020 02:11:31 +0000

Have you ever needed to change from an old computer to a new one and needed to manually clone all the Git repositories?

The below script keeps your folder structure and migrates all repositories to a new computer.

dirs=$(find . -name '.git' -type d | sed 's/\/\.git//')

for dir in $dirs; do
  GIT_DIR=$dir/.git

  # Creates the folder structure
  echo mkdir -p $dir;

  # Clones the repository in the same folder
  echo git clone $(git --git-dir=$GIT_DIR remote get-url origin) $dir;

  # Re-adds other remotes
  for r in $(git --git-dir=$GIT_DIR remote | grep -v origin); do
    echo git --git-dir=$GIT_DIR remote add $r $(git --git-dir=$GIT_DIR remote get-url $r);
  done

  echo
done > clone-all.sh

You just need to paste it into a shell session at the base folder where all your repositories are stored. The script creates a file named clone-all.sh. Next, you just need to copy and run this file in the new computer.

Script explanation

The script initially finds all sub-folders that contain a .git folder, which denotes it as a Git repository. From this list it starts to output commands that will be executed in the new computer.

The output commands (for each folder) are these:

A command to create the folder, keeping the same structure;
A command to clone the repository, to the same folder name, using the origin remote;
A command to add the other remotes (different than origin) to the newly cloned repository.

Example

In the old computer, before run the script, the folder structure was this:

➜  pwd
/home/douglas/code

➜  tree -L 2
.
├── create-clone-script.sh
├── other
│   ├── audited_views
│   ├── bitmovin-ruby
│   ├── cucumber-ruby
│   ├── devise-i18n
│   ├── douglaslise.github.io
│   ├── gitlab-ci-monitor
│   ├── rubocop
│   ├── sendgrid-ruby
│   └── summernote
└── private
    ├── ping-monitor
    └── qd

So I pasted the script in the shell and it generated this file clone-all.sh:

➜  cat clone-all.sh           
mkdir -p ./other/bitmovin-ruby
git clone git@github.com:bitmovin/bitmovin-ruby.git ./other/bitmovin-ruby
git --git-dir=./other/bitmovin-ruby/.git remote add fork git@github.com:douglaslise/bitmovin-ruby.git

mkdir -p ./other/audited_views
git clone git@github.com:douglaslise/audited_views.git ./other/audited_views

mkdir -p ./other/cucumber-ruby
git clone git@github.com:cucumber/cucumber-ruby.git ./other/cucumber-ruby

mkdir -p ./other/rubocop
git clone git@github.com:rubocop-hq/rubocop ./other/rubocop

mkdir -p ./other/gitlab-ci-monitor
git clone git@github.com:globocom/gitlab-ci-monitor.git ./other/gitlab-ci-monitor
git --git-dir=./other/gitlab-ci-monitor/.git remote add fork git@github.com:douglaslise/gitlab-ci-monitor.git

mkdir -p ./other/summernote
git clone git@github.com:summernote/summernote.git ./other/summernote

mkdir -p ./other/devise-i18n
git clone git@github.com:tigrish/devise-i18n.git ./other/devise-i18n
git --git-dir=./other/devise-i18n/.git remote add fork git@github.com:douglaslise/devise-i18n.git

mkdir -p ./other/douglaslise.github.io
git clone git@github.com:douglaslise/douglaslise.github.io.git ./other/douglaslise.github.io

mkdir -p ./other/sendgrid-ruby
git clone git@github.com:sendgrid/sendgrid-ruby.git ./other/sendgrid-ruby
git --git-dir=./other/sendgrid-ruby/.git remote add fork git@github.com:douglaslise/sendgrid-ruby.git

mkdir -p ./private/ping-monitor
git clone ssh://hg@bitbucket.org/douglaslise/ping-monitor ./private/ping-monitor
git --git-dir=./private/ping-monitor/.git remote add heroku https://git.heroku.com/pingmonitor.git

mkdir -p ./private/qd
git clone ssh://hg@bitbucket.org/douglaslise/qd ./private/qd
➜

Next, I copied the generated file to the new computer and executed it in the base folder where I wanted to clone all the repositories:

➜  pwd
/home/douglas-new/code

➜  ls
clone-all.sh

➜  sh clone-all.sh
Cloning into './other/bitmovin-ruby'...
remote: Enumerating objects: 56, done.
remote: Counting objects: 100% (56/56), done.
remote: Compressing objects: 100% (43/43), done.
remote: Total 1707 (delta 21), reused 33 (delta 10), pack-reused 1651
Receiving objects: 100% (1707/1707), 245.62 KiB | 1.00 MiB/s, done.
Resolving deltas: 100% (1051/1051), done.

Cloning into './other/audited_views'...
remote: Enumerating objects: 314, done.
remote: Total 314 (delta 0), reused 0 (delta 0), pack-reused 314
Receiving objects: 100% (314/314), 48.93 KiB | 331.00 KiB/s, done.
Resolving deltas: 100% (81/81), done.

Cloning into './other/cucumber-ruby'...
remote: Enumerating objects: 371, done.
remote: Counting objects: 100% (371/371), done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 58832 (delta 227), reused 264 (delta 163), pack-reused 58461
Receiving objects: 100% (58832/58832), 11.76 MiB | 4.73 MiB/s, done.
Resolving deltas: 100% (40075/40075), done.

Cloning into './other/rubocop'...
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 90142 (delta 0), reused 2 (delta 0), pack-reused 90131
Receiving objects: 100% (90142/90142), 29.81 MiB | 4.98 MiB/s, done.
Resolving deltas: 100% (68283/68283), done.

Cloning into './other/gitlab-ci-monitor'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 390 (delta 3), reused 3 (delta 0), pack-reused 376
Receiving objects: 100% (390/390), 471.38 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (210/210), done.

Cloning into './other/summernote'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 21101 (delta 21), reused 11 (delta 5), pack-reused 21060
Receiving objects: 100% (21101/21101), 13.13 MiB | 4.58 MiB/s, done.
Resolving deltas: 100% (13576/13576), done.

Cloning into './other/devise-i18n'...
remote: Enumerating objects: 114, done.
remote: Counting objects: 100% (114/114), done.
remote: Compressing objects: 100% (79/79), done.
remote: Total 4852 (delta 45), reused 56 (delta 19), pack-reused 4738
Receiving objects: 100% (4852/4852), 1.33 MiB | 2.81 MiB/s, done.
Resolving deltas: 100% (2928/2928), done.

Cloning into './other/douglaslise.github.io'...
remote: Enumerating objects: 463, done.
remote: Total 463 (delta 0), reused 0 (delta 0), pack-reused 463
Receiving objects: 100% (463/463), 4.46 MiB | 3.07 MiB/s, done.
Resolving deltas: 100% (92/92), done.

Cloning into './other/sendgrid-ruby'...
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 2861 (delta 7), reused 14 (delta 4), pack-reused 2838
Receiving objects: 100% (2861/2861), 680.79 KiB | 1.60 MiB/s, done.
Resolving deltas: 100% (1313/1313), done.

Cloning into './private/ping-monitor'...
remote: Counting objects: 490, done.
remote: Compressing objects: 100% (436/436), done.
remote: Total 490 (delta 256), reused 75 (delta 25)
Receiving objects: 100% (490/490), 72.60 KiB | 352.00 KiB/s, done.
Resolving deltas: 100% (256/256), done.

Cloning into './private/qd'...
remote: Counting objects: 632, done.
remote: Compressing objects: 100% (418/418), done.
remote: Total 632 (delta 288), reused 332 (delta 128)
Receiving objects: 100% (632/632), 151.00 KiB | 322.00 KiB/s, done.
Resolving deltas: 100% (288/288), done.

Now we can see the newly created structure:

➜  tree -L 2
.
├── clone-all.sh
├── other
│   ├── audited_views
│   ├── bitmovin-ruby
│   ├── cucumber-ruby
│   ├── devise-i18n
│   ├── douglaslise.github.io
│   ├── gitlab-ci-monitor
│   ├── rubocop
│   ├── sendgrid-ruby
│   └── summernote
└── private
    ├── ping-monitor
    └── qd

Conclusion

This could be used to clone all your projects just by copying a unique, small file, without the needing to copy all the files and folders.

If you have suggestions, please let me know in the comments.

Thanks.

How to start using ruby syntatic analysis in a legacy application?

Douglas Lise — Wed, 14 Aug 2019 00:25:47 +0000

As known in ruby community, Rubocop is the most used tool for static code analysis.

It comes with a lot of default rules, what is very useful for new projects, when you could follow the most of defaults and just customize some of them.

But how to proceed in legacy or bigger projects, that have a lot of files already written and potentially having many issues?

Rubocop gem comes with an auto-formatter that could fix a lot of issues automatically in our code.

Additionally, it could create a TODO config file that ignores all the issues already present, file by file. So we can start using rubocop just for new files, or new issues in existing files, and incrementally fix issue by issue.

The recommendation is to fix all possible issues automatically and keep others as ignored, to be fixed in a future step.

You can start by adding the gem to Gemfile (or simply installing it, but it is recommended to keep it in Gemfile).

$> bundle add rubocop

With this you could just run rubocop and all issues will be shown. Like this example:

$> rubocop
...
test/test_helper.rb:5:7: C: Style/ClassAndModuleChildren: Use nested module/class definitions instead of compact style.
class ActiveSupport::TestCase
      ^^^^^^^^^^^^^^^^^^^^^^^
test/test_helper.rb:6:81: C: Metrics/LineLength: Line is too long. [82/80]
  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
                                                                                ^^

135 files inspected, 1659 offenses detected

Now there are two strategies to go ahead.

1. Fix all issues in a single batch

The first strategy is to fix all the issues in a single batch. To do this just run rubocop --auto-correct and all fixable issues are fixed and shown with the "[Corrected]" status.

$> rubocop --auto-correct
...
ENV['RAILS_ENV'] ||= 'test'
^
test/application_system_test_case.rb:1:1: C: [Corrected] Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.
require "test_helper"
^

135 files inspected, 1848 offenses detected, 1485 offenses corrected

In the next run just the remaining issues are shown.

$> rubocop
...
135 files inspected, 363 offenses detected

We can now ignore them all running with --auto-gen-config option. This will create two files:

.rubocop_todo.yml: This file contains ignore rules for all current issues.
.rubocop.yml: The main configuration file. Initially just inheriting/including the TODO file.

$> rubocop --auto-gen-config
...
135 files inspected, 133 offenses detected
Created .rubocop_todo.yml.

The next run should return no issues, because they are all ignored in .rubocop_todo.yml:

$> rubocop
...
135 files inspected, no offenses detected

Now you need to just commit all the changes and, optionally, add a step in your CI pipeline to run rubocop to check your source files.

2. Fix issues cop by cop

The other strategy is to do this incrementally, using a script, to separate each cop fix in a different git commit. This helps in the future identifications of changes and why they happened.
To do this, just run this script and it will apply and commit each issue in a separated commit.

It is important to create this script in a path ignored by git, like tmp/script.rb. After create the script file, just call it running ruby tmp/script.rb.

# tmp/script.rb
require "yaml"
require "rubocop"

puts("Generating config...")
system("rubocop --auto-gen-config")
system("rm .rubocop.yml") # OR git checkout .rubocop.yml if you want to start with some config

puts("Reading TODO config to get all possible issues...")
todo = YAML.load_file(".rubocop_todo.yml")
system("rm .rubocop_todo.yml")

todo.each_with_index do |(cop, _cop_settings), index|
  puts("Fixing #{cop} (#{index + 1}/#{todo.count})...")
  if eval("RuboCop::Cop::#{cop.gsub("/", "::")}").new.support_autocorrect?
    files = `rubocop --auto-correct --force-default-config --only #{cop} --format files`.split("\n")
    system("git add #{files.join(" ")}")
    system("git commit -m \"chore: Correct source files with rubocop #{cop} cop\" \
            --author \"Rubocop Auto Correct <rubocop@rubocop>\"")
  else
    puts("  Does not support auto-correct")
  end
end

puts("Generating final TODO file with _uncorrectable_ issues...")
system("rubocop --auto-gen-config")

puts("Done")

It may take several minutes to run, depending on the number and size of your source files.

Rubocop is designed to be safe, but some fixes could cause undesired behaviors, so it is important to have a great test coverage. You could run your unit tests in each iteration of this script or just run them in the end of the process. In this case, if an error occurs, you could use git bisect to find which issue introduced the error.

In the end the script generates commits like the ones bellow:

$> git log --oneline
fad06ac (HEAD -> rubocop) chore: Correct source files with rubocop Style/WordArray cop
22aca61 chore: Correct source files with rubocop Style/ClassCheck cop
06b12a2 chore: Correct source files with rubocop Style/BracesAroundHashParameters cop
6dcace8 chore: Correct source files with rubocop Style/BlockDelimiters cop
a504890 chore: Correct source files with rubocop Style/BlockComments cop
1fae73b chore: Correct source files with rubocop Naming/RescuedExceptionsVariableName cop
10802ca chore: Correct source files with rubocop Lint/UnusedMethodArgument cop
bd11ffc chore: Correct source files with rubocop Lint/UnusedBlockArgument cop
b8e098d chore: Correct source files with rubocop Layout/TrailingBlankLines cop
4cabbd7 chore: Correct source files with rubocop Layout/Tab cop
f29e766 chore: Correct source files with rubocop Layout/SpaceInsidePercentLiteralDelimiters cop
b451890 chore: Correct source files with rubocop Layout/SpaceInsideHashLiteralBraces cop
ee9eae6 chore: Correct source files with rubocop Layout/SpaceInsideBlockBraces cop
f6611fd chore: Correct source files with rubocop Layout/SpaceInsideArrayLiteralBrackets cop
f858545 chore: Correct source files with rubocop Layout/SpaceInLambdaLiteral cop
3aeed2e chore: Correct source files with rubocop Layout/SpaceBeforeComment cop
8b40371 chore: Correct source files with rubocop Layout/SpaceBeforeBlockBraces cop
444f7ae chore: Correct source files with rubocop Layout/SpaceAroundEqualsInParameterDefault cop
42d4e69 chore: Correct source files with rubocop Layout/SpaceAfterComma cop
1acc003 chore: Correct source files with rubocop Layout/SpaceAfterColon cop
28ed714 chore: Correct source files with rubocop Layout/MultilineOperationIndentation cop
3e08ad7 chore: Correct source files with rubocop Layout/MultilineMethodCallIndentation cop
f4611e5 chore: Correct source files with rubocop Layout/MultilineMethodCallBraceLayout cop
6cc68b6 chore: Correct source files with rubocop Layout/MultilineHashBraceLayout cop
9f66fe9 chore: Correct source files with rubocop Layout/LeadingCommentSpace cop
...

Regardless of the strategy used, it is recommended to resolve all the issues that remained in the TODO file and, incrementally, move all ignores that really needs to be ignored to the main .rubocop.yml file. This will ensure that you keeps improving your code or, or at least, maintaining its quality.

Finally, rubocop is a great tool that helps us keeping and improving the quality of our code. It is extremelly recomended to use it in all of our ruby projects.

Hi, I'm Douglas Lise

Douglas Lise — Mon, 03 Apr 2017 02:27:13 +0000

I have been coding for 16 years.

You can find me on Twitter as @douglaslise

I live in Porto Alegre, Brazil.

I work for Globo.com.

I mostly program in these languages: Ruby,JavaScript.

I am currently learning more about Reactive frameworks.

Nice to meet you.