<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Lucas Pacheco</title>
    <description>The latest articles on DEV Community by Lucas Pacheco (@lucaspsilveira).</description>
    <link>https://dev.to/lucaspsilveira</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F156037%2F1cbd5ced-d9ca-4323-9e0f-e01a6e81d928.jpeg</url>
      <title>DEV Community: Lucas Pacheco</title>
      <link>https://dev.to/lucaspsilveira</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/lucaspsilveira"/>
    <language>en</language>
    <item>
      <title>Tips to debug authentication error with Kerberos on Windows Server</title>
      <dc:creator>Lucas Pacheco</dc:creator>
      <pubDate>Tue, 11 Jun 2024 00:39:50 +0000</pubDate>
      <link>https://dev.to/lucaspsilveira/tips-to-debug-authentication-error-with-kerberos-on-windows-server-a45</link>
      <guid>https://dev.to/lucaspsilveira/tips-to-debug-authentication-error-with-kerberos-on-windows-server-a45</guid>
      <description>&lt;p&gt;Authentication errors with Kerberos and Windows Server are not unusual. To be able to find these errors, there are a lot of internet pages about Kerberos and Windows Server. Therefore, below are some summarized steps that may help you identify any problems and possibly help you resolve them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Steps
&lt;/h2&gt;

&lt;p&gt;1 - Check that the application on the Windows Server has all authentication options disabled, except the Windows Authentication option, which must be enabled.&lt;/p&gt;

&lt;p&gt;2 - In the Windows Authentication section, click on "Advanced Settings" and make sure that the Negotiate option is available and in the first position of the available list, if you wish to have authentication through NTLM, leave it second in the list.&lt;/p&gt;

&lt;p&gt;3 - For performance matters, check the "Use kernel mode" option. This option uses the machine account by default to decrypt Kerberos tickets.&lt;/p&gt;

&lt;p&gt;4 - If you use a custom account for your ApplicationPool, check the application configuration regarding WindowsAuthentication in the Configuration Editor. Access the path system.webServer/security/authentication/windowsAuthentication and make sure the UseAppPoolCredentials option is set to True.&lt;/p&gt;

&lt;p&gt;5 - After ensuring that your WebServer and apps are well configured to receive Kerberos authentications and the error persists, you must check whether the settings of the account used are correct to authenticate with Kerberos.&lt;/p&gt;

&lt;p&gt;6 - Check if the account's SPNs are correctly configured. To list the SPNs of an account you can run the following command in PowerShell from a machine inserted in the same AD domain you are using. It could be the server itself. Run the following command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;setspn -l domain\account_name
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The command must return the SPNs configured on the account. Guarantee the account has the correct SPN with the name of the server being used for authentication. E.g. HTTP/myserver.com, SERV/myserv. Also verify that no repeated SPNs occur, as this can cause failures.&lt;/p&gt;

&lt;p&gt;If you do not have SPNs configured, you can add SPNs to an AD account with the following command, if you have access to do so:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;setspn -s HTTP/myserver.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;7 - If you use a MachineAccount to authenticate with Kerberos on your WebServer, check the SPNs configured in the MachineAccount on the server itself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;setspn -l server_name
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you do not have the correct SPNs, follow the steps to add an SPN in item 6.&lt;/p&gt;

&lt;p&gt;I hope these steps helped you solve your problem as they helped me!&lt;/p&gt;

&lt;p&gt;For other issues, please comment below, maybe we can help each other. &lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://learn.microsoft.com/en-us/sql/reporting-services/report-server/register-a-service-principal-name-spn%20-for-a-report-server?view=sql-server-ver16"&gt;Register a Service Principal Name (SPN) for a report server&lt;/a&gt;&lt;br&gt;
&lt;a href="https://learn.microsoft.com/en-us/troubleshoot/windows-server/windows-security/kerberos-authentication-troubleshooting-guidance"&gt;Kerberos authentication troubleshooting guidance&lt;/a&gt;&lt;/p&gt;

</description>
      <category>kerberos</category>
      <category>windowsserver</category>
      <category>authentication</category>
      <category>iis</category>
    </item>
    <item>
      <title>How to authenticate with Kerberos using .NET</title>
      <dc:creator>Lucas Pacheco</dc:creator>
      <pubDate>Thu, 02 Mar 2023 18:13:21 +0000</pubDate>
      <link>https://dev.to/lucaspsilveira/how-to-authenticate-with-kerberos-using-net-2gh2</link>
      <guid>https://dev.to/lucaspsilveira/how-to-authenticate-with-kerberos-using-net-2gh2</guid>
      <description>&lt;p&gt;Recently, I have had the not-so-easy task to authenticate some API calls using the Kerberos method authentication. After a while of reading some docs and digging stack overflow about this subject, I decided to post a summary of what I have discovered. &lt;/p&gt;

&lt;p&gt;First of all, hope that your infrastructure is all set up within the same network and all SPNs are configured. Having this set up correctly will prevent you from some headaches when things go wrong, even if your code is correct. &lt;/p&gt;

&lt;p&gt;If you are using Windows as your environment and your server is within the same network as the Kerberos Authentication Server you should be able to easily authenticate using the default credentials or passing the username and password using the Network Credentials class of .NET inside your HttpClientHandler when you configure your HttpClient. Make sure your server is added to the Active Directory that manages the authentication. &lt;/p&gt;

&lt;p&gt;To authenticate using custom credentials you can use the following code inside your ConfigureServices method:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;services.AddHttpClient&amp;lt;IConnector, Connector&amp;gt;().ConfigurePrimaryHttpMessageHandler(
      serviceProvider =&amp;gt;
            {
                var credentialsCache = new CredentialCache {
                    { new Uri("https://yoururl.com/endpoint"),
                      "Negotiate",
                      new NetworkCredential("USERACCOUNT", "PASSWORD", "DOMAIN")
                    }
                };

                var httpClientHandler = new HttpClientHandler {
                    UseProxy = false,
                    Credentials = credentialsCache,
                    UseDefaultCredentials = false,
                };
                return httpClientHandler;
            });
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The above code should work fine both in Linux and Windows when doing a request to an endpoint that accepts the Kerberos authentication. &lt;/p&gt;

&lt;p&gt;If you want to use the default credentials of your server, and you already have that set up in your environment you can use the following code inside your ConfigureServices method:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;services.AddHttpClient&amp;lt;IConnector, Connector&amp;gt;().ConfigurePrimaryHttpMessageHandler(
   serviceProvider =&amp;gt;
            {
                var httpClientHandler = new HttpClientHandler {
                    UseProxy = false,
                    UseDefaultCredentials = true
                };
                return httpClientHandler;
            });
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To get the above code working on Linux, you first need to configure the Kerberos tickets inside your server. To do that you should have the Kerberos configuration file, usually called krb5.conf file. Read more on &lt;a href="https://web.mit.edu/kerberos/krb5-1.12/doc/admin/conf_files/krb5_conf.html" rel="noopener noreferrer"&gt;Krb5.conf file&lt;/a&gt;. In addition to the configuration file, you need to have some libraries installed in your server: krb5-config, krb5-user, syslog-ng realmd, gss-ntlmssp. After you have all set up you should initialize your ticket using the kinit command, read more on &lt;a href="https://web.mit.edu/kerberos/krb5-1.12/doc/user/user_commands/kinit.html" rel="noopener noreferrer"&gt;kinit command&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;Command example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;kinit -V keytabfile.keytab account@domain.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You can check your tickets running this other command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;klist
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After configuring it, you should be able to make a standard request using your HttpClient and .NET will add the Negotiate header with your Kerberos ticket. Hopefully, if the service you are calling is configured correctly, this should work. &lt;/p&gt;

&lt;p&gt;All the above code uses the default built in HttpClient provided by .NET. Alternatively, you can try to authenticate with this great library called Kerberos.NET, read more on &lt;a href="https://github.com/dotnet/Kerberos.NET" rel="noopener noreferrer"&gt;Kerberos.NET&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;Also, you can try to execute a direct call inside your Linux server using curl to check if everything is right with the configuration of your Kerberos credentials and tickets. To run a curl command using the Kerberos tickets use this syntax:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl -k --negotiate -u : -X --location --request &amp;lt;VERB&amp;gt; &amp;lt;URL&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Additionally, if you want to add the functionality of authorization and authentication with Kerberos or windows AD in a Linux environment you need to add a package in your project from Nuget, and add the following code to your Startup.cs :&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;package to install:  Microsoft.AspNetCore.Authentication.Negotiate
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;public void ConfigureServices(IServiceCollection services) {
   services.AddAuthentication(NegotiateDefaults.AuthenticationScheme)
      .AddNegotiate();

   builder.Services.AddAuthorization(options =&amp;gt;
   {
       options.FallbackPolicy = options.DefaultPolicy;
   });
   // code continues
}

public void Configure(IApplicationBuilder app) {
   // code before
   app.UseAuthentication();
   app.UseAuthorization();
   // code continues
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hope that helps someone who is trying to authenticate with Kerberos using .NET!&lt;/p&gt;

&lt;p&gt;Additional resources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://learn.microsoft.com/en-us/aspnet/core/security/authentication/windowsauth?view=aspnetcore-7.0&amp;amp;tabs=visual-studio" rel="noopener noreferrer"&gt;Configure Windows Authentication in ASP.NET Core&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/dotnet/Kerberos.NET" rel="noopener noreferrer"&gt;Kerberos.NET library&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kerberos.net/docs/index.html" rel="noopener noreferrer"&gt;Kerberos documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Originally posted on my blog: &lt;a href="https://lucaspacheco.dev/kerberos-authentication-dotnet" rel="noopener noreferrer"&gt;Lucas Pacheco Blog&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dotnet</category>
      <category>webdev</category>
      <category>linux</category>
      <category>csharp</category>
    </item>
    <item>
      <title>Thoughts on Legacy Code</title>
      <dc:creator>Lucas Pacheco</dc:creator>
      <pubDate>Wed, 11 Nov 2020 00:54:59 +0000</pubDate>
      <link>https://dev.to/lucaspsilveira/thoughts-on-legacy-code-1elk</link>
      <guid>https://dev.to/lucaspsilveira/thoughts-on-legacy-code-1elk</guid>
      <description>&lt;p&gt;Behold, you get your first job as a developer, excited to work on new technologies, the ones you looked at on YouTube videos, on Twitter trending topics, on blog posts, and on many other websites. Then, in the first months of work, you receive the following dose of reality: maintenance of a legacy code. Whoever has never had to keep a legacy code, can be sure that one day it will!&lt;/p&gt;

&lt;p&gt;Recently, I had to carry out some maintenance, and, believe me, adding new features to legacy systems. This prompted a question in me about the need to maintain legacy software and how to perform this maintenance. In addition, how we should develop our current codes to be maintained in the future.&lt;/p&gt;

&lt;h2&gt;
  
  
  Knowing your application
&lt;/h2&gt;

&lt;p&gt;Before start running the code, try to understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What does this application accomplish?&lt;/li&gt;
&lt;li&gt;How important is this application for the company?&lt;/li&gt;
&lt;li&gt;What are the affected areas?&lt;/li&gt;
&lt;li&gt;What is the purpose of your change?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I believe that making any changes to the code, before having answers to these questions, can be very dangerous for the business, because you simply do not know the possible impacts. Have you ever imagined a login service down because you didn't know there was a dependency? In addition, the time it takes you to develop something you don't fully understand is much longer than the time it took if you had studied the application before.&lt;/p&gt;

&lt;h2&gt;
  
  
  Application refactoring
&lt;/h2&gt;

&lt;p&gt;It is very common during the analysis of the old code to see that thought: "What did you do here? Why did you do it like this? This is very wrong!". Before judging someone else's code, remember: you don't know the circumstances under which this application was developed and you yourself have certainly written very similar code that you are seeing. At this point, you have the opportunity to improve this part of the code, and when possible, this should certainly be done, as you will hardly be the last to touch this application. There are several refactoring techniques and several books, so if you want to go deeper into the subject, I recommend reading one of these specialized books on the subject.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tests
&lt;/h2&gt;

&lt;p&gt;At this point, you may think, "There are no tests on the project! This technology is very old!" Again, here is the opportunity for you to accomplish them. Nowadays, delivering any application without testing in production must be well thought out and justified. Of course, there may be some very old technologies that performing unit tests or other tests are more complicated or even impossible to perform. For these cases, it will be necessary to perform very good validation with functional tests. Whenever possible, look for the person in charge of the business area to validate these changes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What should I do with my current applications?
&lt;/h2&gt;

&lt;p&gt;First, reliable documentation can make future application maintenance much easier. It can be a board in JIRA, some User Stories, a document with functional requirements, any information you may have about what is being developed. But the most important thing is to make this information accessible. It is useless to document everything and keep it on your computer.&lt;/p&gt;

&lt;p&gt;What environment settings are needed to run and develop your application? In one of my last projects, I had to perform maintenance on a system that only ran on Windows XP! Imagine how it was to get an XP machine in 2020 to develop. In addition, by knowing what resources are needed to start the application, a lot of time can be saved with execution errors.&lt;/p&gt;

&lt;p&gt;Develop tests for your applications! They will certainly assist you during the current development and will assist you when you need to change it in the future. In addition to ensuring that your future changes will not break the code that has been in production for years.&lt;/p&gt;

&lt;p&gt;When developing your current system, you can be sure that it will be bequeathed one day and you or someone will have to work on it in the future. So always think about it while working with your super-cool technology and writing code.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Today's hype is tomorrow's legacy!&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Thanks for your time! Comment below any tips or doubts.&lt;/p&gt;

</description>
      <category>legacycode</category>
      <category>legacy</category>
      <category>coding</category>
      <category>opinion</category>
    </item>
    <item>
      <title>Web Scraping com .NET Core 3.1</title>
      <dc:creator>Lucas Pacheco</dc:creator>
      <pubDate>Sun, 20 Sep 2020 23:00:00 +0000</pubDate>
      <link>https://dev.to/lucaspsilveira/web-scrapping-com-net-core-3-1-228m</link>
      <guid>https://dev.to/lucaspsilveira/web-scrapping-com-net-core-3-1-228m</guid>
      <description>&lt;p&gt;Recentemente tive a necessidade de buscar algumas informações em sites que não possuíam uma API para recuperar os dados necessários. Então, decidi que iria utilizar algumas técnicas de web scraping para buscar as informações que precisava. Já havia construído alguns web scrapers com Python e PHP. Porém, decidi me desafiar e criar um com .NET Core 3.1 e C#. Para isso utilizei um pacote, disponível no gerenciador de pacotes Nuget, chamado AngleSharp. Este pacote pode ser encontrado &lt;a href="https://www.nuget.org/packages/AngleSharp/1.0.0-alpha-827" rel="noopener noreferrer"&gt;aqui&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Para exemplificar aqui, vamos acessar o site &lt;a href="https://www.pensador.com/" rel="noopener noreferrer"&gt;pensador.com&lt;/a&gt;, e recuperar as citações que ficam disponíveis para visualização. Nosso objetivo então é &lt;strong&gt;recuperar as citações de um determinado autor ou tema e armazenarmos em nosso dispositivo.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Criação do projeto
&lt;/h2&gt;

&lt;p&gt;Minha necessidade era construir esse banco de citações uma vez, sem a necessidade de ficar atualizando ou ter algum acesso remoto. Por este motivo decidi criar uma aplicação de linha de comando, para isso utilizei o próprio template disponibilizado pelo Visual Studio. &lt;br&gt;
&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F3z4c3hhzzp3od47uw9wl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F3z4c3hhzzp3od47uw9wl.png" alt="Template"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Após nomear o projeto e realizar sua criação, é necessário instalar o pacote AngleSharp como uma dependência do projeto. Para isso é necessário clicar com o botão direito do mouse em 'Dependencies', e selecionar a opção 'Manage NuGet Packages'. Você deve visualizar uma tela semelhante com a imagem abaixo:&lt;br&gt;
&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Frcuudepvmkiabybkxadf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Frcuudepvmkiabybkxadf.png" alt="NuGet Manager"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Após clicar em instalar, nosso projeto está pronto para começarmos a realizar chamadas ao site.&lt;/p&gt;
&lt;h2&gt;
  
  
  Acessando o site
&lt;/h2&gt;

&lt;p&gt;Poderíamos realizar as chamadas diretamente no método Main da aplicação, mas eu prefiro separar em classes diferentes algumas operações. Sendo assim, decidi criar uma pasta Services e criar uma classe &lt;code&gt;ScrapperService.cs&lt;/code&gt; dentro. Essa será a classe responsável por acessar o site que desejamos e trazer nossos dados já modelados.&lt;/p&gt;

&lt;p&gt;Nessa classe &lt;code&gt;ScrapperService.cs&lt;/code&gt; vamos criar uma propriedade do tipo &lt;code&gt;IBrowsingContext&lt;/code&gt;, que é uma interface do pacote AngleSharp, que será responsável por fazer a requisição às páginas web que desejamos acessar. A classe junto com o seu construtor ficará assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;AngleSharp&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;namespace&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper.Services&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ScrapperService&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="n"&gt;IBrowsingContext&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;get&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;ScrapperService&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Configuration&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Default&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WithDefaultLoader&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
            &lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;BrowsingContext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nessa etapa já podemos escrever o método que irá realizar a leitura dos nossos dados de interesse, mas antes é importante abrirmos a página em nosso navegador para vermos como o DOM está organizado, para então organizarmos nossa estratégia de obtenção dos dados. Então, ao abrir a página inicial de algum tema de interesse podemos perceber a organização da página. Vou usar como exemplo citações com o tema de filosofia, a página pode ser visualizada abaixo:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F0g2dxsmc5b6rdttob3fy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F0g2dxsmc5b6rdttob3fy.png" alt="Site"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Ao analisarmos dessa forma, conseguimos visualizar onde estão os elementos de interesse para nossa aplicação, que são: as citações e número total de citações. Necessitamos verificar o número total de citações para podermos calcular o número de páginas estimado que cada tema ou autor terá. Assim, conseguiremos extrair todas as citações de cada um. &lt;/p&gt;

&lt;p&gt;Após a análise visual é necessário verificar a organização do documento HTML, como já sabemos nossos elementos de interesse, podemos clicar com o botão direito e inspecionar o elemento para irmos direto a ele no código. Verificando o elemento, podemos então clicar com o botão direito novamente e selecionar a opção de copiar seletor. Esse caminho é que utilizaremos para encontrar esse elemento em nossa aplicação com o AngleSharp, então guarde ele em algum lugar. Abaixo uma imagem exemplificando o que foi realizado.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F3afu8iy75hzzhxosjdqq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2F3afu8iy75hzzhxosjdqq.png" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Realizando a mesma etapa para o bloco das citações, temos dois seletores copiados:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Informação total de citações: #content &amp;gt; div.top &amp;gt; div.total&lt;/li&gt;
&lt;li&gt;Todas citações da página: #content &amp;gt; div.phrases-list&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Com esses dados em mãos podemos criar o método para buscar as citações na classe &lt;code&gt;ScrapperService.cs&lt;/code&gt;, o método fica assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="n"&gt;IAsyncEnumerable&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Quote&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;GetQuotes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;pagina&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;$"https://www.pensador.com/&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;pagina&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;OpenAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;quotesHtml&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelectorAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.phrases-list &amp;gt; .thought-card"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;quotesHtml&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;autor&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;".autor"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

                &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetAttribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"data-id"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

                &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;texto&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;".frase"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

                &lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;".insert-onlist"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;!=&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;

                &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;Quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;autor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;texto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Explicando o código acima:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Criamos um método assíncrono para buscar as citações, pois assim nossa aplicação não fica bloqueada enquanto realiza uma chamada HTTP para o servidor, e também pois o pacote do AngleSharp realiza essa leitura da página de forma assíncrona.&lt;/li&gt;
&lt;li&gt;Como a lógica do método pode ser aplicada pra qualquer página ou tema, vamos passar esses dois parâmetros, tornando o método mais adaptável.&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Montamos então a url de acesso concatenando os textos conforme o padrão que o site utiliza, que é algum desses dois:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;pensador.com.br*&lt;em&gt;/tema/pagina&lt;/em&gt;*&lt;/li&gt;
&lt;li&gt;pensador.com.br*&lt;em&gt;/autor/nome_autor/pagina&lt;/em&gt;*&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Na linha com o código &lt;code&gt;var document = await context.OpenAsync(url);&lt;/code&gt; é que realizamos a leitura do documento, com auxílio do pacote AngleSharp. Esse método nos retorna um objeto com a página mapeada onde é possível navegar através dos seus nós.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;

&lt;p&gt;Como já havíamos anotado nossos seletores, podemos utilizar a função QuerySelectorAll para retornar os nós de interesse nesse documento. Para isso então passamos o nosso seletor de todas as citações como parâmetro, adicionado da classe que todas citações compartilham em comum, como pode ser observado abaixo:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;var quotesHtml = document.QuerySelectorAll("#content &amp;gt; div.phrases-list &amp;gt; .thought-card");&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Após termos todos os nós das citações, podemos iterar sobre eles e buscar as informações internas de cada nó. Para isso utilizamos novamente o QuerySelector, porém agora passando os seletores de cada informação que queremos. Que nesse caso são: autor, frase e ID.&lt;/p&gt;&lt;/li&gt;

&lt;li&gt;&lt;p&gt;Como você pode perceber, esse método retorna um objeto Quote, e esse objeto deve ser criado por nós, para isso criei uma pasta Models, e então criei a classe modelo &lt;code&gt;Quote.cs&lt;/code&gt;, que pode ser visualizada abaixo:&lt;br&gt;&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;

&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;namespace&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper.Models&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Quote&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;Id&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;get&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;Autor&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;get&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;Texto&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;get&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;set&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="nf"&gt;Quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;autor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;texto&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;Id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="n"&gt;Autor&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;autor&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="n"&gt;Texto&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;texto&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Dentro do laço então retornamos esse objeto Quote, com auxílio da palavra-chave &lt;code&gt;yield&lt;/code&gt;, que fará com que as citações sejam retornadas assim que forem sendo recuperadas do site.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  E como descobrimos quantas páginas existem?
&lt;/h2&gt;

&lt;p&gt;Se nós chamarmos o método construído acima, nossa aplicação já vai retornar todas citações de cada página que passarmos como parâmetro. Mas e se quisermos executar nossa aplicação para buscarmos todas as páginas de um tema ou autor? Para isso que salvamos o seletor daquela parte da página que nos informava o número total de posts. Porém, se analisarmos outra página, de algum autor específico, por exemplo, veremos que algumas informações estão diferentes:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fqm4fl1oxavaw1xnu4l0q.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fqm4fl1oxavaw1xnu4l0q.png" alt="Site de Autor específico"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;O site pensador, quando tem um artista famoso, muda o texto de total de citações, e também muda a estrutura do HTML, para isso então precisamos mapear esse seletor também. Logo, temos dois seletores distintos:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Artista Famoso: #content &amp;gt; div.autorTotal&lt;/li&gt;
&lt;li&gt;Tema geral: #content &amp;gt; div.top &amp;gt; div.total&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Precisamos construir então um método que retorna o número total de páginas que cada autor ou tema terá. Uma forma simples de fazer isso é dividir o número total de citações pela quantidade que é exibida na primeira página, para termos um valor aproximado. Poderíamos também sempre analisarmos os elementos de paginação no final da página, até não existir mais nenhum. Mas nessa implementação vou somente dividir o número total de citações pelas exibidas na tela. Mas sintam-se livres para alterar o código no github e implementar essa funcionalidade. Segue abaixo como o método ficou:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="n"&gt;Task&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;GetTotalPages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;

            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;OpenAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"https://www.pensador.com/&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;quotesHtml&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelectorAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.phrases-list &amp;gt; .thought-card"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;textoTotal&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

            &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;quantidadePorPagina&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.top &amp;gt; div.total"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;!=&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;textoTotal&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.top &amp;gt; div.total"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;TryParse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Regex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;textoTotal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;@"(?&amp;lt;=de )(.*)(?= pensamentos )"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;out&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;quantidadePorPagina&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Regex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;textoTotal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;@"(?&amp;lt;=-\n)(.\d)(?=)"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                    &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Regex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;textoTotal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;@"(?&amp;lt;=de )(.*)(?= pensamentos )"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;else&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Regex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;textoTotal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;@"[^\d]"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
                    &lt;span class="n"&gt;quantidadePorPagina&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;quotesHtml&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Count&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.autorTotal &amp;gt; strong:nth-child(2)"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
                &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;textoPagina&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;QuerySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"#content &amp;gt; div.autorTotal &amp;gt; strong:nth-child(1)"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
                &lt;span class="n"&gt;quantidadePorPagina&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Regex&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;textoPagina&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;@"(?&amp;lt;= )(.\d)(?=)"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;paginas&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="p"&gt;/&lt;/span&gt; &lt;span class="n"&gt;quantidadePorPagina&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

            &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Total: "&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"Paginas: "&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="n"&gt;paginas&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;paginas&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Esse método ficou um pouco extenso, e poderia ser otimizado, mas deixaremos para fazer isso em uma segunda versão. Para extrair as informações de total de citações e paginação utilizei Regex. Explicando de uma forma geral, o método acessa a primeira página do autor ou tema e verifica se está no padrão de totais de citação do tema ou autor, então utiliza as técnicas de QuerySelector para realizar a captura dos nós. Após isso, o valor total de citações é dividido pela quantidade de citações por página e armazenado em um inteiro para arredondarmos para baixo o total de páginas. &lt;/p&gt;

&lt;h2&gt;
  
  
  Acessando todas páginas de um tema específico ou autor
&lt;/h2&gt;

&lt;p&gt;Agora que já possuímos um método para estimar o número total de páginas e outro método para buscar as citações dessa página, só necessitamos realizar um laço de repetição alterando a página até o total de página daquele autor ou tema. Para isso, podemos adicionar em nosso método Main da classe &lt;code&gt;Program.cs&lt;/code&gt; . A classe fica assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper.Services&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;System&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;namespace&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Program&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Threading&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Tasks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Task&lt;/span&gt; &lt;span class="nf"&gt;Main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;ScrapperService&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;ScrapperService&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

            &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;assunto&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"autor/william_shakespeare"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

            &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"\n\n\n\n\n\nAUTOR: william_shakespeare \n\n\n"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetTotalPages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"N de paginas: "&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;quote&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetQuotes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"ID: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt; | Autor: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Autor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;&lt;span class="s"&gt; | Frase: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Texto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"\n --------Página &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;--------- \n\n"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ao executarmos o código podemos ver como está funcionando nossa aplicação:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fwywvmyfdcvbvigink8i9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fwywvmyfdcvbvigink8i9.png" alt="Aplicação rodou"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Ainda, se desejarmos buscar dados de mais de um autor ou tema poderíamos criar uma Lista de autores e iterar sobre ela chamando os nossos mesmos métodos, dessa maneira:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper.Models&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper.Services&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;System&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;System.Collections.Generic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;namespace&lt;/span&gt; &lt;span class="nn"&gt;PensadorScrapper&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Program&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Threading&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Tasks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Task&lt;/span&gt; &lt;span class="nf"&gt;Main&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;ScrapperService&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;ScrapperService&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

            &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;autores&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;List&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;{&lt;/span&gt;
                &lt;span class="s"&gt;"jean_jacques_rousseau"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"rene_descartes"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"immanuel_kant"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"john_locke"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"blaise_pascal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"galileu_galilei"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"michel_de_montaigne"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"mary_wollstonecraft"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"angela_davis"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"hipatia_de_alexandria"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"maquiavel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"adam_smith"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"zygmunt_bauman"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"baruch_espinosa"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"friedrich_engels"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"georg_wilhelm_friedrich_hegel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"soren_kierkegaard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"epicteto"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"martin_heidegger"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"michel_foucault"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"martin_heidegger"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s"&gt;"hannah_arendt"&lt;/span&gt;
            &lt;span class="p"&gt;};&lt;/span&gt;

            &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;autor&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;autores&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;assunto&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;"autor/"&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="n"&gt;autor&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

                &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"\n\n\n\n\n\nAUTOR: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;autor&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt; \n\n\n"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

                &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetTotalPages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"N de paginas: "&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

                &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;totalPaginas&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
                &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;quote&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;scrapper&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetQuotes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;assunto&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
                    &lt;span class="p"&gt;{&lt;/span&gt;
                        &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"ID: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt; | Autor: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Autor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;&lt;span class="s"&gt; | Frase: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Texto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Trim&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                        &lt;span class="n"&gt;QuotesRepository&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;SaveQuote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                    &lt;span class="p"&gt;}&lt;/span&gt;
                    &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"\n --------Página &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;--------- \n\n"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;


        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Conclusão
&lt;/h2&gt;

&lt;p&gt;Pronto! Temos uma aplicação que realiza um Web Scraping no site pensador.com e retorna todas as citações disponíveis de um determinado autor ou tema. Claro que normalmente não é só isso que desejamos. Muitas vezes necessitamos armazenar essas informações. Por esse motivo eu já realizei também a integração dessa aplicação com um banco de dados, onde essas informações ficam armazenadas. Mas como o post já ficou extenso, vou tratar desta etapa em um outro post aqui. Utilizaremos SQLite para armazenar e recuperar esses dados. O código completo desse post, com a versão já utilizando o banco de dados pode ser encontrado no meu &lt;a href="https://github.com/lucaspsilveira/PensadorScrapper" rel="noopener noreferrer"&gt;Github&lt;/a&gt;. Abaixo temos um GIF da aplicação funcionando:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fomf08m52os4wcnzgarbz.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fomf08m52os4wcnzgarbz.gif" alt="Aplicação rodando"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dotnet</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>Testes unitários em .NET Core utilizando xUnit</title>
      <dc:creator>Lucas Pacheco</dc:creator>
      <pubDate>Mon, 27 Jul 2020 22:47:14 +0000</pubDate>
      <link>https://dev.to/lucaspsilveira/testes-unitarios-em-net-core-utilizando-xunit-45gc</link>
      <guid>https://dev.to/lucaspsilveira/testes-unitarios-em-net-core-utilizando-xunit-45gc</guid>
      <description>&lt;h2&gt;
  
  
  Sobre testes unitários
&lt;/h2&gt;

&lt;p&gt;Testes unitários são aqueles onde é possível verificar a execução da menor parte testável do seu código. Geralmente essa parte é uma função ou um método. Utilizar a automação desses testes é crucial para garantirmos que o código realize aquilo que nós desenvolvedores esperamos. &lt;/p&gt;

&lt;p&gt;Assim, devemos escrever nossos testes em um cenário limitado e específico. Testes mais complexos e que abrangem mais funcionalidades devem ser realizados através de testes de integração e serviços. Assim, os testes unitários devem certificar os códigos dentro do seu controle, abstraindo as questões de infraestrutura.&lt;/p&gt;

&lt;p&gt;Acredito que realizando tarefas práticas é o melhor modo de compreender, então vamos ao código!&lt;/p&gt;

&lt;h2&gt;
  
  
  Criando nosso primeiro teste
&lt;/h2&gt;

&lt;p&gt;Nesse exemplo irei demonstrar os testes unitários com .NET Core 3.1 e xUnit.&lt;br&gt;
O primeiro passo é criarmos um projeto, pode ser uma aplicação de linha de comando para testarmos alguns métodos. Irei utilizar o Visual Studio 2019 para os exemplos. Ao criarmos nosso projeto, devemos ter uma classe Program, com esse conteúdo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;using System;

namespace UnitTestsNetCore
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello World!");
        }
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Para exemplificar a realização dos testes, vamos criar uma abstração de uma calculadora. Essa calculadora terá 4 métodos: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Somar(decimal Num1, decimal Num2)&lt;/li&gt;
&lt;li&gt;Subtrair(decimal Num1, decimal Num2)&lt;/li&gt;
&lt;li&gt;Multiplcar(decimal Num1, decimal Num2)&lt;/li&gt;
&lt;li&gt;Dividir(decimal Num1, decimal Num2)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Para simplificar, cada método retornará o resultado apenas, logo nossa calculadora não terá um estado que armazenará os resultados. Nossa classe calculadora será apenas responsável por executar os métodos e retornar esse valor. No final temos uma classe com o conteúdo abaixo:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;namespace UnitTestsNetCore
{
    class Calculadora
    {
        public decimal Somar(decimal Num1, decimal Num2) =&amp;gt; Num1 + Num2;
        public decimal Subtrair(decimal Num1, decimal Num2) =&amp;gt; Num1 - Num2;
        public decimal Multiplicar(decimal Num1, decimal Num2) =&amp;gt; Num1 * Num2;
        public decimal Dividir(decimal Num1, decimal Num2) =&amp;gt; Num1 / Num2;
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Agora que já temos nossa calculadora, podemos criar nossa classe de testes! Para isso utilizei o Visual Studio para criar um novo projeto do tipo XUnitTest na mesma solução. Então vamos adicionar o primeiro projeto que criamos como referência para este último. Para realizar isto basta adicionar este item no arquivo .csproj.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;lt;ItemGroup&amp;gt;
    &amp;lt;ProjectReference Include="..\UnitTestsNetCore\UnitTestsNetCore.csproj" /&amp;gt;
&amp;lt;/ItemGroup&amp;gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Vamos criar então nossa classe de testes, e nela vamos instanciar a classe que desejamos em seu construtor (Calculadora). Após fazer isso já podemos escrever nosso primeiro teste. A classe de testes ficará assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;using Xunit;
using UnitTestsNetCore;

namespace XUnitTestProject
{
    public class CalculadoraTest
    {
        private readonly Calculadora _sut;
        public CalculadoraTest()
        {
            _sut = new Calculadora();
        }

        [Fact]
        public void Somar_DeveRetornarValor_QuandoInformadoAmbosNumeros()
        {

            var resultado = _sut.Somar(1, 2);
            Assert.Equal(3, resultado);
        }
    }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nesse momento, nós temos nossa classe Calculadora como uma propriedade da nossa classe de testes, e podemos utilizar ela em todos os testes que desejarmos. Para definirmos algum método como um método de testes precisamos adicionar o atributo [Fact], na linha acima do método. Desse modo o xUnit saberá que esse método é um teste e deve ser chamado. No conteúdo do método temos a execução da função Somar(). Na última linha, verificamos se a função ocorreu como esperado. O primeiro parâmetro é o valor esperado e o segundo o valor atual que temos de retorno. Se executarmos os testes, tudo deve ocorrer sem nenhuma falha. Como pode ser visto abaixo:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fym1cfatzv0w94hsmjazy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fym1cfatzv0w94hsmjazy.png" alt="First Test"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Antes de escrevermos mais testes, se olharmos para o nosso método veremos que ele está testando somente um cenário. Que é a soma de 1 e 3. Para testarmos diferentes cenários no mesmo método, podemos mudar o atributo [Fact] para [Theory]. Ao realizar isso, precisamos também passar quais os cenários que desejamos, para isso utilizaremos o atributo [InLineData(params)]. Cada [InLineData(params)] que colocarmos, o xUnit irá realizar um teste. Vamos alterar para Theory e alterar o nosso método para aceitar 3 parâmetros. Precisamos informar qual o valor esperado e os dois valores para soma. Então o código ficará assim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[Theory]
[InlineData(3, 2, 1)]
[InlineData(1, 2, -1)]
[InlineData(20, 20, 0)]
public void Somar_DeveRetornarValor_QuandoInformadoAmbosNumeros(decimal esperado,
                                                                decimal Num1,
                                                                decimal Num2)
{
    var resultado = _sut.Somar(Num1, Num2);
    Assert.Equal(esperado, resultado);
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ao rodarmos os nossos testes, a função será executada 3 vezes com os parâmetros definidos nos [InLineData(params)]. O retorno no Visual Studio ficará assim: &lt;/p&gt;

&lt;p&gt;&lt;a href="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fn5fvme91z79zqwrus0ab.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fi%2Fn5fvme91z79zqwrus0ab.png" alt="Second Test"&gt;&lt;/a&gt; &lt;/p&gt;

&lt;p&gt;A biblioteca xUnit possui diversas funcionalidades para testarmos os nossos códigos que vão além de verificar se o retorno é igual ou diferente. Podemos verificar logs, status de requisições, conteúdos, e inúmeras outras funcionalidades. Podemos configurar se os testes irão rodar em paralelo ou não, definirmos se a classe que está sendo testada será ou não compartilhada entre os testes, as possibilidades são enormes. Aqui está somente o básico de como testar uma função, recomendo fortemente a leitura dos materiais de referência que coloquei no final do post.&lt;/p&gt;

&lt;p&gt;O próximo post então será mais rápido e irá explicar algumas outras possibilidades um pouco mais avançadas utilizando xUnit e Mocks.&lt;/p&gt;

&lt;p&gt;O código do final do projeto, com os testes dos outros métodos, pode ser encontrado aqui: &lt;a href="https://github.com/lucaspsilveira/testes-dotnetcore-blog" rel="noopener noreferrer"&gt;Github&lt;/a&gt; &lt;/p&gt;

&lt;h2&gt;
  
  
  Referências
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.microsoft.com/pt-br/dotnet/core/testing/unit-testing-with-dotnet-test" rel="noopener noreferrer"&gt;Unit Test Microsoft&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=2Wp8en1I9oQ" rel="noopener noreferrer"&gt;Video aula no youtube&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://xunit.net/" rel="noopener noreferrer"&gt;Documentação xUnit&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>dotnet</category>
      <category>tests</category>
      <category>iniciante</category>
      <category>xunit</category>
    </item>
  </channel>
</rss>
